If you’re trying to add Rubocop to an existing project, you might’ve already stumbled upon the --auto-gen-config option for rubocop. This option will generate a rubocop_todo.yml configuration, which will allow you to run Rubocop without enforcing all the rules at once, and you can enable them one by one as you resolve them.

This is great, but we wanted to enable all the rules right at the start, so our editors would report all the issues, but we also wanted to add rubocop to the CI pipeline, and we didn’t want the pipeline to be failing until we fixed all of the issues.

So our solution was to create a regular rubocop configuration file that would be used by our editors, and a separate .rubocop-ci.yml file to be used by the CI pipeline. That second config would extend the first one and exclude all of the files in the codebase which have linting issues. This way, instead of fixing rule by rule, we can fix issues file by file.

So let’s say you’re working on some file where you’re adding a new feature. Now it should be your responsibility to fix all the rubocop issues in that file, and remove it from the exclude list in the .rubocop-ci.yml file.

One issue with this is that there will probably be a lot of files which won’t be touched for a long while, meaning they won’t be fixed anytime soon. You could plan some sessions in which you will take a chunk of those files, and go through them to fix the issues.

You can also run rubocop -a to autofix some issues.

Configuration setup

After you created the .rubocop.yml configuration file with the rules you want to use, you can create the .rubocop-ci.yml file with the following content:

inherit_from: .rubocop.yml

AllCops:
  Exclude:
    - "path/to/file/with/linting/issues/**/*"
    - "another/path/to/file/with/linting/issues/**/*"

Now here’s the fun part, generating the list of files with linting issues. Now you can use the --list-target-files/-L option to list all the files rubocop will inspect, and wrap them in the yaml format like so:

bundle exec rubocop -L | uniq | sed -e "s/\(.*\)/- '\1'/"

This will return a list of files with linting issues, which you can copy and paste into the .rubocop-ci.yml file.

Now you can run rubocop with this configuration in your CI pipeline, and it will only fail if you introduce new issues in the files that are not excluded.

bundle exec rubocop -c .rubocop-ci.yml

This one is a bit different than your usual useDebounce hook, instead of debouncing the change of some value, this instead debounces the call to some function. So here’s a simple debounced callback hook in React with TypeScript types:

type UnaryVoidFunction<T> = (arg: T) => void;

const useDebouncedCallback = <T,>(func: UnaryVoidFunction<T>, wait: number) => {
  const timeout = useRef<number>();

  const debouncedFunc = (arg: T) => {
    clearTimeout(timeout.current);
    timeout.current = setTimeout(() => func(arg), wait);
  };

  return useCallback(debouncedFunc, [func, wait]);
};

const App: FunctionComponent = () => {
  const [searchQuery, setSearchQuery] = useState("");

  const debouncedSearch = useDebouncedCallback(setSearchQuery, 500);

  return (
    <div>
      <input onChange={(e) => debouncedSearch(e.target.value)} />
      <div>{searchQuery}</div>
    </div>
  );
};

I have been a long time proponent against using password managers, and the reason for that being I do not want to trust some third party company in storing my passwords properly and securely. And yes, I know that those companies have been in the business for a long while, and that even if their DB does get breached (which happens), the passwords would still be secure (probably). But I would like complete control over that (while also not paying anyone anything).

So instead of using password managers, for a long time I had a system for having almost unique passwords for all the services, while not writing them down anywhere! Essentially, all passwords follow the same template, and then using site information (like the site name for example) to fill that template in. I was quite satisfied with this solution, but it was definitely not perfect. The main reason was that the passwords didn’t differ that much from each other, and another reason was websites with stupid password requirements which would force me to deviate from my password template, and then having to remember that deviation. And of course, being required to change passwords for some things on some set interval (for various security reasons), which was very difficult with this system.

I know about open source password managers that you can self host, like bitwarden. And honestly, you should probably just use that.

But I stumbled upon something a bit more interesting, and a lot more “simple” (simple as in the actual system is simple, but you will spend more time setting this up than using a service). It is the pass unix utility. It is super simple, it stores generated passwords in a plain directory, and each password is a gpg encrypted file. And has quite a few options for how the passwords are generated. And that’s it, nothing more.

Since it is a simple directory of files, you can use it as a git repository, and track password changes with it. And it does that for you, so you don’t have to manually commit the changes. And since it’s a git repostory, now you have unlimited options when it comes to hosting your password store. In my case, I use github, gitlab and my own self hosted gitea instance as the remotes for the git repo. The repos are private, but even if somehow someone breached those services and got the repo data, it would still be a challenge to decrypt those files.

I’m not going to explain how to set it up, it’s quite simple. There are executables for Windows, macOS and Linux, as well as apps for Android and iOS (which might be a bit more tricky to set up). And of course, there are also browser extensions (browserpass), a Windows program (pass-winmenu), a macOS menu bar app (Pass-for-macOS) and a cross-platform GUI app (QtPass) to make using those passwords easier (so you don’t have to open your terminal every time you want to log in somewhere).

I have been using pass for quite some time now (around 2 years), and I am quite happy with it, because it just works, and I am in complete control over the entire process, so if my passwords do get leaked somehow, it will be my own fault.

Now would I recommend using pass? If you’re a normal person, just pay for a password manager, or use the free tier of bitwarden. If you’re a little bit less normal, self host your own bitwarden instance. But if you’re more daring, and like to do things yourself, I would fully recommend pass.

You no longer have an excuse for having bad and insecure passwords, with so many options and approaches to password management, you’re welcome.

If you ever need to temporarily expose a service running on localhost (your new fancy website for example) and you have a server of sorts (let’s say a VPS) that’s publicly accessible from the internet already (since you just love to self host everything), you can use a reverse SSH tunnel to forward your localhost traffic to your server.

Here’s the command:

ssh -R 5125:localhost:5000 user@host.com

The first port number (5125 in this case) is the port on the remote machine where the traffic will be exposed to. You can then serve traffic from this port in whatever way you like. And the second port is your localhost port on which a service is running on.

This will tunnel the connection between your local machine and the remote server through the SSH connection. And once you kill that session, the tunnel stops.

I of course don’t recommend serving anything like this for long periods of time, at that point you might as well properly host the app on the server directly. One good use case for this is receiving third party API calls. This is a good alternative to using services like ngrok.

On my server I am running nginx and simply proxy passing port 5125 on some subdomain like so:

server {
    server_name sub.domain.com;

    location / {
        proxy_pass http://127.0.0.1:5125;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /robots.txt {
        return 200 "User-agent: *\nDisallow: /";
    }
}

Nothing fancy, passing some headers so my apps work properly, and disallowing search engines to index it.

And that’s it! Now the entire internet can access your absolute masterpiece of a web application!

Highlighting code is easy, but as soon as you need to be able to edit the code, and support a lot of languages (like I don’t know… how about 300+ languages?), and also have good highlighting, it gets really hard and confusing.

This post is about my findings on the topic of highlighting code in the browser, my ideas, my attempts and the final solution I decided to go with. There’s most likely a lot of flaws in my approaches, you can leave a comment here if you have thoughts on anything.

Motive

Let’s start with the motive behind this, why am I looking into a ton of different ways to do code highlighting?

Maybe you know this already, maybe you don’t, but I’m a creator of pastemyst, a website for sharing code. In the current version (v2) I’m using the CodeMirror editor, and it’s pretty great, it’s fast, easy to use and supports a bunch of languages. I’m using it both for editing the code and showing the paste to the users.

If CodeMirror is so great, why am I looking at other ways then? Because of basically just one reason, but it’s a big reason. CodeMirror uses pretty simple rules for highlighting, so that’s why it’s fast, but the results aren’t the best.

Here you can see the difference between pastemyst (CodeMirror) on the left and VSCode on the right. There’s only slight difference, but I couldn’t be bothered finding a better example.

Having good syntax highlighting on a website based around showing code is pretty important. So I tried finding other ways for syntax highlighting.

TextMate grammars

I first found about TextMate language grammars, used by editors like VSCode, Atom, and GitHub as well. The TextMate grammar files are really long, and hard to write, but they tokenize the code much better, and there’s grammar files for a ton of languages already made.

NOTE: Atom and parts of GitHub have switched to using tree-sitter, it works in a different way, and the grammar files are written in JavaScript / CSON. Supposedly you get better highlighting with it, and it’s not as monstrous as TextMate grammars.

Of course I tried looking for libraries that highlight code based on these grammar files, and found shiki. It’s an awesome library! It allows you to run the highlighter on a server (node) or in the browser (although you have to do a workaround for loading some wasm file).

But shiki is just a syntax highlighter, and not an editor. And here is where a lot of my problems actually begin. If you just want to show code, and don’t need editing and/or realtime highlighting, just use shiki, but if you do need an editor, then you’re going to have to sacrifice some stuff.

First approach - Separate editor

There are a few ways I thought of on how to deal with this.

The first is the most simple one, continue using CodeMirror for editing the page, and when the users view the paste render it using shiki. This is the easiest to implement, but you will end up having different highlighting in two different places. And yes, you can mitigate this somewhat by using the exact same theme in both places. But there is a bigger issue, CodeMirror just doesn’t support as many languages as I can support with TextMate grammars, the GitHub linguist repository, which has a list of all grammars they use, lists like 370+ languages and CodeMirror lists around 140 languages “built in”. So on a lot of languages you won’t even have any syntax highlighting in the editor, but will get rendered just fine once you make a paste.

And I know you could probably find language definitions for CodeMirror that are missing, but you won’t find a lot of them, and they probably aren’t getting updated anyway. I have a partial solution to this, when you are editing in CodeMirror, add a preview button which when pressed will render using shiki and show the actual result.

Here’s a quick video of this in action, and I know the themes don’t match up, it was a quick demo.

Second approach - Custom editor

The second approach requires a lot more effort. The idea is to use another editor library, one that supports using your own custom highlighter, or make my own editor that does this.

I don’t really want to write my own editor, that kind of stuff is just pain. So I found out about CodeJar, it’s a small editor that requires you to use your own syntax highlighter.

I wrote a quick demo where I used CodeJar for editing code, there’s actually two editors. One editor highlights code by running shiki in the browser, and the other editor connects to a node server through a websocket, and the server does the rendering.

Let’s look at this image now:

The last update time shows the time in milliseconds that it took CodeJar to update, basically says how long it took to highlight code. And just from this we can see that running shiki in the browser is much faster for some reason.

This is the code that’s running in a node server for this test:

And how about a… I don’t know… 10k lines of code?

Yup, that’s like 3.5 seconds on the browser test, and when I pasted that in both editors, it took a lot more than 3-4 seconds for it to highlight, it’s very slow. And for comparison pasting that into CodeMirror is instant, so yeah… This approach really doesn’t pan out.

Another issue with this approach is loading language files and themes, you don’t really wanna load 370+ json files when you open the website. But I guess a simple solution would be to just load the language once selected, and set a long TTL for caching.

Final words

In the end I have decided to go with the first approach, using CodeMirror for editing, and for now use shiki on a server to render code when a paste is viewed. I will also look into trying out tree-sitter and switch between these two depending on the language.

This way I can also cache every grammar file and theme on startup, and maybe even figure out a way to render the paste just once and store the result somewhere (the tricky part is supporting themes).

Now there are other syntax highlighters, like vscode-textmate, which is what VSCode uses, but I haven’t tested the performance of that.

There is also another editor which runs in the browser, which actually powers VSCode, monaco, but it’s really heavy and doesn’t support mobile. And I don’t think you can easily get more languages supported, but I’m not that sure on that.

And if VSCode can use TextMate grammars for editing, and VSCode is essentially a website bundled into an electron app, there’s probably some way to do fast highlighting for editors. But it requires you to write your own editor, using a lot of tricks probably, and that project alone could take a lot of time, and I simply don’t want to pour a ton of time into something like this, so I’m sticking with the not so elegant approach, but in the end it’s not so bad anyway.

If you have made it this far, thank you for reading! You are of course welcome to leave any thoughts you have here, you can also contact me through mail on my home page. Until next time.

If you don’t care for the backstory you can totally skip this chapter. And if you just want the complete script you can scroll to the end.

Let’s begin with some backstory first. If you do not know who I am, I created pastemyst, a pastebin clone of sorts (that is actually better than pastebin!). It is hosted on a cheap $5/m Vultr instance (very good VPS host, can recommend, use my referral link if you plan on getting an instance: Vultr ref), and ever since the huge 2.0 update of pastemyst building the program on that instance with just 1 CPU and 1gb of RAM was impossible because of the scale of the application. The biggest memory usage are the view templates because they get generated at compile time.

There were a couple of solutions to this. First was using swap memory which was a failure, even with the additional 2gb of swap dmd kept crashing with the out of memory error. Next I tried compiling it on my machine and uploading it to the server with scp, but alas, my genius didn’t last for long. I run Arch Linux on my machine (I use arch btw) which of course has every lib that ever existed updated to the most recent version there is, while my server running Ubuntu 18.04 (ancient I know) does not. Curse me and my obsession with being the cool kid who uses Arch (I am cool… right??). The library in issue was glibc, I tried updating it on the server (which can’t build it from source because it’s not very powerful as we established) and installing the same old version on my machine, but couldn’t figure it out. So to the next obvious solution we go.

VMs are a wonderful thing, you can run any OS of any version inside your other better OS. I installed the same version of Ubuntu in a VM using qemu, built pastemyst on that and uploaded to the server. And finally it worked! It was a long and painful process but I got pastemyst v2 running on the server and every user was amazed by the new version and my hard work paid off (I still don’t make any money from it though). With this approach it took a while to boot up the VM, and build pastemyst (which still wasn’t blazing fast because it’s a VM) and my upload speed is super slow, so in the end it effectively took around 30 minutes to get a build onto the server, but with no other way in mind I settled with it, I don’t release a new version too frequently anyway. But ha! I found a better way, I bet you couldn’t guess this was going to happen, my genius didn’t reach its limit just yet!

Using GitHub Actions

So while being bored one day I realized that hey, I can use the fancy GitHub Actions (which I used so far for building and testing every commit) to also build pastemyst and upload to the server. It took a bit of tinkering but I got it to work and the result is pretty simple.

Let’s go step by step on how I got to the final result. First you will need to create a new workflow script (which is actually just a YAML file… why GitHub, why… relevant article). The workflow files are located in .github/workflows/.

So do you want to deploy your program on every commit to the main branch? No you don’t, but GitHub has a solution for that! You can set the workflow to execute on: workflow_dispatch and so you will have a nice big button to press every time you want to deploy. You could make this execute on every new tag / release if you Googled hard enough.

name: Deploy to server

on: workflow_dispatch

jobs:
  deploy:
    name: Deploy to server
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@v2

Let’s add another step to install D, in this example I’m also downloading the libscrypt dependency and checking out all the submodules (which for some reason requires a third party plugin… why).

- name: install D
  uses: dlang-community/setup-dlang@v1
  with:
    compiler: dmd-latest
- name: Install libscrypt
  run: |
    sudo apt-get install libscrypt0
    sudo apt-get install libscrypt-dev
- name: Checkout submodules
  uses: srt32/git-actions@v0.0.3
  with:
    args: git submodule update --init --recursive

Time to now build the project in release mode.

- name: build
  run: dub build -b release

Now the interesting part. There’s two more steps, copy the resulting binary to the server, and then ssh into the server and move the binary to the right place and then run git pull to get all necessary assets like css and stuff.

You should generate an SSH key on your local machine like so:

ssh-keygen -t rsa -b 4096 -C "your@mail"

When you execute the command it will ask you for the path, as you will use this SSH key just for the GitHub action make sure not to leave it as a default or it might overwrite your existing SSH key (you can delete the key from your local machine after you have this setup).

This is now a very important part, when it asks you for a passphrase leave it empty, I learned this the hard way after many times asking myself why isn’t this working???? (I think you can set a passphrase and then provide a passphrase key to the action but I haven’t tested this).

Go copy the public key and add it to your ~/.ssh/authorized_keys files on your server, this allows anyone with the private key that matches the provided public key to ssh into the server, and in this case GitHub.

Now you should go into your GitHub repo, go to settings and open the Secrets section. You should add a new secret for the SSH username, host, port and private key. The SSH_SECRET should contain the private key you got from generating the SSH key.

We can now add the SCP step to the workflow.

- name: upload to server
  uses: appleboy/scp-action@master
  with:
    host: $
    username: $
    key: $
    port: $
    source: "bin/pastemyst"
    target: "builds"

The source and target fields are weird, with the way I set them up from the example the binary ends up in builds/bin/pastemyst for some reason, but oh well, we are going to SSH into the server and move it anyway.

And for the final step.

- name: move binary and pull repo
  uses: appleboy/ssh-action@master
  with:
    host: $
    username: $
    key: $
    port: $
    script: |
      rm pastemyst/pastemyst
      mv builds/bin/pastemyst pastemyst/pastemyst
      cd pastemyst
      git pull origin main

The script first deletes the current binary file, then it moves the new one to the right place and finally pulls the repo.

You should now have a “Run workflow” button when you go to Actions -> Deploy to server -> Run workflow

If you did everything correctly your new program will now deploy super fast and you shouldn’t need to do anything else (except maybe do systemctl restart service, but that can also be added to the script). But of course it won’t work first try and you will end up with 10 commits saying “updated deploy script” and 10 failed workflow runs, but CI/CD do be like that.

Also big thanks to GitHub for allowing us to run bad code on their servers for free.

Full workflow

Here’s the full workflow script for all your lazy people (or people who just want to get stuff done fast). And it’s displayed using a pastemyst embed just to flex this amazing feature you should definitely use (if the script doesn’t actually load let’s just say this was the comedic part of the post).