Let’s see… so I’ve got an app that can be built for Mac, Linux and Windows, a CI/CD pipeline that can test and make release builds so I don’t have to do it manually, and if users try to use the app now they’ll be able to get something out of it.

That’s almost everything. The only thing left that I can see is ensuring the process is secure.

I’m sure every dev would agree that security in an app is an important consideration, especially in something networked, like a webapp. But it’s also important in an app’s release process, especially an app like mealplanner.

mealplanner is desktop app and is distributed as a binary that users must install and run. And unlike electron, I build the binary myself rather than provide JavaScript that is run by a pre-existing program within a sandboxed environment.

An attacker that gains control over mealplanner’s release process would be able to distribute malicious code to users, code that is run with fewer restrictions. Which is a very bad thing that I do not want to occur. So I need to ensure the release process cannot be compromised.

A quick disclaimer: I have experience in thinking about security from being a developer for the past 10+ years, but… I have no idea if I’m a security expert or not. I am probably far more paranoid than the average developer though.

Goals for a secure build system

First I should think about what it means for something to be secure. For me, this means two things:

  • Control of the system cannot be compromised by a third party and made to do something it wasn’t designed to. In the case of a build system, it means that an attacker cannot modify the contents of a release.

  • In case the above proves to be false, then the damage that can be done to users is somehow mitigated/minimized, and the presence of an attack can be quickly uncovered.

To analyze how secure the system currently is and find any issues to fix, I’m going to look at every point in the system that is potentially vulnerable to an outsider. So every part of the system I don’t have complete control over, which includes external systems it communicates with.

For each system, I’ll pretend the external system is compromised and think about:

  • how to prevent that from causing problems for mealplanners release system
  • how to mitigate the effect of such a compromise in case prevention is not possible
  • and how to quickly detect if the external system is compromised

Gitlab

Gitlab is where I put my code, where the docker containers used in building are stored, and where the CI/CD pipeline is run. If it were compromised (or my account credentials compromised), well, an attacker would be able to do whatever they wanted with mealplanner.

Unfortunately, there isn’t much I can do about this. Other than keeping my credentials safe, using 2FA, rotating keys/passwords, it’s mostly out of my hands. I can, however, keep an attacker from causing a lot of harm, by keeping part of the release process, specifically code signing, out of Gitlab.

Code signing is the process of cryptographically signing a file using a secret key to show it hasn’t been altered by someone else. If the secret used is not accessible to Gitlab, then an attacker wouldn’t be able to create official releases, despite being able to make arbitrary changes to the code.

Anyway, if Gitlab can’t sign mealplanner, then it doesn’t really make sense for the CI servers to actually build and bundle releases. If the code or build process is compromised and I don’t notice, and I just take one of the releases uploaded by the pipeline and sign it, well that defeats the purpose of the limitation in the first place.

So it was a mistake, then, to rely on Gitlab for the actual release process. From now on I’m going to use it just for testing. Since my build process goes through docker, doing it locally isn’t particularly difficult. And it’ll be more secure.

Note: this probably wouldn’t be an issue for a large company. They might have their own enterprise version they manage themselves on a private network and wouldn’t have to worry as much about an attacker getting access. Alas as a single person, I can’t afford that (both in terms of price & time).

It would be nice to find a way to automate this process this though. I would like to support the ability for a single person to create secure software rather than having to rely on a capitalist system.

There’s also potentially an issue of an attacker changing the code, and me using it locally, without realizing, to create a release. Since I’m using git, any changes in Gitlab will become apparent locally due to the git hash being different after the change. If for some reason I create a new repo from scratch though, and an attacker changes something, I may not realize it.

It is certainly an edge case, but I can’t think of a way of preventing this. Not at the moment anyway. Maybe it wouldn’t be an issue if gitlab kept an unalterable history of changes made to a git tag, but this isn’t a change I can make.

Action items

  • Add code signing to the pipeline for every target OS.
  • Remove release building from .gitlab-ci.yml.
  • Explore if there are protections that can be made to disallow force-pushing to main/tags.

docker hub

mealplanner’s build process for every target OS is run through docker containers. The containers are built from base containers defined on docker hub. If a container that I use is compromised on docker hub, then so is my release process.

Fortunately, there’s a pretty simple way to solve this. Docker supports image signing to prove an image has not been altered, and supports limiting the containers that are run to only those that are signed.

Action items

  • Locally configure docker to only use signed images.
  • Create an environment setup checklist that includes steps like these.

cargo/npm dependencies

If any one of the MANY cargo/npm dependencies mealplanner depends on ends up compromised, the app could be.

Fortunately this problem has been solved already. Nowadays package managers have lock files that include a hash used for an integrity check. I just need to be careful when updating dependencies. Though I guess that’s easier said than done.

Action items

  • Create a checklist for updating dependencies that includes verifying updated dependencies are ok.
  • Some of the dependencies currently used are from git repos. For them, switch to using a specific revision instead of tag or branch.

build tools like cross & osxcross

These tools both come from forks I made to fix some things, so they are installed through git. The easiest way to make sure they do not get modified is to lock them to specific revisions. If the content changes, the git hash changes so I’ll always be using the code I want to use.

Action items

  • Switch to using revisions for osxcross/cross in the build Dockerfiles.

apt

There are many supporting packages used in the docker images that come from apt repositories. No choice for me to do anything but assume they are secure. But I can reduce the risk by only using official repositories.

Action items

  • Get rid of any uses of custom PPAs (assuming there are any).

detecting if the build is compromised

Part of doing defensive security is assuming that even after trying to keep attackers away I will at some point fail. In this case, how would I be able to tell if the release process is compromised?

The answer here, for both users and myself, is, I believe, code signing again. For users, the OS they’re using may provide a warning before they try and run an app that isn’t signed or is signed incorrectly. That’s probably the best I can do there; there’s not much I can do to make users take that warning more seriously, especially as someone who routinely ignores them myself.

For me as a developer, I want to make sure I notice when the release process has been compromised. There are two possible results of the release process being compromised: an attacker uploads their own binaries to gitlab for users to download, or an attacker changes some code used during the build process causing the binary to be different than intended.

The first result I can detect by checking the signing of release binaries myself periodically. This could be automated in fact. The second is a bit harder to check, but it might be possible to notice an issue if I periodically build the release locally and check that it has the same checksum as the uploaded binary of the same version. If at any point something in the binary changes, I’d notice.

Action items

  • Automated test to check uploaded binaries are signed correctly.
  • Automated test that builds the app for the latest version, for every target OS, and checks that the result is the same as the binary hosted on gitlab.
  • Some mechanism to run the above two tests weekly or so. Shouldn’t be on gitlab.

Think I’m done now… and oh look! A whole new list of things to do! Oh well. Once I finish them, I’ll be able to create an early alpha release. Then people can use if if they want to. Yay!