"Official" production setup / Nix usage / Migration Plan

[richardebeling]

Follow-Up to #2274: Do we also want production instances to be set-up using nix? It might simplify stuff. We may be able to test on a dev server.

• 2274 removed the requirements text files, so venv management might currently need to be done with poetry (although poetry can, according to the internet, produce a requirements file)
• We should make sure that the update_production.sh and load_production_backup.sh scripts match the envisioned setup.
  • Production update currently doesn't necessarily work, due to commented/missing pip package update after switching to new version
• We might need a plan for migration from a native installation to a nix installation
• We maybe should document why we trust the software we're using.

[richardebeling]

With poetry, we get poetry.lock. While it's nice to have reproducible, tested environments for new contributors, I think I'd want to ignore the lock file in production environments, so that it can upgrade patch versions if security issues are present in dependencies without us having to do a release. I don't want dependabot to spam us about patch updates in the lock file, so I'd want to put it in a "only for security patches" mode. This would also be a nice notification service to update those dependencies on production, if something ever comes up.

This is even more relevant for the webserver stack: Having a fixed version of python/postgres/redis/apache is nice, but on production, I'd want security fixes to be easily deployable. On Ubuntu Server, such updates happen automatically for apt-installed packages through unattended-upgrades. We even stuff like backported security fixes by the ubuntu team. From my understanding, the philosophy of nix is contrary to that: I want a compatible environment with all security patches, not an identical environment.

@niklasmohrin are you aware of anything that could get us the unattended-upgrades (and possibly even: auto-upgrade python dependencies within compatible versions, if security updates are released) experience with nix? A cheap and close solution could be a cronjob that rebuilds the environment regularly, but I don't want anything to happen if no security updates are available. Patch updates that fix bugs that we're not affected by shouldn't cause downtime.

I can see this working out if we keep a native installation, adapt the upgrade/restore scripts to work that way, and put a disclamer in the wiki that says "lock files are for dev environments, we don't release frequently enough to cover security updates in all dependencies". For CI-testability, the scripts would probably get a few more lines that would be non-nix-exclusive.

[Kakadus]

I don't know what we could do with nix in production, so here's my 2 cents about python and poetry.

We could install the evap package via pip. AFAIK when installing poetry packages with pip, the lockfile is "ignored", so only the compatible version restrictions per pyproject.toml are considered. When we wanted to update the dependencies, we could do something like pip install git+https://github.com/e-valuation/evap.git@release --upgrade (or pip install . --upgrade)
If we used poetry entirely, we could update with poetry update which updates and installs a new lockfile. This introduces changes to the git source, but makes it explicit that we have a changed (v)env

[niklasmohrin]

While it's nice to have reproducible, tested environments for new contributors, I think I'd want to ignore the lock file in production environments, so that it can upgrade patch versions if security issues are present in dependencies without us having to do a release.

Sounds reasonable. I would, intuitively, prefer an approach where we run with a changed lockfile over one where it is completely uncoupled from the running environment. It of course depends on the actual solutions we come up with though.

Having a fixed version of python/postgres/redis/apache is nice

I believe that EvaP should (within reason) work independent of the running database and not using nix for this is definitely in scope. This would also mean that for the production setup, we don't need to include process-compose-flake and services-flake in the "trusted software" list, only leaving nixpkgs, poetry2nix, and poetry. To facilitate this, we could make the scripts behind the npm-ci and init-django jobs available for nix run, but it doesn't seem like an urgent need.

Additionally, nixpkgs historically is rather bad at pushing out security updates in a quick fashion because a change to a package might trigger hash changes to a lot of child-packages, and those should be completed before an update can be published (this was astronomically bad with the xz incident, as it is included in the standard build environment).

So yeah, sticking with Ubuntu with unattended upgrades for postgres etc. seems like a good move.

For Python, using the system installation with nix is not possible. A Python update in nixpkgs would also take some time to propagate, because it's used in a lot of packages. Not sure how long it takes through the fastest channel though.

@niklasmohrin are you aware of anything that could get us the unattended-upgrades (and possibly even: auto-upgrade python dependencies within compatible versions, if security updates are released) experience with nix?

Not really, one can periodically update and reload, but afaik there is no builtin way. I am not even sure whether PyPI has information on which versions of a package are secure. For example Django 4.0.0 seems to be insecure, but the version is not yanked on PyPI. If we would get reliable information about insecure versions, I suppose one could also run a periodic check to see if any of the installed packages are affected and update them.

---

One thing that is sometimes done in Rust projects is building and testing with the minimally allowed version of every package. I was wondering whether this would also make sense for us, but given that we are not a library that others depend on and are therefore able to only allow dependency versions to vary in the "patch" category, this probably makes little sense for us.

---

We could install the evap package via pip.

I think that would be very cool, if it works reliably and without too many hacks.

If we used poetry entirely, we could update with poetry update which updates and installs a new lockfile. This introduces changes to the git source, but makes it explicit that we have a changed (v)env

Shouldn't pip also not care about our use of poetry, because we have specified it in the build-system section of pyproject.toml? As I previously hinted, I am not super into poetry itself but rather just want it to generate lockfiles every now and then. If at some point, there comes a tool that can read the then-surely-standardized pep-725 dependencies (or something similar) and generate hashes for our nix setup, I am absolutely ready to turn my back on poetry. However, if we like poetry we can also stick with it and/or use it in production. I just wouldn't get too invested with it unless it actually brings us value.

I think that the unavoidability of writing down what versions of dependencies are installed is nice and would much rather have it explicit like this than hidden away. In that sense, I think that running poetry to update the lockfile, then reloading the nix environment would also be an okay workflow. I am not the one who has to use it though :D