Refactor securedrop-admin to be a Debian package

[micahflee]

Fixes #7574, #7647.

This PR adds Debian packaging for securedrop-admin. But in order to do that, I also had to completely refactor the admin code.

The way it currently works, files get dropped into install_files/ansible-base, and also we write settings to install_files/ansible-base/group_vars/all/site-specific. With debian packaging, we need to save writable files somewhere else. In Qubes this is ~/.securedrop-admin/, and in Tails this is ~/Persistent/.securedrop-admin/.

This should work in both Qubes and Tails, with installation instructions described below -- we'll need to update the docs. But first, here are some other big changes.

I've made check_for_updates no longer rely on git tags and instead use a new ansible playbook and apt to check if the securedrop_admin package is out-of-date or not. I've also completely deleted the GUI updater, and modified the Tails extension to no longer have a "Check for SecureDrop updates" option in the menu.

securedrop-admin verify does not work. It requires test dependencies to be installed in the virtualenv, and it runs code in the devops folder in the root of the git repo--and we don't have the git repo anymore. I suggest that we remove verify and instead find a different way to run testinfra tests.

I did spend time getting the admin tests working though. When building the admin test container, it builds the debian package and installs it, and then run tests against that. I also fixed all of the tests so that they work again with all the refactoring, and I deleted a bunch of tests. I removed all of the GUI updater tests, since I deleted the GUI updater, and I also deleted all of the tests related to checking for updates and verifying git tags and stuff, since that has all been ripped out too.

Test plan

Here's the new SecureDrop instructions. There's different instructions when testing from Qubes and from Tails to start, and then you follow the same instructions on both.

From Qubes

Set up servers by following the docs, but save setting up ssh keys to later.

Build the deb:

make build-debs-admin

This will make build/trixie/securedrop-admin_2.13.0~rc1+trixie_amd64.deb. Copy this to a debian 13-based template (e.g. debian-13-clone) and install it.

Create sd-admin VM based on debian-13-clone.

Open a terminal in sd-admin.

Now set up ssh keys.

Create the ~/.config/securedrop-admin folder:

mkdir -p ~/.config/securedrop-admin

From Tails

• Build the deb in Qubes.

• Create a Tails USB with a Persistent volume, with all checkboxes turned on.

• Transfer ./tails-bootstrap.sh and build/trixie/securedrop-admin_2.13.0~rc1+trixie_amd64.deb to Tails:

• In Tails, run sudo bash tails-bootstrap.sh to set up persistent configs, then restart tails to apply the persistence changes

• After restarting, install the debian package:

  sudo apt update
  # this installs the dpkg but fails to configure it
  sudo dpkg -i ./securedrop-admin_2.13.0~rc1+trixie_amd64.deb
  # this installs all the deps and configures it
  sudo apt --fix-broken install

Continue for both

• Copy the following files to ~/.config/securedrop-admin/ :

  • the Submission Public Key file
  • the OSSEC Alert Public Key file

• Configure:

  securedrop-admin --force sdconfig

• Install:

  securedrop-admin --force install

• Then configure the admin workstation:

  securedrop-admin --force localconfig

Now you should be able to ssh app, ssh mon, and run securedrop-admin --force install again, and it should all go over onion services.

Checklist

This change accounts for:

• any required additional documentation
• any necessary AppArmor changes (added or removed application files)
• any impact on new SecureDrop installs and upgrades
• our dependency update policy

This PR requires a lot of new documentation, and has major impacts on new SecureDrop installs and upgrades. securedrop-admin doesn't have an AppArmor policy, but should it?

[zenmonkeykstop]

Took an initial code pass (have not yet tried building and installing packages). The main thing that leaps out is the different config paths for Tails and Debian. It should be possible to have both use the same path if it's a dotfile directory in the user home.

[zenmonkeykstop]

Looks like having a unified config path across Tails and Qubes is trickier than I thought. The initial idea of using Tails' dotfiles persistence is too unwieldy, as individual files in the dotfiles directory are symlinked into place from /live/persistence at login, so new files just get created in the underlying non-persistent filesystem. So ~/.securedrop-admin (or wherever) would need to be created as follows:

• add the persistent dir owned by amnesia:amnesia, say /live/persistence/TailsData_unlocked/securedrop-admin
• add a line in /live/persistence/TailsData_unlocked/persistence.conf like: `/home/amnesia/.securedrop-admin source=securedrop-admin
• reboot (or mount it ourselves but rebooting probably best)

this would need to happen before sdconfig ran (but only for Tails) - we could do it in a securedrop-admin tails-setup command or in the hypothetical bootstrapping script that would also need to make changes in persistence.conf to persist FPF apt repo details.

[legoktm]

Possibly stupid question (and I haven't read everything yet so might've missed something), but can we just use ~/Persistent/.securedrop-admin in both places? It's a special path in Tails, but in Qubes it just happens to be in a folder named "Persistent" (despite the whole VM being persistent).

[zenmonkeykstop]

It's not stupid, but it would be weird :) - I actually wanna stick it in $XDG_CONFIG_HOME for good citizen reasons. Given that we're changing things up anyway, aiming to have as little potential for user error as possible seems worthwhile.

There are some other gotchas in either case - default ansible behaviour is to write to the ansible home dir, so backups are also gonna end up in the hidden directory. It might be nice to write to the user's current working directory by default or optionally be able to specify a path. Same with the pubkeys, would be nice to copy them to the config dir directly once the user gives their initial location in sdconfig

[zenmonkeykstop]

Couple more notes

[zenmonkeykstop]

This is never used as far as I can tell.

[zenmonkeykstop]

We can probably either drop files in this directory (desktop icon stuff, no longer used) or move them to the .securedrop-admin config dir.

[zenmonkeykstop]

I've added a script to setup persistence in Tails (eventually it will also do the apt setup) and updated the test plan accordingly.

[legoktm]

Just did a pass on the Debian packaging.

As a general note I do think it'll be worth extracting out some of the broader fixes from this PR and merging them separately and ahead of time, like the UBUNTU_VERSION -> OS_VERSION stuff.

[legoktm]

This logic should all happen inside the packaging, i.e. debian/rules. There's a sed command in the current packaging that fixes up the paths so the venv will work.

And the other files should be included via install, etc.

[zenmonkeykstop]

virtualenv creation was moved to debian/rules, other files installed via debian/install - though the commands to move them to a working copy still live here.

[legoktm]

Should we default to bookworm here? I guess it doesn't really matter since it's always set.

[zenmonkeykstop]

trixie now :)

[legoktm]

I don't think we need most of these python- dependencies at runtime, the e.g. virtualenv has already been created, we shouldn't be pip installing anything. Same with the -dev ones, since we're not compiling anything.

[zenmonkeykstop]

removed.

[legoktm]

IMO non-reproducibility is a blocker to move forward, but it's literally just the bytecode that's unreproducible: https://gist.github.com/legoktm/fc213ffb0dbe20385e50441aaf544a28

If you look in the current Debian packaging we already have an override to strip all the bytecode, we should do the same here.

[zenmonkeykstop]

package should now be reproducible

[micahflee]

I've hit blocks doing some of the client-side write work this week. So now that removing focal has landed, I'm going to spend some time rebasing this PR.

[micahflee]

With the help of Claude Code, I have rebased develop into this branch. I'm also currently working on getting all of the Github Actions checks to pass. Rather than force pushing (in case we want to move forward in a different way), I'm pushing my changes to admin-deb-rebased: https://github.com/freedomofpress/securedrop/tree/admin-deb-rebased

My plan is, once I succeed in having all of the checks passing, I will do manual testing of admin tools.

[zenmonkeykstop]

Got wires crossed and did a manual rebase while the Claude one was in progress. Manual looks slightly better so I've force-pushed it. Next up:

• double checking stuff still works under trixie (Tails 7 or Qubes/debian-13-xfce)
• resurrecting enough of the updater to support a 1-pass upgrade for git-based workstations
• more upgrade script.

[micahflee]

I've been working on #7647, but pushing commits into this PR, so when it's ready it will close that issue as well. I've made a lot of progress. Here are the recent changes.

Migration scripts

The ./securedrop-admin file in the root of the repo is a new script that only only supports the setup and tailsconfig commands. This is only meant to facilitate the migration from git-based updated to deb-based updates, and the only time this script should run is when running the GUI updater.

After the migration is complete and the user reboots, there will be no more GUI updater. From that point on, the ./securedrop-admin script from the git repo isn't executed, and instead /usr/bin/securedrop-admin is executed.

The migration scripts include:

• migrate-to-deb.sh: this is what's executed when the GUI updater runs ./securedrop-admin setup. It validates that the user is in Tails and has an existing config. Then it installs the deb package, configures persistence for ~/.config/securedrop and bind-mounts it, and copies all of the config files from ~/Persistent/securedrop/install-files into ~/.config/securedrop.
  • migrate-install-package.sh: executed as root by migrate-to-deb.sh. It installs ~/Persistent/securedrop-admin.deb temporarily -- we still need to update this to add the FPF SD repo and install the package from there
  • migrate-setup-persistence.sh: executed as root by migrate-to-deb.sh. This sets up persistence for ~/.config/securedrop-admin and then bind-mounts it, so that it can be used right away without requiring a reboot
• migrate-tailsconfig.sh: this is what's executed when the GUI updater runs ./securedrop-admin tailsconfig. This just runs /usr/bin/securedrop-admin localconfig, and then pops up an alert telling the user to reboot Tails when it's done

Changes to ansible playbooks

The GUI installer has logic to ask the user for their sudo password, and then it passes this password into the ansible-playbook command after the "SUDO password:" prompt.

The way securedrop-admin was written, there was a @update_check_required decorator before all of the commands that used ansible to check for updates using apt (instead of git fetch), which could be bypassed with --force. So this means that if you run securedrop-admin localconfig it would run two ansible playbooks:

• for checking for updates
• for configuring Tails

The GUI installer only provided the sudo password for the first time it was prompted for "SUDO password", and when it was prompted a second time, it would just hang and after 2 minutes the GUI installer would timeout.

To fix this, I refactored the ansible playbooks. I made a new update-check role that checks for updates (which is skipped if the skip_update_check is true), and I added this role to the beginning of all of the playbooks. When securedrop-admin subprocesses to ansible, it includes skip_update_check=true if --force is passed in.

So in the end, it does the same thing, but with only a single ansible playbook getting executed instead of two separate ones, so it succeeds with the GUI installer.

Testing

Here's how to test it:

• Set up an admin Tails USB that is configured to admin a SecureDrop server (it should have a ~/Persistent/securedrop folder), and make sure it works to ssh to app and mon, connect to the source interface, etc.
• In Qubes, checkout this admin-deb branch and make a deb with make build-debs-admin
• Copy build/trixie/securedrop-admin_2.13.0~rc1+trixie_amd64.deb to the Tails persistent volume as ~/Persistent/securedrop-admin.deb (the exact path is important, since it's hardcoded into the migration script)
• In your Tails, create ~/Persistent/changes.diff that contains this file:

diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
index faac742df..4c9c9ac49 100755
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -762,7 +762,7 @@ def update_check_required(cmd_name: str) -> Callable[[_FuncT], _FuncT]:
         @functools.wraps(func)
         def wrapper(*args: Any, **kwargs: Any) -> Any:
             cli_args = args[0]
-            if cli_args.force:
+            if True:
                 sdlog.info("Skipping update check because --force argument was provided.")
                 return func(*args, **kwargs)
 
@@ -1028,6 +1028,10 @@ def get_release_key_from_keyserver(
 
 def update(args: argparse.Namespace) -> int:
     """Verify, and apply latest SecureDrop workstation update"""
+    subprocess.call(["git", "checkout", "."], cwd=args.root)
+    subprocess.call(["git", "checkout", "admin-deb"], cwd=args.root)
+    subprocess.call(["git", "pull", "origin", "admin-deb"], cwd=args.root)
+    return 0
     sdlog.info("Applying SecureDrop updates...")
 
     update_status, latest_tag = check_for_updates(args)
diff --git a/journalist_gui/journalist_gui/SecureDropUpdater.py b/journalist_gui/journalist_gui/SecureDropUpdater.py
index 75e801803..14882ccfb 100644
--- a/journalist_gui/journalist_gui/SecureDropUpdater.py
+++ b/journalist_gui/journalist_gui/SecureDropUpdater.py
@@ -88,6 +88,12 @@ class UpdateThread(QThread):
         self.failure_reason = ""
 
     def run(self):
+        subprocess.call(["git", "checkout", "."], cwd="/home/amnesia/Persistent/securedrop")
+        subprocess.call(["git", "checkout", "admin-deb"], cwd="/home/amnesia/Persistent/securedrop")
+        subprocess.call(["git", "pull", "origin", "admin-deb"], cwd="/home/amnesia/Persistent/securedrop")
+        self.update_success = True
+        self.signal.emit({"status": True, "output": "", "failure_reason": ""})
+        return
         sdadmin_path = "/home/amnesia/Persistent/securedrop/securedrop-admin"
         update_command = [sdadmin_path, "update"]
         try:

• In Tails, checkout the latest git tag and apply that diff:

  cd ~/Persistent/securedrop
  git checkout 2.12.10
  git apply ../changes.diff

• Before proceeding, make a clone of your Tails USB, and backup the persistent volume. This is required to be able to test this more than once. After you do the migration process, your Tails will be totally changed, so you can just boot from the backup USB, use the "Back Up Persistent Storage" tool to overwrite your other Tails persistence, and then try again, switching between USBs each time you try.

When you're ready (your ~/Persistent/securedrop must be in the 2.12.10 branch and patched, and you need a ~/Persistent/securedrop-admin.deb), do the upgrade.

• Click SecureDrop > Check for SecureDrop Updates.
• Switch to the Detailed Update Progress, and click "Update Now"

This will use the patched GUI updater to switch to the admin-deb branch, to simulate switching to the latest release branch. Then it will run /home/amnesia/Persistent/securedrop-admin commands, which calls the new migration scripts.

It will ask for your password twice, using pkexec to run commands as root. The first one is installing securedrop-admin.deb, though this will change to instead add the repo and install the package via apt:

The second time it's setting up ~/.config/securedrop-admin persistence:

Then it asks for your password again, in a slightly different way, to run the ansible playbook:

Then you wait a moment for the localconfig playbook to finish, and it shows this popup, telling you to reboot Tails:

Reboot Tails. To test it, you'll also need to manually install securedrop-admin.deb, but we won't need to once we install the package from our apt repo:

sudo dpkg -i ~/Persistent/securedrop-admin.deb

After you connect to Tor, you'll see the new GNOME extension no longer has an option to check for updates:

You should be able to ssh into app and mon, connect to the journalist interface, etc. But this time, it's pulling the securedrop config from ~/config/securedrop-admin.

You should also be able to open the console and run securedrop-admin install, to reinstall the server. This should ensure that you're using the latest version of securedrop-admin from apt before it proceeds (unless you use --force).

(However, I just discovered that the journalist key and the OSSEC key can have arbitrary filenames, and the migration script isn't copying them, so I'm going to fix that too.) (fixed)

Next steps: install from apt repo

Before this is done, we'll need to make the migration scripts add the FPF apt repo and install securedrop-admin from it. It's a chicken and egg problem though, since we won't have the securedrop-admin deb package made an in an apt repo until this PR is merged.

So I think the next step is to create a deb from this branch and publish it to the apt-test.freedom.press apt repo. I don't know how to go about doing that.

Then, I can update this branch to install the test repo and then install securedrop-admin from it.

Once we have confirmed that that works, we can switch the migration script to add the apt.freedom.press repo. It's unclear exactly how this might conflict with users who also have the dangerzone apt.freedom.press repo installed (called dangerzone.sources), so maybe this should actually just follow the dangerzone instructions and keep the source dangerzone.sources.

[zenmonkeykstop]

rebased to clear conflicts - @legoktm stg-admin-deb should now be passing (with one consistently flaky testinfra test newly skipped), lmk if you're good to merge.

[legoktm]

Most recent set of changes LGTM, with the note that cffi is also a production dependency and in general we should keep the develop + prod versions in sync, but we can fix that for the next release and not block this on the diff review.

I synced stg-admin-deb with this PR so it'll show up on this PR, once it's green I think we're good to go!