Develop and implement Tails/Git -> Tails/package migration script

[legoktm]

We need a migration path to convert existing Tails workstations using a Git checkout ("Tails/Git") to the newer Tails workstation using a Debian package ("Tails/package").

[zenmonkeykstop]

Discussion here: #7617

[zenmonkeykstop]

Following on from other chats, it's likely we can actually migrate in one step using the existing securedrop-admin script in the project root. Say version N is the migration version:

• when the N tag is available, the N-1 QT-based updater will do the following:
  • update the checked out git repo to N (with `./securedrop-admin update from version N-1)
  • Code is now on version N - next the updater starts a thread with version N's ./securedrop-admin setup
  • if the thread completes successfully, start a thread with version N's ./securedrop-admin tailsconfig

To support a migration to the package-based install, ./securedrop-admin setup can instead test for the package being installed and trigger a migration if it's not present, or just exit if it is. Then ./securedrop-admin tailsconfig can just trigger the package-installed tailsconfig command instead.

This may be complicated by the GUI updater being removed while running in the initial update - if it does break when tested, we can restore the updater code for one final release, just not call it from the new network hook, and remove it in a subsequent release once all migrations are complete.

[legoktm]

This may be complicated by the GUI updater being removed while running in the initial update

Are we actively removing the updater-via-Git? Any reason we can't leave ~/Persistent/securedrop in place, with the init hook disabled?

[zenmonkeykstop]

It's removed in the current PR - we could reinstate it.

[micahflee]

@zenmonkeykstop I've spent some time digging into the GUI updater and I think your plan will work. I'm planning out how to develop this feature.

The migration script

The GUI updater executes the ./securedrop-admin script from the root of the git repo, and that file is not used in /usr/bin/securedrop-admin, so we can rewrite it to just handle the migration.

I need to update ./securedrop-admin to:

• make it so ./securedrop-admin setup instead executes a migration script (below)
• make it so ./securedrop-admin tailsconfig instead executes a new custom tailsconfig script (below)
• this script doesn't need to do anything else, since all future securedrop-admin tasks will be executed from /usr/bin/securedrop-admin

The migration script itself needs to do the following:

• add the FPF apt repo, doing what's described in the Dangerzone Tails instructions, which is all of the following as root:
  • make sure /live/persistence/TailsData_unlocked/persistence.conf has the line: /etc/apt/sources.list.d source=apt/sources,link
  • mkdir -p /live/persistence/TailsData_unlocked/apt/sources
  • ln -s /usr/share/tails/apt/dangerzone.sources /live/persistence/TailsData_unlocked/apt/sources/ (this should work, because dangerzone.sources is packages.freedom.press/apt-tools-prod)
• install the securedrop-admin package from the apt repo
• make sure this package is set to install every time
• copy relevant config files from ~/Persistent/securedrop/install_files/ansible-base/ into ~/Persistent/.securedrop-admin/
  • site-specific (from install_files/ansible-base/group_vars/all/site-specific)
  • tor_v3_keys.json
  • app-journalist.auth_private
  • app-ssh.auth_private
  • mon-ssh.auth_private
  • app-sourcev3-ths
• delete ~/Persistent/.securedrop/securedrop_update.flag if it exists

It seems that, considering the type of work it needs to do, maybe the migration should script be in ansible. Though I could do it in bash too.

The new custom tailsconfig script will:

• run /usr/bin/securedrop-admin localconfig (from the installed package)
• pop up an alert box telling the user to reboot (after reboot, it should load the updated GNOME shell extension, the desktop shortcuts should use the new config paths, and the securedrop_init.py startup script should no longer try running the GUI updater)

With any luck, this will work while still deleting the GUI updater.

Setup SD servers and Tails environment for testing the migration

First, I need a git-based environment to test from, so I can make it like this:

• Set up Tails 7.1
• Create digitalocean droplets for app and mon
• From Tails, install SD to the droplets
• Make a backup of the persistent volume so I can restore and test again as often as I need

Testing the migration script

The tricky part is actually testing the migration script. I can't run the GUI updater in the test environment because it will try to checkout the latest git tag in the ./securedrop-admin update step. Instead, I can simulate the GUI updater by manually doing:

• checkout the admin-deb branch (to simulate ./securedrop-admin update)
• run ./securedrop-admin --force setup
• run ./securedrop-admin --force tailsconfig

The migration script (./securedrop-admin setup) needs to install the FPF apt repo, and then install the securedrop-admin package from it.

Does FPF have a dev repo we could use to create a dev securedrop-admin package in, and try that?

Alternatively, I could temporarily make the ./securedrop-admin setup command just install a local securedrop-admin.deb file to test the migration, without adding the repo and installing a package from it. This should work to test that the migration actually works. However, it would be good to test the logic of installing a new apt repo too, and I'm not sure how to test that without using a dev repo with a dev package in it.

And finally, if I mess up my Tails environment for testing, I can restore the persistent volume from the backup and try again as many times as I need.

[zenmonkeykstop]

Looks good, a few things:

• see https://github.com/freedomofpress/securedrop/blob/76d98c7f3ee43efc21d675aca7ec19fc634e43b2/tails-bootstrap.sh for the start of a bootstrap/migration script - I would agree that doing it in ansible is nicer but it might be good to stick with a shell script that could also work for from-scratch setups.
• SecureDrop has a separate repo to Dangerzone (apt.freedom.press for prod, apt-test.freedomp.press for testing) so there's a little more to do there than there would be if we could use dangerzone.sources
• when it comes to testing, i would patch the GUI updater to just pull admin-deb instead of the latest tag. The reason being, it would be good to see how it behaves as the update proceeds.

[micahflee]

Just to keep a note for myself, here's a patch for the GUI updater:

diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
index faac742df..4c9c9ac49 100755
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -762,7 +762,7 @@ def update_check_required(cmd_name: str) -> Callable[[_FuncT], _FuncT]:
         @functools.wraps(func)
         def wrapper(*args: Any, **kwargs: Any) -> Any:
             cli_args = args[0]
-            if cli_args.force:
+            if True:
                 sdlog.info("Skipping update check because --force argument was provided.")
                 return func(*args, **kwargs)
 
@@ -1028,6 +1028,10 @@ def get_release_key_from_keyserver(
 
 def update(args: argparse.Namespace) -> int:
     """Verify, and apply latest SecureDrop workstation update"""
+    subprocess.call(["git", "checkout", "."], cwd=args.root)
+    subprocess.call(["git", "checkout", "admin-deb"], cwd=args.root)
+    subprocess.call(["git", "pull", "origin", "admin-deb"], cwd=args.root)
+    return 0
     sdlog.info("Applying SecureDrop updates...")
 
     update_status, latest_tag = check_for_updates(args)
diff --git a/journalist_gui/journalist_gui/SecureDropUpdater.py b/journalist_gui/journalist_gui/SecureDropUpdater.py
index 75e801803..14882ccfb 100644
--- a/journalist_gui/journalist_gui/SecureDropUpdater.py
+++ b/journalist_gui/journalist_gui/SecureDropUpdater.py
@@ -88,6 +88,12 @@ class UpdateThread(QThread):
         self.failure_reason = ""
 
     def run(self):
+        subprocess.call(["git", "checkout", "."], cwd="/home/amnesia/Persistent/securedrop")
+        subprocess.call(["git", "checkout", "admin-deb"], cwd="/home/amnesia/Persistent/securedrop")
+        subprocess.call(["git", "pull", "origin", "admin-deb"], cwd="/home/amnesia/Persistent/securedrop")
+        self.update_success = True
+        self.signal.emit({"status": True, "output": "", "failure_reason": ""})
+        return
         sdadmin_path = "/home/amnesia/Persistent/securedrop/securedrop-admin"
         update_command = [sdadmin_path, "update"]
         try:

In Tails, save this as ~/Persistent/changes.diff. Then from ~/Persistent/securedrop:

git checkout 2.12.10
git apply ../changes.diff

Then run the GUI updater to test it out.