HostManager leaves containers from old services

[BrianJKoopman]

Probably more of a feature request than a bug, but we should add the --remove-orphans flag to the HostManager's docker-compose command. This'll remove old containers automatically in the rare instances that a service is renamed.

I think this just needs to be added here:

ocs/ocs/agents/host_manager/drivers.py

Lines 325 to 330 in da25bd8

	def up(self):
	self.d = _run_docker_compose(
	['-f', self.service['compose_file'],
	'up', '-d', self.service['service']],
	docker_compose_bin=self.docker_compose_bin)
	self.status = None, time.time()

For some context, we ran into this on site twice somewhat recently, once we were careful to bring down the service before deploying the new one and bringing the new agent up, the other we weren't as careful, and the old copy of the Agent prevented the new one from working properly.

[BrianJKoopman]

I tried briefly to add this today, but it wasn't behaving as I expected. The Host Manager loses track of the renamed/removed compose service and then enters a [?] state. The --remove-orphans flag doesn't seem to get a chance to run:

2024-04-25T16:30:59-0400 startup-op: launching manager
2024-04-25T16:30:59-0400 start called for manager
2024-04-25T16:30:59-0400 manager:0 Status is now "starting".
2024-04-25T16:30:59-0400 manager:0 Status is now "running".
2024-04-25T16:30:59-0400 Installed OCS Plugins: ['socs', 'ocs']
2024-04-25T16:30:59-0400 manager:0 Successfully parsed /home/koopman/git/ocs-dev-configs/hostmanager-local-test/docker-compose.yml and its service states.
2024-04-25T16:31:00-0400 manager:0 Requested launch for FakeDataAgent:fake-data1
2024-04-25T16:31:00-0400 manager:0 Requested launch for FakeDataAgent:fake-data2
2024-04-25T16:31:01-0400 manager:0 Launched FakeDataAgent:fake-data1
2024-04-25T16:31:01-0400 manager:0 Launched FakeDataAgent:fake-data2
2024-04-25T16:31:59-0400 manager:0 Failed to interpret /home/koopman/git/ocs-dev-configs/hostmanager-local-test/docker-compose.yml and/or its service states: Consistency problem: image does not self-report as a listed service? (ocs-fake-data2)
2024-04-25T16:31:59-0400 manager:0 Marking missing service for fake-data1
2024-04-25T16:31:59-0400 manager:0 Marking missing service for fake-data2
2024-04-25T16:32:07-0400 start called for update
2024-04-25T16:32:07-0400 update:1 Status is now "starting".
2024-04-25T16:32:07-0400 update:1 Status is now "running".
2024-04-25T16:32:07-0400 update:1 Retiring FakeDataAgent:fake-data2, which has disappeared from configuration file(s) or has manage:no.
2024-04-25T16:32:07-0400 Installed OCS Plugins: ['socs', 'ocs']
2024-04-25T16:32:07-0400 update:1 Update requested.
2024-04-25T16:32:07-0400 update:1 Status is now "done".
2024-04-25T16:32:23-0400 start called for update

[BrianJKoopman]

I came to file this same bug report again. There was an instance of trying to rename a service on the LAT recently that when deployed caused the HM to enter into a '?' state for all existing agents. Running 'update' after renaming the service didn't help, despite it realizing it should "retire" that service:

Aug 25 20:51:57 daq-lat launcher-daq-hm-daq-lat.sh[362303]: 2025-08-25T20:51:57+0000 start called for update
Aug 25 20:51:57 daq-lat launcher-daq-hm-daq-lat.sh[362303]: 2025-08-25T20:51:57+0000 update:46 Status is now "starting".
Aug 25 20:51:57 daq-lat launcher-daq-hm-daq-lat.sh[362303]: 2025-08-25T20:51:57+0000 update:46 Status is now "running".
Aug 25 20:51:57 daq-lat launcher-daq-hm-daq-lat.sh[362303]: 2025-08-25T20:51:57+0000 update:46 Retiring LabJackAgent:env-labjack-inclinometer, which has disappeared from configuration file(s) or has manage:no.
Aug 25 20:51:57 daq-lat launcher-daq-hm-daq-lat.sh[362303]: 2025-08-25T20:51:57+0000 Installed OCS Plugins: ['socs', 'ocs']
Aug 25 20:51:58 daq-lat launcher-daq-hm-daq-lat.sh[362303]: 2025-08-25T20:51:58+0000 update:46 Update requested.
Aug 25 20:51:58 daq-lat launcher-daq-hm-daq-lat.sh[362303]: 2025-08-25T20:51:58+0000 update:46 Status is now "done".

The solution was to stop the running orphaned container. I don't have an exact set of steps to reproduce in a test environment yet.