The watcher api gets used in dynamic allocation here:
https://github.com/armadaproject/armada-spark/blob/master/src/main/scala/org/apache/spark/scheduler/cluster/armada/ArmadaEventWatcher.scala#L94-L100

But that code probably should be in the Armada client.

So the goal of this ticket is to move that code from Armada spark to the armada client.