activity cache replication

If there is a communications fault for the master redis instance, normally activity-cache0, haproxy will automatically start using the slave, activity-cache1. However when the redis master re-joins the cluster haproxy does not switch back to the master.

This procedure should reset the slave-master setup to normal, activity-cache0=master (now called 0) and activity-cache1=slave (now called 1).

Obviously check the for the private IPs of activity-cache{0,1} in case they've changed for some reason.

1. first swap master/slave relationship;
   

   on 1...
   

   redis-cli slaveof no one
   

   on 0...
   

   redis-cli slaveof 10.224.14.104 6379
   redis-cli sync
   
2. wait until replication is normalized
   

   on 0
   

   redis-cli info | grep master_
   

   if master_sync_in_progress=0 then no sync is in progress
   

   if master_last_io_seconds_ago>0 but only a few seconds then a sync has been done recently
   

   A better definitive check would be nice
   

3. swap master/slave back again
   

   on 0...
   

   redis-cli slaveof no one
   

   on 1...
   

   redis-cli slaveof 10.224.14.103 6379
   redis-cli sync
   
4. reload haproxy
   

   on proxy0...
   

   oldpid=$(ps -A | awk '/redis/ { print $1 }')
   service haproxy reload
   kill $oldpid
   

Checking the activity cache is easy, log in and upload a doc and it should appear in your activity stream.