Option for post-deploy sleep delay to stagger tservers competing with one another for DDL version locks (#268)

* post-deploy jitter on tservers

* fixup post-deploy-delay arithmatic

* increase delay factor to 10, which might be too high, but it should help

Author: aegershman

jobs/yb-tserver/spec

@@ -66,6 +66,24 @@ properties:
       until then, the default will be the recommended 60 seconds.
       see also: https://docs.yugabyte.com/latest/manage/upgrade-deployment
     default: 60
+  post_deploy_delay_factor:
+    description: |
+      must be a whole number.
+      consider removing this property altogether a TODO.
+      multiplication factor used in the post-deploy hook to calculate how long each node should
+      sleep before attempting to run itself. this number is multipled by the node index to
+      calculate how long, in seconds, to sleep.
+      for example, given a delay factor of 5, on node index 0,
+      the node will sleep 0 seconds before running its post-deploy. meanwhile on node index 1,
+      the node will sleep for 5 seconds before running its post-deploy. and on node index 2, the
+      node will sleep for 10 seconds before running its post-deploy.
+      and for example, given a delay factor of 8, node index 0 sleeps for 0 seconds, index 1
+      sleeps for 8 seconds, index 2 sleeps for 16 seconds, and so on.
+      this is not the most incredible solution in the world, but it is a relatively okay
+      way to prevent nodes from stumbling over each other.
+      in the future it may be preferrable to have only a subset of nodes try to perform
+      post-deploy at all, but this should be an acceptable solution for the time being.
+    default: 10
 
   tls.allow_insecure_connections:
     description: |

jobs/yb-tserver/templates/bin/post-deploy.erb.sh

@@ -2,10 +2,20 @@
 
 set -eu
 
-echo "running post-deploy..."
+echo "$(date --rfc-3339=seconds) entering post-deploy..."
 
 source /var/vcap/packages/python*/bosh/runtime.env
 
+INDEX=<%= spec.index %>
+POST_DEPLOY_DELAY_FACTOR=<%= p('post_deploy_delay_factor') %>
+POST_DEPLOY_DELAY_SECONDS="$((${POST_DEPLOY_DELAY_FACTOR} * ${INDEX}))"
+
+echo "$(date --rfc-3339=seconds) post-deploy on node index ${INDEX} sleeping for ${POST_DEPLOY_DELAY_SECONDS} seconds before beginning post-deploy..."
+sleep "${POST_DEPLOY_DELAY_SECONDS}"
+echo "$(date --rfc-3339=seconds) post-deploy delay complete, running..."
+
+####################################
+
 echo "running post-deploy ycql rotate admin password check..."
 /var/vcap/jobs/yb-tserver/bin/ycql-rotate-default-admin-password.sh