Skip to content

Commit 8d0c6cf

Browse files
committed
WIP: Set empirically determined Adaptive values
Based on some experiments on the cluster, it appears that it takes anywhere from 9s-15s to startup a single worker. So set the startup cost to the average value of 12s. Given it takes a bit to startup workers, bump the interval for rechecking whether to adjust workers to 6s (leaving the number of checks the same as the default of 3). Thus the time for a job to startup and get processing some data should occur a little after if not at least by the 2nd check, which avoid rapid adjustment of workers, while still allowing it to be frequent enough to notice if things need a slight tweak. This should slow down rapid downscaling and upscaling a bit hopefully avoiding fluctuations.
1 parent 14ee9e0 commit 8d0c6cf

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

nanshe_ipython.ipynb

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,8 @@
9191
"}\n",
9292
"client_kwargs = {}\n",
9393
"adaptive_kwargs = {\n",
94+
" \"startup_cost\": \"12s\",\n",
95+
" \"interval\": \"6s\",\n",
9496
" \"minimum\": 0,\n",
9597
" \"maximum\": int(os.environ.get(\"CORES\", cpu_count())) - 1\n",
9698
"}"

0 commit comments

Comments
 (0)