eCHIS KE just went through a slew of upgrades and faced a number of issues. some of it was running out of disk space, some of it was API getting stuck and some of it was maybe something else!

@derickl said:

It would be great to get a nice summary of how we are intervening as part of an faq

So....Let's do it! We should clean up all the fixes that were used and write a guide on how to detect them and how to work around them.

we should research these comments and tickets, see if they're applicable and put relevant info in the new docs page:

• slack comment from @dianabarsan about 4.2 -> 4.11 upgrades

  I have a new workaround for the indexing stuck issue:

  1. When API goes stuck after view indexing, restart
  2. The admin upgrade page will say that the upgrade was interrupted, click retry upgrade.
  3. Depending on the state of the database, you might see view indexing again. Depending on how many docs need to be indexed, indexing might get stuck again. Go back to 1 if that happens.
  4. Eventually, when indexing jobs are short enough not to trigger a request hang, you will get the button to complete the upgrade.

• Starting an upgrade that involves view indexing can cause CouchDB to crash cht-core#9286
• Starting an upgrade that involves view indexing can become stuck after indexing is finished cht-core#9617
• slack comment from @kennsippell :

  how to unstick upgrades which are all stuck in the "indexing" state

  1. I'm checking that all the indexes are warmed by loading them one by one in fauxton.
  2. If all views load, then I'm going to medic-logs database and finding the latest upgrade_log
  3. I'm setting the state from indexing to indexed and creating a fake section in state_history

• https://github.com/moh-kenya/config-echis-2.0/issues/2578
• https://github.com/moh-kenya/config-echis-2.0/issues/2579

cc @eljhkrr and @1yuv too!