Message send processing improvements

[ccostino]

With many of the immediate stability issues taken care of, we've uncovered some follow-on work that we need to dig into further. This is work that gets to the heart of the performance and stability issues we were seeing in the first place and will help the application scale to the level that it needs to for the future.

• refactor check_job_status #1507
• Review and improve/correct our Celery task retry logic #1391
• Investigate 'adaptive' mode for retries and improve concurrency of message sends from multiple services #1452
• We need a clean up task for eternal Pending messages #1520
• Gradually increase thread concurrency to 20 #1523
• Set the worker prefetch multiplier to 2 #1524
• fix db connection pool #1529
• Consider bulk_update_mappings or similar for delivery receipts #1494
• Investigate bulk_save_objects for notification inserts #1531

[ecayer]

Can I suggest that with the addition of time-slicing different jobs, we postpone the "Create way to stop an in-progress job" story, @ccostino?

[ecayer]

Needs to be its own epic:

• Consider bulk_update_mappings or similar for delivery receipts #1494
• Investigate bulk_save_objects for notification inserts #1531

UPDATE FROM @ccostino: These are already in flight, so moving them back up to be completed with the other work!

[ccostino]

Turns out the last couple of items here are already in flight - @terrazoon has been working on them!