Readiness + Startup probes #804
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #804 +/- ##
=======================================
Coverage 94.88% 94.88%
=======================================
Files 89 89
Lines 3816 3816
=======================================
Hits 3621 3621
Misses 195 195 ☔ View full report in Codecov by Sentry. |
| command: | ||
| - bash | ||
| - -c | ||
| - 'curl http://127.0.0.1:{{ .Values.meshweb.port }}/api/v1/ -H "Host: db.nycmesh.net" -s | grep meshin' |
There was a problem hiding this comment.
The grep is fine but knowing that it's we're meshin' confused my brain for a sec lol
| exec: | ||
| command: | ||
| - bash | ||
| - -c | ||
| - 'curl http://127.0.0.1:{{ .Values.meshweb.port }}/api/v1/ -H "Host: db.nycmesh.net" -s | grep meshin' | ||
| periodSeconds: 3 | ||
| initialDelaySeconds: 10 | ||
| timeoutSeconds: 3 | ||
| {{ end }} | ||
| {{ if eq .Values.meshweb.startup_probe "true" }} | ||
| startupProbe: | ||
| exec: | ||
| command: | ||
| - bash | ||
| - -c | ||
| - 'curl http://127.0.0.1:{{ .Values.meshweb.port }}/api/v1/ -H "Host: db.nycmesh.net" -s | grep meshin' | ||
| periodSeconds: 3 | ||
| initialDelaySeconds: 20 | ||
| timeoutSeconds: 3 | ||
| failureThreshold: 20 | ||
| {{ end }} |
There was a problem hiding this comment.
I don't think it's worth having redundant probes. We should consider what kind of probes to use in place of these. Maybe we should just check if we get anything back from nginx at all as the liveness probe, then make the readiness probe check we're meshin'?
Not sure about startupProbe
There was a problem hiding this comment.
Actually, I think I might be wrong to suggest that.
A common pattern for liveness probes is to use the same low-cost HTTP endpoint as for readiness probes, but with a higher failureThreshold. This ensures that the pod is observed as not-ready for some period of time before it is hard killed.
| exec: | ||
| command: | ||
| - bash | ||
| - -c | ||
| - 'curl http://127.0.0.1:{{ .Values.meshweb.port }}/api/v1/ -H "Host: db.nycmesh.net" -s | grep meshin' | ||
| periodSeconds: 3 | ||
| initialDelaySeconds: 10 | ||
| timeoutSeconds: 3 | ||
| {{ end }} | ||
| {{ if eq .Values.meshweb.startup_probe "true" }} | ||
| startupProbe: | ||
| exec: | ||
| command: | ||
| - bash | ||
| - -c | ||
| - 'curl http://127.0.0.1:{{ .Values.meshweb.port }}/api/v1/ -H "Host: db.nycmesh.net" -s | grep meshin' | ||
| periodSeconds: 3 | ||
| initialDelaySeconds: 20 | ||
| timeoutSeconds: 3 | ||
| failureThreshold: 20 | ||
| {{ end }} |
There was a problem hiding this comment.
Actually, I think I might be wrong to suggest that.
A common pattern for liveness probes is to use the same low-cost HTTP endpoint as for readiness probes, but with a higher failureThreshold. This ensures that the pod is observed as not-ready for some period of time before it is hard killed.
The liveness probe was not working because the element was duplicated with the second being empty. Re-enable it, make it work, and add readiness + startup probes to allow for slow startups, while pulling pods from the service and eventually restarting them if they are unresponsive.
Let's let this soak in dev for a while.