Improve efficiency of handling gocd alerts

[robrap]

It takes a while to understand what is wrong in GoCD. Following up on an alert requires VPN and finding failure log in GoCD, and then knowing how to search for the actual failure, depending on where it failed. It would be great if the alerts had more context.

AC:

Timeboxed effort -- 1 day.

• Some useful extract of the logs shows up in the Opsgenie alert (so that we can tell if it's a known/unknown issue, etc.)

Questions/Notes:

• This work would only help in situations where you don't have to go on GoCD to re-run a stage anyhow (e.g. self-closing alerts).
• We want to switch to ArgoCD & Kubernetes relatively soon; are there quick improvements to get more context in alerts, or are there improvements that would carry over?
• Could we get the error details into the alert so VPN and GoCD login isn’t required?
• The Runbook has some notes that can be referenced (or added to) for searching to find errors in logs of various stages.

[dianakhuang]

From what I understand, the best we can do is try to pull the GoCD logs from within a Python script in order to extract the failure information, I'm not sure how easy this will be to implement, and the testing for it would be a significant overhead in terms of doing this. I don't think it will be possible to fit this work into the timebox, and I'm not sure it's worth it.

[robrap]

@dianakhuang: Is there any way for the script that is failing to gather this info and put it somewhere (environment variable, other) that can be found and used elsewhere in the pipeline?

[dianakhuang]

Considering how the GoCD pipelines are configured and the many and varied scripts that are called from it, I don't think there's any way to centralize this behavior and make it consistent.

[robrap]

Sure. But is there a way to do it at all? Maybe once we can do it, it wouldn't take much to add it as-needed so that we can quickly see if we've hit one of the common/known issues vs not knowing what we hit. The alternative is what we have now, where every alert requires looking in to GoCD, even to see if it is a known issue.