Add "New Issue" (error tracking) monitor for edx-edxapp-lms

[robrap]

Blocked on:

• Ensure Django errors are represented in Error Tracking #647

Acceptance Criteria:

• Create "New Issue" alert for code_owner_squad arch-bom for edx-edxapp-lms
• Create Safety Net "New Issue" alert (no code owner)
• Create Arch-BOM version of "New Issue" alert (code owner)
• Create or update runbooks
• Update how-tos and communicate as appropriate

Notes:

• See https://app.datadoghq.com/monitors/create/error-tracking, "New Issue" alert.

[robrap]

@robrap: TODO:

• Add datadog wiki links
• Add link to alerting integration changelog

[robrap]

For code owner:

@code_owner:(arch-bom OR -*)

For filtering to lms:

env:(prod OR edge) service:edx-edxapp-lms 

For filtering to django requests (which we may or may not want for this monitor:

operation_name:django.request 

[robrap]

[UPDATE]: I've edited this comment after I first wrote it.

@srathod-apphelix: Here are some assorted comments:

• These alerts are noisy in Slack: #alert-arch-bom.
• I temporarily muted them. You can un-mute when you are working on them.
• You named the alerts with "Django Exceptions", but I don't think we are filtering (nor want to) for Django (yet), so these should be for any errors/issues.
• In terms of name, DD call these "New Issues", but I think "New Errors/Issues" would help this be more clear.
• We're going to want to filter out all of the following:
  • rest_framework.exceptions.PermissionDenied
  • rest_framework.exceptions.Throttled
  • rest_framework.exceptions.NotAuthenticated
  • rest_framework.exceptions.AuthenticationFailed
  • Note: it may be we choose to ignore some status codes (401, 403, etc.) rather than, or in addition to, filtering out specific errors that we don't care about.
• Also, this doc may be helpful for DD in general: https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/1008042104/How+to+use+Datadog+with+edX+services

[robrap]

@srathod-apphelix: Here are some additional notes:

1. I marked as Resolved in DD, and got a ton of Recovered alerts in the Slack channel. like this one: https://twou.slack.com/archives/C04A10SLPFZ/p1748543687367449
2. Here is one of the resolved errors that are not showing the error information in the alert output: https://app.datadoghq.com/error-tracking?query=...
3. I had no idea how noisy this would be. If and when we experiment with this again, instead of the Slack channel, you should either not use notification (and just use the DD UI), or notify your email so you can filter it into a folder, until we have it well tuned.

[srathod-apphelix]

Yes @robrap. I saw all the recovered alerts too. I will keep in mind to use email or just the DD UI next time until it's better tuned. Thanks for the suggestion!