[FEATURE] Synthetics and Monitors

[synhershko]

This is a request for creating a Monitors section under Observability which allows to monitor status and health for services, certificates, http endpoints, hosts (via tcp / ping) and anything else that'd fall into the defined schema of "Up" and "Down" statuses.

It should also be possible to define Monitors not only on boolean up / down statuses but also on an avg latency.

Synthetics are the source of most (if not all) of data that Monitors would monitor.

For this feature to be developed efficiently, I propose that we focus on ingestion and display of existing synthetics monitoring agents (such as heartbeat) instead of developing one from scratch. Synthetics ingestion should be created in such a way to allow for this flexibility, mapping any synthetic to a up / down status, and an optional numeric value that a monitor can attach to and create a threshold.

There needs to be Alerting integration as well, but it's likely out of scope for this feature request.

So the steps required for this feature -

1. Define a list of initial synthetics monitoring agents to support. heartbeat would be the first.
2. Given the above list, settle on a schema that would serve opensearch-synthetics.
3. Define an ingestion architecture. I propose a dedicated ingestion pipeline for synthetics which can take an indexing input with a source hint, and translate to the defined schema which essentially should always have up/down (or fail/success), @timestamp and a metric (to represent "time took" for example in http sampling, "time since", "time to" for example in cert expiration test, etc)
4. Monitors UI where monitors would be defined and displayed. Monitor definitions would be saved in an .observability-monitors index.
5. Follow up with spec'ing Alerting ingegration.

[ps48]

Thanks for the feature proposal. This would really be a great addition OpenSearch Observability users. I have some comments/suggestions below:

1. Define a list of initial synthetics monitoring agents to support. heartbeat would be the first.

I agree using the existing Open-Source agents should be a good approach. Apart from Heartbeat, I would nominate OTEL HTTP reciever and K6. It would be interesting to see how many of these agents can connect to/export data to OpenSearch today.

3. Define an ingestion architecture.

Let's make use of ingestion processors in pipelines to transform this to our common schema. We do have a lot of processors including script to minor key-value transformations.

4. Monitors UI where monitors would be defined and displayed. Monitor definitions would be saved in an .observability-monitors index.

Can we store these in OpenSearch Dashboards' Kibana index. Currently we have moved most of the dashboards-observability objects to .kibana index from our .opensearch-observability index. This helps us to make sure OpenSearch Dashboards metadata remains in a single location.

Misc. Future scope items:

1. When designing this can we touch-base on how to extend this later to workflow and UI testing may be with playwright or other test frameworks.
2. Along with alerting may be we can also check integration with Anomaly detection in Synthetics.