Application Health Monitoring
We collect metrics about the usage of OBS, such as logins of users, creation of packages and projects and alike.
The monitoring dashboards are hosted at https://obs-measure.opensuse.org/. You can login with your GitHub account and should get the Editor role.
The openSUSE RabbitMQ is running at https://rabbit.opensuse.org/.
Two dashboards are particularly important: The Application Health Overview and the Detailed Errors Dashboard
This dashboard gives a general overview about the health status of the application. You could say if the application is up or not by looking at the following panels:
This panel tracks the application traffic from the application's point of view.
This panel tracks requests with an http status error code.
If the number of errors gets too high, it means something is happening. Our exception tracker collects some of them.
This panel monitor burst of authentication failures within 10 minutes.
This panel tracks request creation and request state changes.
This panel tracks projects destroyed and created within a minute.
This panel tracks packages destroyed and created.
This panel tracks the total amount of projects that were created and destroyed.
This panel tracks the total amount of packages that were created and destroyed.
This panel tracks the total amount of users that were created within an hour.
This panel tracks the total amount of users who joined and left the beta program.
This dashboard gives a detailed picture of the errors happening in the application. Each type of error has its own panel.
This panel tracks 500 Server errors, translated as Unhandled Exceptions thrown by the application.
Exhaustive details for each exception can be found in our exception tracker
- 400 (Bad Request).
- 401 (Unauthorized)
- 403 (Forbidden)
- 404 (Not found) / min
- 408 (Request Timeout)
- 422 (Unprocessable Entity)
Our AHM stack consists of:
Metrics we collect are sent to RabbitMQ to the metrics queue.
Some of those metrics are:
- Login information, ranging from user creation or user deletion to authentication errors or even users entering or exiting our public beta program.
- Status message acknowledging
- Several events, like package updates, branching performed or reviews wanted.
Telegraf fetches these metrics using the amqp_consumer input plugin and reports them to InfluxDB using the influxdb output plugin.
InfluxDB stores the time series data we collect (database telegraf).
Grafana is used to create graphs to visualize the collected data.
Instructions for setting up the development environment including application health monitoring can be found on Site-Reliability#development-environment