Application Health Monitoring

We are going to collect metrics about the usage of OBS, such as logins of users, creation of packages and projects and alike. Since we are planning to refresh the OBS UI and might change and improve workflows in OBS, we want to be able to track if that has any negative, or positive, effects.

Architectural overview

Our AHM stack consists of:

RabbitMQ

Metrics we collect are sent to RabbitMQ to the metrics queue.

Telegraf

Telegraf fetches these metrics and reports them to InfluxDB.

InfluxDB

InfluxDB stores the time series data we collect (database telegraf).

Grafana

Grafana is used to create graphs to visualize the collected data.

Development setup

Instructions for setting up the development environment for AHM can be found in our docker documentation

Prepare and start the container

rake docker:ahm:prepare
docker-compose -f docker-compose.ahm.yml -f docker-compose.yml up

Configure Grafana

Go to Grafana frontend, http://localhost:8000, and login (admin/admin).

Add a new data source by adding following data:

Type: InfluxDB
URL: http://influx:8086
Database: telegraf
User: grafana
Password: grafana

Production setup

The Grafana dashboards are hosted at https://obs-measure.opensuse.org/. You can login with your GitHub account and should get the Editor role. The openSUSE RabbitMQ is running at https://rabbit.opensuse.org/.

Main Dashboards

Application Health Overview

Located here, this dashboard should tell if the application is up or not.

• Number of successful requests per minute. :warning: It will send an alert when the traffic is too low.
• Error rates tracks requests with an http status error code.
• Authentication Failures monitor burst of authentication failures within 10 minutes.
• Request State Change tracks request creation and request state changes.
• Projects / min tracks projects destroyed and created within a minute.
• Packages / min tracks packages destroyed and created.
• Total project tracks the total amount of projects that were created and destroyed.
• Total package tracks the total amount of packages that were created and destroyed.
• User Creation tracks the total amount of users that were created within an hour.
• Beta Users tracks the total amount of users who joined and left the beta program.

Main error panel

Perf overview

response time

health

Alerts

There is an alert for the number of successful requests per minute that will trigger when the total number of queries drops below 5 per minute during two full minutes.