Observability

Setting up Prometheus and Grafana on an EC2 instance with Ubuntu involves installing both components, configuring Prometheus to collect metrics, and connecting Grafana to Prometheus as a data source. Here's a step-by-step guide:

Install Basic Tooling

apt-get update
apt-get upgrade
apt-get install software-properties-common tar unzip curl wget libfontconfig1 musl supervisor

Install Prometheus

Make sure the packages are updated and you have latest updated list of repositories.

sudo apt-get install prometheus

Create a Systemd service file for Prometheus to ensure it runs as a background service

sudo nano /etc/systemd/system/prometheus.service

Add Following configuration to the service file

[Unit]
Description=Prometheus monitoring system
After=network.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/bin/prometheus
Restart=always

[Install]
WantedBy=multi-user.target

Enable and start Prometheus as a service

sudo systemctl enable prometheus
sudo systemctl start prometheus

Go to official Prometheus site to learn more about it https://prometheus.io/

Node Exporter https://github.com/prometheus/node_exporter

Node Exporter is a Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors.

Install Node Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvfz node_exporter-1.7.0.linux-amd64.tar.gz
cd node_exporter-1.7.0.linux-amd64
cp node_exporter /usr/local/bin/node_exporter

Configure Node Exporter port by editing the following file, At the moment i am using PORT 9100

sudo nano /etc/default/node-exporter

Add the following lines to the file

NODE_EXPORTER_OPTS="--web.listen-address=:9100"

Restart Node exporter

sudo systemctl restart node-exporter

Collect Matrixes

Add your first configuration to Prometheus to collect matrixes. You can add multiple matrixes from different sources.

sudo nano /etc/prometheus/prometheus.yml

Add the following scrap configuration to the file

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']

Save the configuration and restart the Prometheus server

sudo systemctl restart prometheus

Grafana

All those matrixes are useless if we can visualize them. Grafana is a multi-platform open source analytics and interactive visualization web application which can help use to put meaning to the records we were adding to our server.

To Install Grafana

sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo curl https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafana

Alternate installation:

wget https://dl.grafana.com/oss/release/grafana_10.2.2_amd64.deb
dpkg -i  grafana_10.2.2_amd64.deb

Grafana have the capability to connect with multiple sources and from those sources we can build a dashboard to visualize information we need to make decisions.

Here are the Steps:

• Access the Grafana web interface at http://localhost:3000
• In Grafana, go to Configuration > Data Sources and click the "Add data source" button.
• Select Prometheus as the data source type and enter the following details:
• Provide name, Type, URL, and Access Type
• Click the "Add data source" button.

Grafana is now connected to Prometheus and can visualize the collected metrics. You can create dashboards to monitor the health and performance of your system.

Loki For Grafana

Loki with Promtail is used to collect deferent kind of logs which later can be used for visualization and recording system logs

Install Promtail

download the zip archive of the latest release

curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.1/promtail-linux-amd64.zip"
unzip "promtail-linux-amd64.zip"
chmod a+x "promtail-linux-amd64"
cp promtail-linux-amd64 /usr/local/bin/promtail

Configure Promtail to scrap nginx logs

#/etc/promtail/promtail-config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /etc/promtail/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: nginx
  static_configs:
  - targets:
      - localhost
    labels:
      job: nginx
      hostname: container
      __path__: /var/log/nginx/*.log

Start Promtail

Create supervisor service file by using following content to run the promtail

#/etc/supervisor/conf.d/promtail.conf

[program:promtail-worker]
command=/bin/bash -c "promtail --config.file=/etc/promtail/promtail-config.yaml"
autostart=true
autorestart=true
stderr_logfile=/var/log/promtail.err.log
stdout_logfile=/var/log/promtail.out.log

Install Loki For Grafana

Start by downloading zip archive of the latest release

curl -O -L "https://github.com/grafana/loki/releases/download/v2.8.6/loki-linux-amd64.zip"
unzip "loki-linux-amd64.zip"
chmod a+x "loki-linux-amd64"
cp loki-linux-amd64 /usr/local/bin/loki

Start Loki

Create Supervisor service file by using following content to run Loki,

#/etc/supervisor/conf.d/loki.conf

[program:loki-worker]
command=/bin/bash -c "loki --config.file=/etc/loki/loki.yaml"
autostart=true
autorestart=true
stderr_logfile=/var/log/loki.err.log
stdout_logfile=/var/log/loki.out.log

Note: instead of using supervisor you can also use service worker to run Promtail and Loki. Here is an example which can be used to create service for Loki as well

# /etc/systemd/system/promtail.service

[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail/promtail-config.yaml
Restart=on-failure
RestartSec=20
StandardOutput=append:/etc/promtail/logs/promtail.log
StandardError=append:/etc/promtail/logs/promtail.log

[Install]
WantedBy=multi-user.target