Notes Prometheus

Voir :

https://linuxfr.org/news/decouverte-de-l-outil-de-supervision-prometheus

https://www.scaleway.com/en/docs/configure-prometheus-monitoring-with-grafana/

PromQL

https://enix.io/fr/blog/comment-creer-exporter-prometheus/

Voir aussi :

Prometheus VictoriaMetrics victoriadb

Alertmanager (as part of Prometheus)

https://developers.redhat.com/articles/2023/10/04/5-steps-build-self-healing-server-alertmanager

Voir Identity correlation / CorrelationId

https://en.wikipedia.org/wiki/Identity_correlation

https://opentelemetry.io/

https://grafana.com/oss/tempo/

Serveur

#docker run -p 9090:9090 -v ~/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus mkdir -p prometheus/nodes #sudo docker run -p 9090:9090 -v ~/prometheus:/etc/prometheus prom/prometheus podman run -p 9090:9090 -v ~/prometheus:/etc/prometheus docker.io/prom/prometheus

Config

prometheus.yml

# my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. # static_configs: # - targets: ['localhost:9090'] - job_name: 'node' file_sd_configs: - files: [ "/etc/prometheus/nodes/*.yml" ]

prometheus/nodes/vm1.yml

- targets: [ "172.17.0.1:9100" ] labels: # Vous pouvez ajouter ce que vous voulez pour taguer la machine host: "vm1"

Exporter (client)

Voir aussi :

check-mk-agent

Créer un compte node_exporter

/etc/systemd/system/node_exporter.service

[Unit] Description=node_exporter Wants=network-online.target After=network-online.target [Service] User=node_exporter Group=node_exporter Restart=on-failure Type=simple ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.ntp --collector.processes --collector.tcpstat [Install] WantedBy=multi-user.target

Grafana

Prometheus

Voir

docker run -d --name=grafana -p 3000:3000 grafana/grafana

Voir https://grafana.com/docs/grafana/latest/administration/configure-docker/

docker-compose.yml

version: '3.7' services: grafana: image: grafana/grafana ports: - 3000:3000 volume: - "$PWD/data:/var/lib/grafana"

Run Grafana container with persistent storage (recommended)

Create a persistent volume for your data in /var/lib/grafana (database and plugins)

docker volume create grafana-storage

Start grafana

docker run -d -p 3000:3000 --name=grafana -v grafana-storage:/var/lib/grafana grafana/grafana

Run Grafana container using bind mounts

You may want to run Grafana in Docker but use folders on your host for the database or configuration. When doing so, it becomes important to start the container with a user that is able to access and write to the folder you map into the container.

mkdir data # creates a folder for your data ID=$(id -u) # saves your user id in the ID variable # starts grafana with your user id and using the data folder #docker run -d --user $ID --volume "$PWD/data:/var/lib/grafana" -p 3000:3000 grafana/grafana:7.2.1 podman run --volume "$PWD/data:/var/lib/grafana" -p 3000:3000 docker.io/grafana/grafana:7.2.1