PROOFLINE Monitoring Guide

Install Guide

Monitoring, Alerting, Telegram

Operator-grade Monad observability with Prometheus, Grafana, and custom health checks.

This guide reflects the monitoring baseline Proofline uses on its live Monad testnet node: Prometheus, node-exporter, custom textfile metrics, Monad OTEL telemetry, Grafana dashboards, and Telegram delivery through Grafana Alerting.

Prometheus node-exporter Grafana Telegram
Live Baseline Prometheus Grafana

What this guide installs

  • prometheus for scraping and rule evaluation
  • prometheus-node-exporter for host metrics and textfile integration
  • otelcol metrics ingestion from Monad telemetry
  • grafana for dashboards and alert routing
  • Custom exporter script for Monad health and listener-state metrics
  • Telegram alert delivery through Grafana Alerting

Prerequisites

  • Running Monad node with reachable local RPC on 127.0.0.1:8080
  • otelcol already exposing metrics on 127.0.0.1:8889/metrics
  • Ubuntu access with sudo
  • jq, curl, and ss available on the host

1. Install Prometheus and node exporter

sudo apt-get update
sudo apt-get install -y prometheus prometheus-node-exporter jq curl

sudo sed -i 's/^ARGS=.*/ARGS="--web.listen-address=127.0.0.1:9100 --collector.textfile.directory=\\/var\\/lib\\/prometheus\\/node-exporter-textfile"/' /etc/default/prometheus-node-exporter
sudo install -d -o prometheus -g prometheus -m 755 /var/lib/prometheus/node-exporter-textfile
sudo systemctl restart prometheus-node-exporter

2. Configure Prometheus scrape jobs

Save the following to /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: node_exporter
    static_configs:
      - targets: ['127.0.0.1:9100']

  - job_name: monad_otel
    metrics_path: /metrics
    static_configs:
      - targets: ['127.0.0.1:8889']
sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl restart prometheus
curl -s http://127.0.0.1:9090/api/v1/targets | jq

3. Install the custom Monad exporter

Download the example script from the downloads list on the hub, then install it:

sudo install -m 755 monad-export-metrics-example.sh /usr/local/bin/monad-export-metrics

sudo tee /etc/systemd/system/monad-health-exporter.service >/dev/null <<'EOF'
[Unit]
Description=Export Monad custom health metrics for Prometheus textfile collector

[Service]
Type=oneshot
ExecStart=/usr/local/bin/monad-export-metrics
EOF

sudo tee /etc/systemd/system/monad-health-exporter.timer >/dev/null <<'EOF'
[Unit]
Description=Run Monad health exporter every minute

[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
Unit=monad-health-exporter.service

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now monad-health-exporter.timer
sudo systemctl start monad-health-exporter.service

4. Install Prometheus alert rules

sudo install -d -m 755 /etc/prometheus/rules
sudo install -m 644 monad-alerts-example.yml /etc/prometheus/rules/monad-alerts.yml
sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl restart prometheus

At minimum, alert on:

monad_service_up{service="monad-bft"} == 0
monad_service_up{service="monad-execution"} == 0
monad_service_up{service="monad-rpc"} == 0
monad_rpc_up == 0
monad_rpc_syncing == 1
monad_ssh_listener_up == 0
monad_p2p_tcp_listener_up == 0
monad_p2p_udp_listener_up == 0
monad_root_free_bytes < 21474836480

5. Install Grafana and route alerts to Telegram

sudo apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/grafana.gpg
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana
sudo systemctl enable --now grafana-server

Then in Grafana:

  • Add Prometheus data source pointing to http://127.0.0.1:9090
  • Create a Telegram bot via @BotFather
  • Extract your target chat.id using getUpdates
  • Create a Telegram contact point in Grafana Alerting
  • Route Monad alerts to that contact point

6. Verification commands

sudo systemctl status prometheus prometheus-node-exporter grafana-server --no-pager
curl -s http://127.0.0.1:9090/api/v1/targets | jq
curl -s http://127.0.0.1:9090/api/v1/rules | jq
curl -s http://127.0.0.1:9090/api/v1/alerts | jq
grep '^monad_' /var/lib/prometheus/node-exporter-textfile/monad.prom
curl -s -H 'Content-Type: application/json' \
  --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  http://127.0.0.1:8080 | jq
Important limitation

Local alert rules are strong for service health and listener validation, but they do not prove full outside-in reachability. A second off-host probe should eventually be added for complete path monitoring.

Back to Monad Hub.