Operator-grade Monad observability with Prometheus, Grafana, and custom health checks.

This guide reflects the monitoring pattern Prooflines publishes and reuses for Monad operator environments: Prometheus, node-exporter, custom textfile metrics, Monad OTEL telemetry, Grafana dashboards, and Telegram delivery through Grafana Alerting.

What this guide installs

• prometheus for scraping and rule evaluation
• prometheus-node-exporter for host metrics and textfile integration
• otelcol metrics ingestion from Monad telemetry
• grafana for dashboards and alert routing
• Custom exporter script for Monad health and listener-state metrics
• Telegram alert delivery through Grafana Alerting

Prerequisites

• Running Monad node with reachable local RPC on 127.0.0.1:8080
• otelcol already exposing metrics on 127.0.0.1:8889/metrics
• Ubuntu access with sudo
• jq, curl, and ss available on the host

1. Install Prometheus and node exporter

sudo apt-get update
  sudo apt-get install -y prometheus prometheus-node-exporter jq curl
  sudo sed -i 's/^ARGS=.*/ARGS="--web.listen-address=127.0.0.1:9100 --collector.textfile.directory=\\/var\\/lib\\/prometheus\\/node-exporter-textfile"/' /etc/default/prometheus-node-exporter
  sudo install -d -o prometheus -g prometheus -m 755 /var/lib/prometheus/node-exporter-textfile
  sudo systemctl restart prometheus-node-exporter

2. Configure Prometheus scrape jobs

Save the following to /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  rule_files:
  - /etc/prometheus/rules/*.yml
  scrape_configs:
  - job_name: prometheus
  static_configs:
  - targets: ['127.0.0.1:9090']
  - job_name: node_exporter
  static_configs:
  - targets: ['127.0.0.1:9100']
  - job_name: monad_otel
  metrics_path: /metrics
  static_configs:
  - targets: ['127.0.0.1:8889']

sudo promtool check config /etc/prometheus/prometheus.yml
  sudo systemctl restart prometheus
  curl -s http://127.0.0.1:9090/api/v1/targets | jq

3. Install the custom Monad exporter

Download the example script from the downloads list on the hub, then install it:

sudo install -m 755 monad-export-metrics-example.sh /usr/local/bin/monad-export-metrics
  sudo tee /etc/systemd/system/monad-health-exporter.service >/dev/null <<'EOF'
  [Unit]
  Description=Export Monad custom health metrics for Prometheus textfile collector
  [Service]
  Type=oneshot
  ExecStart=/usr/local/bin/monad-export-metrics
  EOF
  sudo tee /etc/systemd/system/monad-health-exporter.timer >/dev/null <<'EOF'
  [Unit]
  Description=Run Monad health exporter every minute
  [Timer]
  OnBootSec=30s
  OnUnitActiveSec=60s
  Unit=monad-health-exporter.service
  [Install]
  WantedBy=timers.target
  EOF
  sudo systemctl daemon-reload
  sudo systemctl enable --now monad-health-exporter.timer
  sudo systemctl start monad-health-exporter.service

4. Install Prometheus alert rules

sudo install -d -m 755 /etc/prometheus/rules
  sudo install -m 644 monad-alerts-example.yml /etc/prometheus/rules/monad-alerts.yml
  sudo promtool check config /etc/prometheus/prometheus.yml
  sudo systemctl restart prometheus

At minimum, alert on:

monad_service_up{service="monad-bft"} == 0
  monad_service_up{service="monad-execution"} == 0
  monad_service_up{service="monad-rpc"} == 0
  monad_rpc_up == 0
  monad_rpc_syncing == 1
  monad_ssh_listener_up == 0
  monad_p2p_tcp_listener_up == 0
  monad_p2p_udp_listener_up == 0
  monad_root_free_bytes < 21474836480

5. Install Grafana and route alerts to Telegram

sudo apt-get install -y apt-transport-https software-properties-common wget
  wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/grafana.gpg
  echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
  sudo apt-get update
  sudo apt-get install -y grafana
  sudo systemctl enable --now grafana-server

Then in Grafana:

• Add Prometheus data source pointing to http://127.0.0.1:9090
• Create a Telegram bot via @BotFather
• Extract your target chat.id using getUpdates
• Create a Telegram contact point in Grafana Alerting
• Route Monad alerts to that contact point

6. Verification commands

sudo systemctl status prometheus prometheus-node-exporter grafana-server --no-pager
  curl -s http://127.0.0.1:9090/api/v1/targets | jq
  curl -s http://127.0.0.1:9090/api/v1/rules | jq
  curl -s http://127.0.0.1:9090/api/v1/alerts | jq
  grep '^monad_' /var/lib/prometheus/node-exporter-textfile/monad.prom
  curl -s -H 'Content-Type: application/json' \
  --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  http://127.0.0.1:8080 | jq