a plain operator account from Japan.
About a month running a Monad testnet full node as a validator candidate. Plain operator report: what happened, what broke, what we did about it, what we shipped. Operator-honest accounts of month one are useful for the community even when nothing dramatic happens.
full_proofline-jp-01
Monad testnet, chain id 10143
SECP 03187d85...41966b7
Public surfaces: Monad Sentinel, VDP Tracker, MonadBeaconBot.
/dev/nvme1n1p1) exposed as /dev/triedb38.96.30.106:8000
The Japan deployment replaced an earlier Polish host (retired on 2026-04-17) so identity,
monitoring, and the public Sentinel surface continued cleanly across the move.
At the time of writing:
0.14.4in-sync, mode live~310 (typical range 200-350)1-2 steady state~1.1 TB / 3.5 TB69 GiB used / 3.3 TB total38 days, 14h~3 days 14h3% on both drives, 0 media errors, 0
critical warnings, available spare 100%00
Five health audits were produced during the month (2026-04-21, 2026-04-26,
2026-04-30, 2026-05-04, plus a 2026-05-07 v0.14.3 health pass).
Each one looked for unplanned reboots, crash loops, OOM, thermal, NVMe faults, journal error patterns,
and active alerts. Every audit came back clean except for planned upgrade restarts.
2026-04-23. Planned ORANGE release, applied via the
official Category Labs apt repository. Service interruption ~48-84 seconds across the three
core services. Node returned to healthy in-sync operation on the same preserved identity.2026-05-07). AuthUDP enforcement passed without issues because the firewall and OTel
pipeline were already configured for it.2026-05-21, immediately after the announcement. Test
of our own runbook: caught up from blockDifference 93 back to in-sync in
~70 seconds. Headline release item was the eth_fillTransaction gas estimation
fix.All three upgrades used the package path rather than manual binary replacement. The cadence settled into "regular operation" rather than "exception".
We had a small number of real operational issues during the month. None of them resulted in extended downtime, but they are worth naming.
After disabling SMT in BIOS on the Ryzen 9 7950X-class host, a post-reboot device-mapping path change
broke TrieDB access. We diagnosed the device-access permission issue, applied a persistent
root:monad + 0660 layout via a dedicated monad-triedb-prepare.service
unit, and wired the Monad services to wait for it via systemd drop-in ordering. The node recovered to
healthy in-sync operation on the same hardware.
set-hugepages.service ordering fix (2026-05-18)
A community operator (kamunagi) reported on 2026-05-17 that some Monad nodes
fail to start on every reboot because set-hugepages is not explicitly ordered before the
Monad services. We had not hit this because the Japan host has not been rebooted in over a month, but
the bug is reproducible by design. We extended the existing systemd drop-ins for monad-bft,
monad-execution, and monad-rpc to add After=set-hugepages.service
and Wants=set-hugepages.service, verified the dependency chain via systemctl show,
and kept all five relevant services active through the change. Future reboots are now ordered
correctly without waiting for the upstream package to ship the fix.
CVE-2026-31431 mitigation
Applied the official interim mitigation for the announced CVE by blacklisting algif_aead via
/etc/modprobe.d/disable-algif.conf, unloaded the module, and verified the blocked state.
After the Validator Bi-Weekly Sync, we ran the official otel-validator-setup.sh with
SHA-256 verification, replacing our prior local-only otelcol config with the Foundation config that
pushes to otel-external.monadinfra.com:443 and adds a secp_key resource
attribute. Local :8889 Prometheus endpoint preserved, so existing dashboards continue to
work. Foundation now sees Prooflines as compliant on the metrics-reporting requirement.
2026-04-30fail2ban active for
sshd, stricter PermitRootLogin, LoginGraceTime, and
MaxStartups values applied after observing brute-force noise in the journal during the
monthOperating the node was only half of the work. The same month was used to ship public, operator-honest surfaces:
lifecycle_status and funding_source columns, an explicit
VDP-2 (post-2026-02-02) cohort filter, a stake-aware Consensus
column, and a "VDP process under revision" warning above the fold so users do not over-trust current
Foundation workflow framing.~11M -> ~9M alongside ~9M -> ~11M), participation-state transition alerts,
and a stricter unreachable-alert threshold so short blips do not spam users.In the interest of operator-honest framing rather than self-promotion:
#267): the generic "Stake band" label made a snapshot-stake removal look like
total stake had fallen to zero, even though only the snapshot metric had dropped. We rewrote the
wording to explicitly say "Snapshot band" / "Current snapshot" and "Consensus band" /
"Current consensus".RotatingFileHandler permission issue caused rotation failures in the bot's log because
the bot ran as a non-root user but tried to rotate under /var/log. Moved the log directory
under the bot's own home; rotation now works.Month one on Monad testnet looked like this: one clean cutover, three clean upgrades, five clean health audits, three small operational items handled without extended downtime, one pre-emptive fix from community signal, validator metrics push brought into Foundation-compliant shape, and a public surface stack that did not exist on day one. No incidents, no slashing (non-applicable on testnet), no surprise reboots.
We will keep posting these.