Install Raspberry Pi production monitor timer

This commit is contained in:
Курнат Андрей
2026-06-09 21:26:17 +03:00
parent ccd20d76e7
commit 4bf54f7fda
8 changed files with 213 additions and 7 deletions
+44 -6
View File
@@ -168,14 +168,52 @@ internal-only interface and set `PROXY_DEBUG_TOKEN`.
## Monitoring And Alerts
Minimum checks for the Raspberry Pi deployment:
Install the production checker timer from the `Server` directory:
- Run `./scripts/verify-rpi-production.sh` from the `Server` directory by cron
or systemd timer every 5 minutes with `PUBLIC_STATUS_URL` set to the public
QSfera URL.
- Alert when the script exits non-zero.
- Alert on any `WARN:` line that persists for more than one day, especially
```bash
sudo ./scripts/install-rpi-production-monitoring.sh
```
The installer writes:
- `/etc/systemd/system/qsfera-rpi-production-check.service`
- `/etc/systemd/system/qsfera-rpi-production-check.timer`
- `/etc/systemd/system/qsfera-rpi-production-check-alert@.service`
- `/etc/qsfera/rpi-production-check.env`
The timer runs `./scripts/verify-rpi-production.sh` every 5 minutes. The service
uses `OnFailure=qsfera-rpi-production-check-alert@%n.service`; the alert service
logs the failed unit and recent journal output through `systemd-cat`. To forward
alerts off-host, set `QSFERA_ALERT_WEBHOOK_URL` in
`/etc/qsfera/rpi-production-check.env`; `alert-rpi-production-check-failure.sh`
sends a JSON `POST` when that variable is set.
For the current Raspberry Pi deployment checked on 2026-06-09, the timer was
installed under `/etc/systemd/system`, `systemctl is-enabled` returned
`enabled`, `systemctl is-active` returned `active`, and the checker service
reported `Result=success`, `ExecMainStatus=0`, and `ActiveState=inactive` after
the initial run. `systemctl list-timers qsfera-rpi-production-check.timer`
reported the next run 5 minutes after the previous run.
Verify the timer:
```bash
systemctl list-timers qsfera-rpi-production-check.timer --no-pager
systemctl status qsfera-rpi-production-check.timer --no-pager
systemctl show qsfera-rpi-production-check.service -p Result -p ExecMainStatus -p ActiveState
journalctl -u qsfera-rpi-production-check.service -n 80 --no-pager
```
Minimum monitoring expectations:
- Keep `qsfera-rpi-production-check.timer` enabled and active.
- Investigate any non-zero `ExecMainStatus` from
`qsfera-rpi-production-check.service`.
- Investigate any `WARN:` line that persists for more than one day, especially
Docker root outside `/mnt/data` or unavailable metrics.
- Configure `QSFERA_ALERT_WEBHOOK_URL` when off-host alert delivery is required;
without it, failures are still routed to the systemd failure service and
journal, but not to an external notification endpoint.
- Scrape `http://127.0.0.1:9205/metrics` locally or through a private tunnel.
- Keep public HTTPS probing separate from local loopback probing so reverse-proxy
and backend failures are distinguishable.