Install Raspberry Pi production monitor timer
This commit is contained in:
@@ -168,14 +168,52 @@ internal-only interface and set `PROXY_DEBUG_TOKEN`.
|
||||
|
||||
## Monitoring And Alerts
|
||||
|
||||
Minimum checks for the Raspberry Pi deployment:
|
||||
Install the production checker timer from the `Server` directory:
|
||||
|
||||
- Run `./scripts/verify-rpi-production.sh` from the `Server` directory by cron
|
||||
or systemd timer every 5 minutes with `PUBLIC_STATUS_URL` set to the public
|
||||
QSfera URL.
|
||||
- Alert when the script exits non-zero.
|
||||
- Alert on any `WARN:` line that persists for more than one day, especially
|
||||
```bash
|
||||
sudo ./scripts/install-rpi-production-monitoring.sh
|
||||
```
|
||||
|
||||
The installer writes:
|
||||
|
||||
- `/etc/systemd/system/qsfera-rpi-production-check.service`
|
||||
- `/etc/systemd/system/qsfera-rpi-production-check.timer`
|
||||
- `/etc/systemd/system/qsfera-rpi-production-check-alert@.service`
|
||||
- `/etc/qsfera/rpi-production-check.env`
|
||||
|
||||
The timer runs `./scripts/verify-rpi-production.sh` every 5 minutes. The service
|
||||
uses `OnFailure=qsfera-rpi-production-check-alert@%n.service`; the alert service
|
||||
logs the failed unit and recent journal output through `systemd-cat`. To forward
|
||||
alerts off-host, set `QSFERA_ALERT_WEBHOOK_URL` in
|
||||
`/etc/qsfera/rpi-production-check.env`; `alert-rpi-production-check-failure.sh`
|
||||
sends a JSON `POST` when that variable is set.
|
||||
|
||||
For the current Raspberry Pi deployment checked on 2026-06-09, the timer was
|
||||
installed under `/etc/systemd/system`, `systemctl is-enabled` returned
|
||||
`enabled`, `systemctl is-active` returned `active`, and the checker service
|
||||
reported `Result=success`, `ExecMainStatus=0`, and `ActiveState=inactive` after
|
||||
the initial run. `systemctl list-timers qsfera-rpi-production-check.timer`
|
||||
reported the next run 5 minutes after the previous run.
|
||||
|
||||
Verify the timer:
|
||||
|
||||
```bash
|
||||
systemctl list-timers qsfera-rpi-production-check.timer --no-pager
|
||||
systemctl status qsfera-rpi-production-check.timer --no-pager
|
||||
systemctl show qsfera-rpi-production-check.service -p Result -p ExecMainStatus -p ActiveState
|
||||
journalctl -u qsfera-rpi-production-check.service -n 80 --no-pager
|
||||
```
|
||||
|
||||
Minimum monitoring expectations:
|
||||
|
||||
- Keep `qsfera-rpi-production-check.timer` enabled and active.
|
||||
- Investigate any non-zero `ExecMainStatus` from
|
||||
`qsfera-rpi-production-check.service`.
|
||||
- Investigate any `WARN:` line that persists for more than one day, especially
|
||||
Docker root outside `/mnt/data` or unavailable metrics.
|
||||
- Configure `QSFERA_ALERT_WEBHOOK_URL` when off-host alert delivery is required;
|
||||
without it, failures are still routed to the systemd failure service and
|
||||
journal, but not to an external notification endpoint.
|
||||
- Scrape `http://127.0.0.1:9205/metrics` locally or through a private tunnel.
|
||||
- Keep public HTTPS probing separate from local loopback probing so reverse-proxy
|
||||
and backend failures are distinguishable.
|
||||
|
||||
Reference in New Issue
Block a user