Install Raspberry Pi production monitor timer

2026-06-09 21:26:17 +03:00
parent ccd20d76e7
commit 4bf54f7fda
8 changed files with 213 additions and 7 deletions
@@ -41,8 +41,10 @@ Date: 2026-06-08.
 - Added `scripts/verify-production-state.ps1` as a repeatable live production endpoint gate for the public server status, Android Argus manifest/download, and Desktop update feed.
 - `scripts/verify-production-state.ps1 -VerifyAndroidDownload` passed on 2026-06-08: server `edition=stable`/`productversion=7.0.0`, Android `1.3.7`/`35`, downloaded APK size `12220848`, APK SHA-256 `c9c044c1332f8d973c64502ad2e3227ad92398101e7354ce9dbe7f708e7e693b`, and valid Desktop no-update feed.
 - Added `Server/scripts/verify-rpi-production.sh`, declared a loopback-only Pi metrics port in `Server/docker-compose.rpi5.yml`, and expanded the Raspberry Pi Docker runbook with local/public status checks, container/image checks, Prometheus metrics verification, backup/restore drill steps, and Docker data-root/retention planning.
+- Added systemd monitoring units for the Pi: `Server/systemd/qsfera-rpi-production-check.service`, `Server/systemd/qsfera-rpi-production-check.timer`, `Server/systemd/qsfera-rpi-production-check-alert@.service`, `Server/systemd/rpi-production-check.env.example`, plus installer `Server/scripts/install-rpi-production-monitoring.sh` and alert helper `Server/scripts/alert-rpi-production-check-failure.sh`.
 - Deployed the Pi metrics/checker update on 2026-06-09. Before upload, `/mnt/data/qsfera/cloud-server/src/docker-compose.rpi5.yml` was backed up to `/mnt/data/qsfera/backups/docker-compose.rpi5.yml-20260609-181614.bak`; `docker compose -f docker-compose.rpi5.yml up -d` recreated and started `qsfera-cloud-server`; `docker compose ps` reported loopback ports `127.0.0.1:9200->9200/tcp` and `127.0.0.1:9205->9205/tcp`.
 - `Server/scripts/verify-rpi-production.sh` passed on the Pi on 2026-06-09 with `PUBLIC_STATUS_URL=https://qsfera.kusoft.xyz/status.php` and `REQUIRE_METRICS=true`: image `qsfera-cloud-server:7.0.0-rpi5-stable`, container `running`/`healthy`, Docker root `/mnt/data/docker`, root free space `43.0 GiB`, data free space `851.5 GiB`, local and public status `stable`/`7.0.0`, and Prometheus metrics reachable.
+- Deployed the Pi systemd monitoring timer on 2026-06-09. `systemd-analyze verify` for the three installed unit files returned exit `0`; `systemctl is-enabled qsfera-rpi-production-check.timer` returned `enabled`; `systemctl is-active qsfera-rpi-production-check.timer` returned `active`; `systemctl show qsfera-rpi-production-check.service` returned `Result=success`, `ExecMainStatus=0`, `ActiveState=inactive`; and `journalctl -u qsfera-rpi-production-check.service` recorded `QSfera Raspberry Pi production verification passed: 0 warning(s).`

 ## Raspberry Pi 5 Check

@@ -79,5 +81,5 @@ Date: 2026-06-08.
 - Add Google Photos-style backup status UX: overall status, per-item status, queued/paused/permanent-failure states, and low-storage messaging.
 - Add mobile-data controls comparable to Google Photos/Yandex Disk beyond the Wi-Fi-only default: mobile-data limit and no-video-over-mobile-data options.
 - Store auto-upload sync cursors per source folder rather than one timestamp for all folders.
- Add an actual scheduler/alert target for `Server/scripts/verify-rpi-production.sh`; the runbook defines what to check, but no cron/systemd timer is committed yet.
+- Configure `QSFERA_ALERT_WEBHOOK_URL` in `/etc/qsfera/rpi-production-check.env` if off-host alert delivery is required; the installed failure service currently logs through `systemd-cat` and journal without an external endpoint.
 - Verify sharing controls against Google Drive and Yandex Disk expectations: link expiration, password, disable downloads, organization-only access, and role inheritance.
@@ -168,14 +168,52 @@ internal-only interface and set `PROXY_DEBUG_TOKEN`.

 ## Monitoring And Alerts

-Minimum checks for the Raspberry Pi deployment:
+Install the production checker timer from the `Server` directory:

- Run `./scripts/verify-rpi-production.sh` from the `Server` directory by cron
-  or systemd timer every 5 minutes with `PUBLIC_STATUS_URL` set to the public
-  QSfera URL.
- Alert when the script exits non-zero.
- Alert on any `WARN:` line that persists for more than one day, especially
+```bash
+sudo ./scripts/install-rpi-production-monitoring.sh
+```
+
+The installer writes:
+
+- `/etc/systemd/system/qsfera-rpi-production-check.service`
+- `/etc/systemd/system/qsfera-rpi-production-check.timer`
+- `/etc/systemd/system/qsfera-rpi-production-check-alert@.service`
+- `/etc/qsfera/rpi-production-check.env`
+
+The timer runs `./scripts/verify-rpi-production.sh` every 5 minutes. The service
+uses `OnFailure=qsfera-rpi-production-check-alert@%n.service`; the alert service
+logs the failed unit and recent journal output through `systemd-cat`. To forward
+alerts off-host, set `QSFERA_ALERT_WEBHOOK_URL` in
+`/etc/qsfera/rpi-production-check.env`; `alert-rpi-production-check-failure.sh`
+sends a JSON `POST` when that variable is set.
+
+For the current Raspberry Pi deployment checked on 2026-06-09, the timer was
+installed under `/etc/systemd/system`, `systemctl is-enabled` returned
+`enabled`, `systemctl is-active` returned `active`, and the checker service
+reported `Result=success`, `ExecMainStatus=0`, and `ActiveState=inactive` after
+the initial run. `systemctl list-timers qsfera-rpi-production-check.timer`
+reported the next run 5 minutes after the previous run.
+
+Verify the timer:
+
+```bash
+systemctl list-timers qsfera-rpi-production-check.timer --no-pager
+systemctl status qsfera-rpi-production-check.timer --no-pager
+systemctl show qsfera-rpi-production-check.service -p Result -p ExecMainStatus -p ActiveState
+journalctl -u qsfera-rpi-production-check.service -n 80 --no-pager
+```
+
+Minimum monitoring expectations:
+
+- Keep `qsfera-rpi-production-check.timer` enabled and active.
+- Investigate any non-zero `ExecMainStatus` from
+  `qsfera-rpi-production-check.service`.
+- Investigate any `WARN:` line that persists for more than one day, especially
  Docker root outside `/mnt/data` or unavailable metrics.
+- Configure `QSFERA_ALERT_WEBHOOK_URL` when off-host alert delivery is required;
+  without it, failures are still routed to the systemd failure service and
+  journal, but not to an external notification endpoint.
 - Scrape `http://127.0.0.1:9205/metrics` locally or through a private tunnel.
 - Keep public HTTPS probing separate from local loopback probing so reverse-proxy
  and backend failures are distinguishable.
@@ -0,0 +1,56 @@
+#!/usr/bin/env sh
+set -eu
+
+FAILED_UNIT="${1:-qsfera-rpi-production-check.service}"
+JOURNAL_LINES="${QSFERA_ALERT_JOURNAL_LINES:-80}"
+HOSTNAME_VALUE="$(hostname -f 2>/dev/null || hostname)"
+TIMESTAMP_UTC="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+SUMMARY="QSfera Raspberry Pi production check failed on ${HOSTNAME_VALUE}: ${FAILED_UNIT} at ${TIMESTAMP_UTC}"
+
+if command -v journalctl >/dev/null 2>&1; then
+    JOURNAL_OUTPUT="$(journalctl -u "$FAILED_UNIT" -n "$JOURNAL_LINES" --no-pager 2>/dev/null || true)"
+else
+    JOURNAL_OUTPUT=""
+fi
+
+{
+    echo "$SUMMARY"
+    if [ -n "$JOURNAL_OUTPUT" ]; then
+        echo
+        echo "$JOURNAL_OUTPUT"
+    fi
+} | if command -v systemd-cat >/dev/null 2>&1; then
+    systemd-cat -t qsfera-rpi-production-alert -p err
+else
+    cat >&2
+fi
+
+if [ -n "${QSFERA_ALERT_WEBHOOK_URL:-}" ]; then
+    if ! command -v python3 >/dev/null 2>&1; then
+        echo "ERROR: python3 is required to format the alert webhook payload." >&2
+        exit 1
+    fi
+    if ! command -v curl >/dev/null 2>&1; then
+        echo "ERROR: curl is required to send the alert webhook." >&2
+        exit 1
+    fi
+
+    python3 - "$SUMMARY" "$FAILED_UNIT" "$HOSTNAME_VALUE" "$TIMESTAMP_UTC" "$JOURNAL_OUTPUT" <<'PY' |
+import json
+import sys
+
+summary, failed_unit, hostname, timestamp_utc, journal = sys.argv[1:]
+print(json.dumps({
+    "summary": summary,
+    "failedUnit": failed_unit,
+    "host": hostname,
+    "timestampUtc": timestamp_utc,
+    "journal": journal,
+}, ensure_ascii=False))
+PY
+    curl -fsS \
+        -X POST \
+        -H "Content-Type: application/json" \
+        --data-binary @- \
+        "$QSFERA_ALERT_WEBHOOK_URL" >/dev/null
+fi
@@ -0,0 +1,62 @@
+#!/usr/bin/env sh
+set -eu
+
+SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
+SERVER_DIR="$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)"
+SYSTEMD_DIR="${SYSTEMD_DIR:-/etc/systemd/system}"
+ENV_DIR="${ENV_DIR:-/etc/qsfera}"
+ENV_FILE="${ENV_FILE:-$ENV_DIR/rpi-production-check.env}"
+RUN_NOW="${RUN_NOW:-true}"
+
+if [ "$(id -u)" -eq 0 ]; then
+    SUDO=""
+else
+    SUDO="${SUDO:-sudo}"
+fi
+
+run_privileged() {
+    if [ -n "$SUDO" ]; then
+        "$SUDO" "$@"
+    else
+        "$@"
+    fi
+}
+
+install_unit() {
+    source_path="$1"
+    target_path="$2"
+    run_privileged install -m 0644 "$source_path" "$target_path"
+}
+
+tmp_env="$(mktemp)"
+cleanup() {
+    rm -f "$tmp_env"
+}
+trap cleanup EXIT INT TERM
+
+run_privileged install -d -m 0755 "$SYSTEMD_DIR" "$ENV_DIR"
+
+if [ ! -f "$ENV_FILE" ]; then
+    {
+        echo "QSFERA_SERVER_DIR=$SERVER_DIR"
+        grep -v '^QSFERA_SERVER_DIR=' "$SERVER_DIR/systemd/rpi-production-check.env.example"
+    } >"$tmp_env"
+    run_privileged install -m 0640 "$tmp_env" "$ENV_FILE"
+    echo "Created $ENV_FILE"
+else
+    echo "Keeping existing $ENV_FILE"
+fi
+
+install_unit "$SERVER_DIR/systemd/qsfera-rpi-production-check.service" "$SYSTEMD_DIR/qsfera-rpi-production-check.service"
+install_unit "$SERVER_DIR/systemd/qsfera-rpi-production-check.timer" "$SYSTEMD_DIR/qsfera-rpi-production-check.timer"
+install_unit "$SERVER_DIR/systemd/qsfera-rpi-production-check-alert@.service" "$SYSTEMD_DIR/qsfera-rpi-production-check-alert@.service"
+
+run_privileged systemctl daemon-reload
+run_privileged systemctl enable --now qsfera-rpi-production-check.timer
+
+if [ "$RUN_NOW" = "true" ]; then
+    run_privileged systemctl start qsfera-rpi-production-check.service
+fi
+
+run_privileged systemctl --no-pager status qsfera-rpi-production-check.timer
+run_privileged systemctl --no-pager show qsfera-rpi-production-check.service -p Result -p ExecMainStatus -p ActiveState
@@ -0,0 +1,8 @@
+[Unit]
+Description=Report QSfera Raspberry Pi production check failure for %I
+Documentation=file:/mnt/data/qsfera/cloud-server/src/docs/raspberry-pi-5-docker.md
+
+[Service]
+Type=oneshot
+EnvironmentFile=-/etc/qsfera/rpi-production-check.env
+ExecStart=/bin/sh -c 'cd "${QSFERA_SERVER_DIR:-/mnt/data/qsfera/cloud-server/src}" && exec ./scripts/alert-rpi-production-check-failure.sh "$1"' -- "%I"
@@ -0,0 +1,12 @@
+[Unit]
+Description=QSfera Raspberry Pi production verification
+Documentation=file:/mnt/data/qsfera/cloud-server/src/docs/raspberry-pi-5-docker.md
+Wants=network-online.target
+After=network-online.target docker.service
+Requires=docker.service
+OnFailure=qsfera-rpi-production-check-alert@%n.service
+
+[Service]
+Type=oneshot
+EnvironmentFile=-/etc/qsfera/rpi-production-check.env
+ExecStart=/bin/sh -c 'cd "${QSFERA_SERVER_DIR:-/mnt/data/qsfera/cloud-server/src}" && exec ./scripts/verify-rpi-production.sh'
@@ -0,0 +1,13 @@
+[Unit]
+Description=Run QSfera Raspberry Pi production verification every five minutes
+Documentation=file:/mnt/data/qsfera/cloud-server/src/docs/raspberry-pi-5-docker.md
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=5min
+AccuracySec=30s
+Persistent=true
+Unit=qsfera-rpi-production-check.service
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,15 @@
+QSFERA_SERVER_DIR=/mnt/data/qsfera/cloud-server/src
+PUBLIC_STATUS_URL=https://qsfera.kusoft.xyz/status.php
+REQUIRE_METRICS=true
+METRICS_URL=http://127.0.0.1:9205/metrics
+EXPECTED_IMAGE=qsfera-cloud-server:7.0.0-rpi5-stable
+EXPECTED_VERSION=7.0.0
+EXPECTED_EDITION=stable
+EXPECTED_DOCKER_ROOT_PREFIX=/mnt/data
+STRICT_DOCKER_ROOT=true
+MIN_ROOT_AVAILABLE_KB=5242880
+MIN_DATA_AVAILABLE_KB=20971520
+QSFERA_ALERT_JOURNAL_LINES=80
+
+# Optional off-host alert endpoint. The alert script sends a JSON POST when set.
+#QSFERA_ALERT_WEBHOOK_URL=https://alerts.example.com/qsfera-rpi