diff --git a/PRODUCTION_READINESS.md b/PRODUCTION_READINESS.md index 3a95bd97..1c8f177f 100644 --- a/PRODUCTION_READINESS.md +++ b/PRODUCTION_READINESS.md @@ -41,8 +41,10 @@ Date: 2026-06-08. - Added `scripts/verify-production-state.ps1` as a repeatable live production endpoint gate for the public server status, Android Argus manifest/download, and Desktop update feed. - `scripts/verify-production-state.ps1 -VerifyAndroidDownload` passed on 2026-06-08: server `edition=stable`/`productversion=7.0.0`, Android `1.3.7`/`35`, downloaded APK size `12220848`, APK SHA-256 `c9c044c1332f8d973c64502ad2e3227ad92398101e7354ce9dbe7f708e7e693b`, and valid Desktop no-update feed. - Added `Server/scripts/verify-rpi-production.sh`, declared a loopback-only Pi metrics port in `Server/docker-compose.rpi5.yml`, and expanded the Raspberry Pi Docker runbook with local/public status checks, container/image checks, Prometheus metrics verification, backup/restore drill steps, and Docker data-root/retention planning. +- Added systemd monitoring units for the Pi: `Server/systemd/qsfera-rpi-production-check.service`, `Server/systemd/qsfera-rpi-production-check.timer`, `Server/systemd/qsfera-rpi-production-check-alert@.service`, `Server/systemd/rpi-production-check.env.example`, plus installer `Server/scripts/install-rpi-production-monitoring.sh` and alert helper `Server/scripts/alert-rpi-production-check-failure.sh`. - Deployed the Pi metrics/checker update on 2026-06-09. Before upload, `/mnt/data/qsfera/cloud-server/src/docker-compose.rpi5.yml` was backed up to `/mnt/data/qsfera/backups/docker-compose.rpi5.yml-20260609-181614.bak`; `docker compose -f docker-compose.rpi5.yml up -d` recreated and started `qsfera-cloud-server`; `docker compose ps` reported loopback ports `127.0.0.1:9200->9200/tcp` and `127.0.0.1:9205->9205/tcp`. - `Server/scripts/verify-rpi-production.sh` passed on the Pi on 2026-06-09 with `PUBLIC_STATUS_URL=https://qsfera.kusoft.xyz/status.php` and `REQUIRE_METRICS=true`: image `qsfera-cloud-server:7.0.0-rpi5-stable`, container `running`/`healthy`, Docker root `/mnt/data/docker`, root free space `43.0 GiB`, data free space `851.5 GiB`, local and public status `stable`/`7.0.0`, and Prometheus metrics reachable. +- Deployed the Pi systemd monitoring timer on 2026-06-09. `systemd-analyze verify` for the three installed unit files returned exit `0`; `systemctl is-enabled qsfera-rpi-production-check.timer` returned `enabled`; `systemctl is-active qsfera-rpi-production-check.timer` returned `active`; `systemctl show qsfera-rpi-production-check.service` returned `Result=success`, `ExecMainStatus=0`, `ActiveState=inactive`; and `journalctl -u qsfera-rpi-production-check.service` recorded `QSfera Raspberry Pi production verification passed: 0 warning(s).` ## Raspberry Pi 5 Check @@ -79,5 +81,5 @@ Date: 2026-06-08. - Add Google Photos-style backup status UX: overall status, per-item status, queued/paused/permanent-failure states, and low-storage messaging. - Add mobile-data controls comparable to Google Photos/Yandex Disk beyond the Wi-Fi-only default: mobile-data limit and no-video-over-mobile-data options. - Store auto-upload sync cursors per source folder rather than one timestamp for all folders. -- Add an actual scheduler/alert target for `Server/scripts/verify-rpi-production.sh`; the runbook defines what to check, but no cron/systemd timer is committed yet. +- Configure `QSFERA_ALERT_WEBHOOK_URL` in `/etc/qsfera/rpi-production-check.env` if off-host alert delivery is required; the installed failure service currently logs through `systemd-cat` and journal without an external endpoint. - Verify sharing controls against Google Drive and Yandex Disk expectations: link expiration, password, disable downloads, organization-only access, and role inheritance. diff --git a/Server/docs/raspberry-pi-5-docker.md b/Server/docs/raspberry-pi-5-docker.md index 938354bf..034ca525 100644 --- a/Server/docs/raspberry-pi-5-docker.md +++ b/Server/docs/raspberry-pi-5-docker.md @@ -168,14 +168,52 @@ internal-only interface and set `PROXY_DEBUG_TOKEN`. ## Monitoring And Alerts -Minimum checks for the Raspberry Pi deployment: +Install the production checker timer from the `Server` directory: -- Run `./scripts/verify-rpi-production.sh` from the `Server` directory by cron - or systemd timer every 5 minutes with `PUBLIC_STATUS_URL` set to the public - QSfera URL. -- Alert when the script exits non-zero. -- Alert on any `WARN:` line that persists for more than one day, especially +```bash +sudo ./scripts/install-rpi-production-monitoring.sh +``` + +The installer writes: + +- `/etc/systemd/system/qsfera-rpi-production-check.service` +- `/etc/systemd/system/qsfera-rpi-production-check.timer` +- `/etc/systemd/system/qsfera-rpi-production-check-alert@.service` +- `/etc/qsfera/rpi-production-check.env` + +The timer runs `./scripts/verify-rpi-production.sh` every 5 minutes. The service +uses `OnFailure=qsfera-rpi-production-check-alert@%n.service`; the alert service +logs the failed unit and recent journal output through `systemd-cat`. To forward +alerts off-host, set `QSFERA_ALERT_WEBHOOK_URL` in +`/etc/qsfera/rpi-production-check.env`; `alert-rpi-production-check-failure.sh` +sends a JSON `POST` when that variable is set. + +For the current Raspberry Pi deployment checked on 2026-06-09, the timer was +installed under `/etc/systemd/system`, `systemctl is-enabled` returned +`enabled`, `systemctl is-active` returned `active`, and the checker service +reported `Result=success`, `ExecMainStatus=0`, and `ActiveState=inactive` after +the initial run. `systemctl list-timers qsfera-rpi-production-check.timer` +reported the next run 5 minutes after the previous run. + +Verify the timer: + +```bash +systemctl list-timers qsfera-rpi-production-check.timer --no-pager +systemctl status qsfera-rpi-production-check.timer --no-pager +systemctl show qsfera-rpi-production-check.service -p Result -p ExecMainStatus -p ActiveState +journalctl -u qsfera-rpi-production-check.service -n 80 --no-pager +``` + +Minimum monitoring expectations: + +- Keep `qsfera-rpi-production-check.timer` enabled and active. +- Investigate any non-zero `ExecMainStatus` from + `qsfera-rpi-production-check.service`. +- Investigate any `WARN:` line that persists for more than one day, especially Docker root outside `/mnt/data` or unavailable metrics. +- Configure `QSFERA_ALERT_WEBHOOK_URL` when off-host alert delivery is required; + without it, failures are still routed to the systemd failure service and + journal, but not to an external notification endpoint. - Scrape `http://127.0.0.1:9205/metrics` locally or through a private tunnel. - Keep public HTTPS probing separate from local loopback probing so reverse-proxy and backend failures are distinguishable. diff --git a/Server/scripts/alert-rpi-production-check-failure.sh b/Server/scripts/alert-rpi-production-check-failure.sh new file mode 100644 index 00000000..87f8207e --- /dev/null +++ b/Server/scripts/alert-rpi-production-check-failure.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env sh +set -eu + +FAILED_UNIT="${1:-qsfera-rpi-production-check.service}" +JOURNAL_LINES="${QSFERA_ALERT_JOURNAL_LINES:-80}" +HOSTNAME_VALUE="$(hostname -f 2>/dev/null || hostname)" +TIMESTAMP_UTC="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +SUMMARY="QSfera Raspberry Pi production check failed on ${HOSTNAME_VALUE}: ${FAILED_UNIT} at ${TIMESTAMP_UTC}" + +if command -v journalctl >/dev/null 2>&1; then + JOURNAL_OUTPUT="$(journalctl -u "$FAILED_UNIT" -n "$JOURNAL_LINES" --no-pager 2>/dev/null || true)" +else + JOURNAL_OUTPUT="" +fi + +{ + echo "$SUMMARY" + if [ -n "$JOURNAL_OUTPUT" ]; then + echo + echo "$JOURNAL_OUTPUT" + fi +} | if command -v systemd-cat >/dev/null 2>&1; then + systemd-cat -t qsfera-rpi-production-alert -p err +else + cat >&2 +fi + +if [ -n "${QSFERA_ALERT_WEBHOOK_URL:-}" ]; then + if ! command -v python3 >/dev/null 2>&1; then + echo "ERROR: python3 is required to format the alert webhook payload." >&2 + exit 1 + fi + if ! command -v curl >/dev/null 2>&1; then + echo "ERROR: curl is required to send the alert webhook." >&2 + exit 1 + fi + + python3 - "$SUMMARY" "$FAILED_UNIT" "$HOSTNAME_VALUE" "$TIMESTAMP_UTC" "$JOURNAL_OUTPUT" <<'PY' | +import json +import sys + +summary, failed_unit, hostname, timestamp_utc, journal = sys.argv[1:] +print(json.dumps({ + "summary": summary, + "failedUnit": failed_unit, + "host": hostname, + "timestampUtc": timestamp_utc, + "journal": journal, +}, ensure_ascii=False)) +PY + curl -fsS \ + -X POST \ + -H "Content-Type: application/json" \ + --data-binary @- \ + "$QSFERA_ALERT_WEBHOOK_URL" >/dev/null +fi diff --git a/Server/scripts/install-rpi-production-monitoring.sh b/Server/scripts/install-rpi-production-monitoring.sh new file mode 100644 index 00000000..f1fc5c3e --- /dev/null +++ b/Server/scripts/install-rpi-production-monitoring.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env sh +set -eu + +SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)" +SERVER_DIR="$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)" +SYSTEMD_DIR="${SYSTEMD_DIR:-/etc/systemd/system}" +ENV_DIR="${ENV_DIR:-/etc/qsfera}" +ENV_FILE="${ENV_FILE:-$ENV_DIR/rpi-production-check.env}" +RUN_NOW="${RUN_NOW:-true}" + +if [ "$(id -u)" -eq 0 ]; then + SUDO="" +else + SUDO="${SUDO:-sudo}" +fi + +run_privileged() { + if [ -n "$SUDO" ]; then + "$SUDO" "$@" + else + "$@" + fi +} + +install_unit() { + source_path="$1" + target_path="$2" + run_privileged install -m 0644 "$source_path" "$target_path" +} + +tmp_env="$(mktemp)" +cleanup() { + rm -f "$tmp_env" +} +trap cleanup EXIT INT TERM + +run_privileged install -d -m 0755 "$SYSTEMD_DIR" "$ENV_DIR" + +if [ ! -f "$ENV_FILE" ]; then + { + echo "QSFERA_SERVER_DIR=$SERVER_DIR" + grep -v '^QSFERA_SERVER_DIR=' "$SERVER_DIR/systemd/rpi-production-check.env.example" + } >"$tmp_env" + run_privileged install -m 0640 "$tmp_env" "$ENV_FILE" + echo "Created $ENV_FILE" +else + echo "Keeping existing $ENV_FILE" +fi + +install_unit "$SERVER_DIR/systemd/qsfera-rpi-production-check.service" "$SYSTEMD_DIR/qsfera-rpi-production-check.service" +install_unit "$SERVER_DIR/systemd/qsfera-rpi-production-check.timer" "$SYSTEMD_DIR/qsfera-rpi-production-check.timer" +install_unit "$SERVER_DIR/systemd/qsfera-rpi-production-check-alert@.service" "$SYSTEMD_DIR/qsfera-rpi-production-check-alert@.service" + +run_privileged systemctl daemon-reload +run_privileged systemctl enable --now qsfera-rpi-production-check.timer + +if [ "$RUN_NOW" = "true" ]; then + run_privileged systemctl start qsfera-rpi-production-check.service +fi + +run_privileged systemctl --no-pager status qsfera-rpi-production-check.timer +run_privileged systemctl --no-pager show qsfera-rpi-production-check.service -p Result -p ExecMainStatus -p ActiveState diff --git a/Server/systemd/qsfera-rpi-production-check-alert@.service b/Server/systemd/qsfera-rpi-production-check-alert@.service new file mode 100644 index 00000000..ac724573 --- /dev/null +++ b/Server/systemd/qsfera-rpi-production-check-alert@.service @@ -0,0 +1,8 @@ +[Unit] +Description=Report QSfera Raspberry Pi production check failure for %I +Documentation=file:/mnt/data/qsfera/cloud-server/src/docs/raspberry-pi-5-docker.md + +[Service] +Type=oneshot +EnvironmentFile=-/etc/qsfera/rpi-production-check.env +ExecStart=/bin/sh -c 'cd "${QSFERA_SERVER_DIR:-/mnt/data/qsfera/cloud-server/src}" && exec ./scripts/alert-rpi-production-check-failure.sh "$1"' -- "%I" diff --git a/Server/systemd/qsfera-rpi-production-check.service b/Server/systemd/qsfera-rpi-production-check.service new file mode 100644 index 00000000..c8f39fe6 --- /dev/null +++ b/Server/systemd/qsfera-rpi-production-check.service @@ -0,0 +1,12 @@ +[Unit] +Description=QSfera Raspberry Pi production verification +Documentation=file:/mnt/data/qsfera/cloud-server/src/docs/raspberry-pi-5-docker.md +Wants=network-online.target +After=network-online.target docker.service +Requires=docker.service +OnFailure=qsfera-rpi-production-check-alert@%n.service + +[Service] +Type=oneshot +EnvironmentFile=-/etc/qsfera/rpi-production-check.env +ExecStart=/bin/sh -c 'cd "${QSFERA_SERVER_DIR:-/mnt/data/qsfera/cloud-server/src}" && exec ./scripts/verify-rpi-production.sh' diff --git a/Server/systemd/qsfera-rpi-production-check.timer b/Server/systemd/qsfera-rpi-production-check.timer new file mode 100644 index 00000000..d291ae10 --- /dev/null +++ b/Server/systemd/qsfera-rpi-production-check.timer @@ -0,0 +1,13 @@ +[Unit] +Description=Run QSfera Raspberry Pi production verification every five minutes +Documentation=file:/mnt/data/qsfera/cloud-server/src/docs/raspberry-pi-5-docker.md + +[Timer] +OnBootSec=2min +OnUnitActiveSec=5min +AccuracySec=30s +Persistent=true +Unit=qsfera-rpi-production-check.service + +[Install] +WantedBy=timers.target diff --git a/Server/systemd/rpi-production-check.env.example b/Server/systemd/rpi-production-check.env.example new file mode 100644 index 00000000..690259d2 --- /dev/null +++ b/Server/systemd/rpi-production-check.env.example @@ -0,0 +1,15 @@ +QSFERA_SERVER_DIR=/mnt/data/qsfera/cloud-server/src +PUBLIC_STATUS_URL=https://qsfera.kusoft.xyz/status.php +REQUIRE_METRICS=true +METRICS_URL=http://127.0.0.1:9205/metrics +EXPECTED_IMAGE=qsfera-cloud-server:7.0.0-rpi5-stable +EXPECTED_VERSION=7.0.0 +EXPECTED_EDITION=stable +EXPECTED_DOCKER_ROOT_PREFIX=/mnt/data +STRICT_DOCKER_ROOT=true +MIN_ROOT_AVAILABLE_KB=5242880 +MIN_DATA_AVAILABLE_KB=20971520 +QSFERA_ALERT_JOURNAL_LINES=80 + +# Optional off-host alert endpoint. The alert script sends a JSON POST when set. +#QSFERA_ALERT_WEBHOOK_URL=https://alerts.example.com/qsfera-rpi