Add Raspberry Pi production verification runbook

PRODUCTION_READINESS.md

@@ -40,6 +40,7 @@ Date: 2026-06-08.
 - Windows/Desktop now has a configured HTTPS update feed at `https://qsfera.kusoft.xyz/desktop/updates/stable.xml`; the feed returned HTTP `200` on 2026-06-08 and contains the XML elements parsed by `Desktop/src/gui/updater/updateinfo.cpp`.
 - Added `scripts/verify-production-state.ps1` as a repeatable live production endpoint gate for the public server status, Android Argus manifest/download, and Desktop update feed.
 - `scripts/verify-production-state.ps1 -VerifyAndroidDownload` passed on 2026-06-08: server `edition=stable`/`productversion=7.0.0`, Android `1.3.7`/`35`, downloaded APK size `12220848`, APK SHA-256 `c9c044c1332f8d973c64502ad2e3227ad92398101e7354ce9dbe7f708e7e693b`, and valid Desktop no-update feed.
+- Added `Server/scripts/verify-rpi-production.sh`, declared a loopback-only Pi metrics port in `Server/docker-compose.rpi5.yml`, and expanded the Raspberry Pi Docker runbook with local/public status checks, container/image checks, Prometheus metrics verification, backup/restore drill steps, and Docker data-root/retention planning.
 
 ## Raspberry Pi 5 Check
 
@@ -76,6 +77,7 @@ Date: 2026-06-08.
 - Add Google Photos-style backup status UX: overall status, per-item status, queued/paused/permanent-failure states, and low-storage messaging.
 - Add mobile-data controls comparable to Google Photos/Yandex Disk beyond the Wi-Fi-only default: mobile-data limit and no-video-over-mobile-data options.
 - Store auto-upload sync cursors per source folder rather than one timestamp for all folders.
-- Add production monitoring runbook and compose verification for metrics, tracing, alerts, backups, and restore drills.
+- Deploy the Raspberry Pi metrics endpoint change with `docker compose -f docker-compose.rpi5.yml up -d`, then run `Server/scripts/verify-rpi-production.sh` on the Pi with `PUBLIC_STATUS_URL=https://qsfera.kusoft.xyz/status.php` and `REQUIRE_METRICS=true`.
-- Move Docker build cache and/or Docker data-root planning for the Pi to `/mnt/data`, or add documented retention, so future server image rebuilds cannot fill the root filesystem again.
+- Add an actual scheduler/alert target for `Server/scripts/verify-rpi-production.sh`; the runbook defines what to check, but no cron/systemd timer is committed yet.
+- Decide whether to move Docker `data-root` to `/mnt/data/docker` or keep scheduled builder/image retention. The runbook documents both paths, but the host-level Docker daemon setting still needs an operator maintenance window.
 - Verify sharing controls against Google Drive and Yandex Disk expectations: link expiration, password, disable downloads, organization-only access, and role inheritance.

Server/.env.example

@@ -3,6 +3,7 @@ QSFERA_VERSION=7.0.0
 QSFERA_EDITION=stable
 QSFERA_BUILD_DATE=20260608
 QSFERA_RPI_HOST_PORT=9200
+QSFERA_RPI_DEBUG_HOST_PORT=9205
 QSFERA_RPI_CONFIG_PATH=/mnt/data/qsfera/cloud-server/config
 QSFERA_RPI_DATA_PATH=/mnt/data/qsfera/cloud-server/data
 
@@ -10,4 +11,6 @@ OC_URL=https://qsfera.example.com
 OC_LOG_LEVEL=info
 OC_INSECURE=false
 PROXY_TLS=true
+PROXY_DEBUG_ADDR=0.0.0.0:9205
+PROXY_DEBUG_TOKEN=
 FRONTEND_CHECK_FOR_UPDATES=false

Server/docker-compose.rpi5.yml

@@ -25,8 +25,11 @@ services:
       OC_URL: ${OC_URL:?OC_URL must be set in .env}
       OC_INSECURE: ${OC_INSECURE:-false}
       PROXY_TLS: ${PROXY_TLS:-true}
+      PROXY_DEBUG_ADDR: ${PROXY_DEBUG_ADDR:-0.0.0.0:9205}
+      PROXY_DEBUG_TOKEN: ${PROXY_DEBUG_TOKEN:-}
     ports:
       - "127.0.0.1:${QSFERA_RPI_HOST_PORT:-9200}:9200"
+      - "127.0.0.1:${QSFERA_RPI_DEBUG_HOST_PORT:-9205}:9205"
     volumes:
       - ${QSFERA_RPI_CONFIG_PATH:?QSFERA_RPI_CONFIG_PATH must be set in .env}:/etc/qsfera
       - ${QSFERA_RPI_DATA_PATH:?QSFERA_RPI_DATA_PATH must be set in .env}:/var/lib/qsfera

Server/docs/raspberry-pi-5-docker.md

@@ -21,6 +21,7 @@ Files used for this deployment:
   `QSFERA_RPI_DATA_PATH`.
 - Host-side backend port bound only to `127.0.0.1:9200`; external access should go
   through the reverse proxy and HTTPS domain.
+- Host-side proxy debug/metrics port bound only to `127.0.0.1:9205`.
 
 ## Prerequisites On Raspberry Pi 5
 
@@ -31,6 +32,9 @@ Files used for this deployment:
 5. Set `OC_URL` to the public HTTPS URL used by clients and the reverse proxy.
 6. Keep `QSFERA_EDITION=stable` and a concrete `QSFERA_VERSION` for production
    rollouts. Leaving those unset falls back to the defaults in the compose file.
+7. Keep Docker image and builder retention under control. The current Raspberry Pi
+   deployment stores QSfera data on `/mnt/data`; Docker's own `data-root` must
+   either be moved there during a maintenance window or pruned on a schedule.
 
 ## Start
 
@@ -48,6 +52,13 @@ docker compose -f docker-compose.rpi5.yml build
 docker run --rm --entrypoint /usr/bin/qsfera qsfera-cloud-server:7.0.0-rpi5-stable version --skip-services
 ```
 
+Run the production checker from the `Server` directory after every rollout:
+
+```bash
+PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \
+  ./scripts/verify-rpi-production.sh
+```
+
 ## Stop
 
 ```bash
@@ -80,6 +91,36 @@ External reverse-proxied status endpoint:
 curl https://qsfera.example.com/status.php
 ```
 
+Local Prometheus metrics endpoint:
+
+```bash
+curl http://127.0.0.1:9205/metrics
+```
+
+Repeatable production gate from the `Server` directory:
+
+```bash
+PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \
+  REQUIRE_METRICS=true \
+  ./scripts/verify-rpi-production.sh
+```
+
+The checker validates:
+
+- Docker Compose config rendering.
+- Local presence of the pinned image.
+- Image metadata from `qsfera version --skip-services`.
+- Container running state, Docker health status, and container image tag.
+- Docker root location, root filesystem free space, and data filesystem free space.
+- Local and optional public `/status.php` JSON invariants:
+  `installed=true`, `maintenance=false`, `needsDbUpgrade=false`,
+  `edition=stable`, and `productversion=7.0.0`.
+- Optional Prometheus metrics reachability.
+
+The default free-space thresholds are 5 GiB for `/` and 20 GiB for `/mnt/data`.
+Override them with `MIN_ROOT_AVAILABLE_KB` and `MIN_DATA_AVAILABLE_KB` when the
+host has a documented capacity plan.
+
 ## Data Location
 
 Docker bind mounts:
@@ -118,3 +159,94 @@ qsfera.kusoft.xyz {
     }
 }
 ```
+
+Do not proxy `127.0.0.1:9205` to the public Internet. If a remote Prometheus
+server needs to scrape it, connect through a private tunnel or bind it to an
+internal-only interface and set `PROXY_DEBUG_TOKEN`.
+
+## Monitoring And Alerts
+
+Minimum checks for the Raspberry Pi deployment:
+
+- Run `./scripts/verify-rpi-production.sh` from the `Server` directory by cron
+  or systemd timer every 5 minutes with `PUBLIC_STATUS_URL` set to the public
+  QSfera URL.
+- Alert when the script exits non-zero.
+- Alert on any `WARN:` line that persists for more than one day, especially
+  Docker root outside `/mnt/data` or unavailable metrics.
+- Scrape `http://127.0.0.1:9205/metrics` locally or through a private tunnel.
+- Keep public HTTPS probing separate from local loopback probing so reverse-proxy
+  and backend failures are distinguishable.
+- Keep `OC_LOG_LEVEL=info` for production; raise to `debug` only for a bounded
+  incident window.
+
+For larger deployments, the full compose example already contains tracing and
+metrics wiring in `devtools/deployments/qsfera_full/monitoring_tracing/`.
+
+## Backups And Restore Drill
+
+Back up the two bind-mounted paths from `.env`:
+
+- `QSFERA_RPI_CONFIG_PATH`
+- `QSFERA_RPI_DATA_PATH`
+
+Create an archive before every server image rollout:
+
+```bash
+stamp="$(date -u +%Y%m%d-%H%M%S)"
+backup="/mnt/data/qsfera/backups/cloud-server-${stamp}.tar.gz"
+sudo tar -C /mnt/data/qsfera/cloud-server -czf "$backup" config data
+sudo tar -tzf "$backup" >/dev/null
+```
+
+Run a restore drill after backup automation changes and at least monthly:
+
+```bash
+stamp="$(date -u +%Y%m%d-%H%M%S)"
+restore_root="/mnt/data/qsfera/restore-drills/${stamp}"
+sudo mkdir -p "$restore_root"
+sudo tar -C "$restore_root" -xzf "$backup"
+sudo test -d "$restore_root/config"
+sudo test -d "$restore_root/data"
+```
+
+For a full service restore test, copy `.env` to a drill-only file, point
+`QSFERA_RPI_CONFIG_PATH` and `QSFERA_RPI_DATA_PATH` at the extracted directories,
+bind `QSFERA_RPI_HOST_PORT` to an unused loopback port, run
+`docker compose -f docker-compose.rpi5.yml --env-file .env.restore config`, and
+only then start the drill container. Do not run a restore drill against the live
+config/data paths.
+
+## Docker Data-Root And Retention
+
+Check where Docker stores images, layers, and build cache:
+
+```bash
+docker info --format '{{.DockerRootDir}}'
+docker system df
+```
+
+Preferred production state on this Pi is Docker data-root under `/mnt/data`,
+because the QSfera bind-mounted data is already stored there. Move it only during
+a maintenance window:
+
+```bash
+sudo systemctl stop docker
+sudo mkdir -p /mnt/data/docker
+sudo rsync -aHAX --numeric-ids /var/lib/docker/ /mnt/data/docker/
+printf '{\n  "data-root": "/mnt/data/docker"\n}\n' | sudo tee /etc/docker/daemon.json
+sudo systemctl start docker
+docker info --format '{{.DockerRootDir}}'
+docker compose -f docker-compose.rpi5.yml ps
+```
+
+If moving Docker data-root is postponed, keep conservative retention active:
+
+```bash
+docker builder prune --filter 'until=168h'
+docker image prune --filter 'until=168h'
+docker system df
+```
+
+Do not run `docker system prune --volumes` on this host unless a current backup
+has been verified and the target volumes have been explicitly reviewed.

Server/scripts/verify-rpi-production.sh

@@ -0,0 +1,273 @@
+#!/usr/bin/env sh
+set -eu
+
+COMPOSE_FILE="${COMPOSE_FILE:-docker-compose.rpi5.yml}"
+ENV_FILE="${ENV_FILE:-.env}"
+SERVICE="${SERVICE:-qsfera-cloud-server}"
+CONTAINER="${CONTAINER:-qsfera-cloud-server}"
+EXPECTED_IMAGE="${EXPECTED_IMAGE:-qsfera-cloud-server:7.0.0-rpi5-stable}"
+EXPECTED_VERSION="${EXPECTED_VERSION:-7.0.0}"
+EXPECTED_EDITION="${EXPECTED_EDITION:-stable}"
+LOCAL_STATUS_URL="${LOCAL_STATUS_URL:-https://127.0.0.1:9200/status.php}"
+PUBLIC_STATUS_URL="${PUBLIC_STATUS_URL:-}"
+METRICS_URL="${METRICS_URL:-http://127.0.0.1:9205/metrics}"
+REQUIRE_METRICS="${REQUIRE_METRICS:-false}"
+VERIFY_IMAGE_VERSION="${VERIFY_IMAGE_VERSION:-true}"
+ROOT_PATH="${ROOT_PATH:-/}"
+DATA_PATH="${DATA_PATH:-/mnt/data}"
+MIN_ROOT_AVAILABLE_KB="${MIN_ROOT_AVAILABLE_KB:-5242880}"
+MIN_DATA_AVAILABLE_KB="${MIN_DATA_AVAILABLE_KB:-20971520}"
+EXPECTED_DOCKER_ROOT_PREFIX="${EXPECTED_DOCKER_ROOT_PREFIX:-/mnt/data}"
+STRICT_DOCKER_ROOT="${STRICT_DOCKER_ROOT:-false}"
+
+errors=0
+warnings=0
+tmp_dir="$(mktemp -d)"
+
+cleanup() {
+    rm -rf "$tmp_dir"
+}
+trap cleanup EXIT INT TERM
+
+ok() {
+    echo "OK: $*"
+}
+
+warn() {
+    warnings=$((warnings + 1))
+    echo "WARN: $*" >&2
+}
+
+error() {
+    errors=$((errors + 1))
+    echo "ERROR: $*" >&2
+}
+
+is_true() {
+    case "$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" in
+        true|1|yes|on)
+            return 0
+            ;;
+        *)
+            return 1
+            ;;
+    esac
+}
+
+require_command() {
+    if command -v "$1" >/dev/null 2>&1; then
+        ok "Command available: $1"
+    else
+        error "Required command is missing: $1"
+    fi
+}
+
+compose() {
+    if [ -n "$ENV_FILE" ] && [ -f "$ENV_FILE" ]; then
+        docker compose --env-file "$ENV_FILE" -f "$COMPOSE_FILE" "$@"
+    else
+        docker compose -f "$COMPOSE_FILE" "$@"
+    fi
+}
+
+available_kb() {
+    df -Pk "$1" 2>/dev/null | awk 'NR == 2 { print $4 }'
+}
+
+format_gib() {
+    awk -v kb="$1" 'BEGIN { printf "%.1f", kb / 1048576 }'
+}
+
+check_disk() {
+    path="$1"
+    minimum_kb="$2"
+    label="$3"
+    current_kb="$(available_kb "$path")"
+
+    if [ -z "$current_kb" ]; then
+        error "Cannot read free disk space for $label at $path."
+        return
+    fi
+
+    if [ "$current_kb" -ge "$minimum_kb" ]; then
+        ok "$label free space $(format_gib "$current_kb") GiB >= $(format_gib "$minimum_kb") GiB"
+    else
+        error "$label free space $(format_gib "$current_kb") GiB < $(format_gib "$minimum_kb") GiB"
+    fi
+}
+
+check_status_url() {
+    label="$1"
+    url="$2"
+    curl_opts="$3"
+    output="$tmp_dir/status.json"
+
+    if curl $curl_opts "$url" -o "$output"; then
+        ok "$label returned a successful HTTP response"
+    else
+        error "$label request failed: $url"
+        return
+    fi
+
+    if python3 - "$output" "$EXPECTED_EDITION" "$EXPECTED_VERSION" "$label" <<'PY'
+import json
+import sys
+
+path, expected_edition, expected_version, label = sys.argv[1:]
+with open(path, "r", encoding="utf-8") as handle:
+    payload = json.load(handle)
+
+checks = [
+    ("installed", payload.get("installed") is True, True, payload.get("installed")),
+    ("maintenance", payload.get("maintenance") is False, False, payload.get("maintenance")),
+    ("needsDbUpgrade", payload.get("needsDbUpgrade") is False, False, payload.get("needsDbUpgrade")),
+    ("edition", payload.get("edition") == expected_edition, expected_edition, payload.get("edition")),
+    ("productversion", payload.get("productversion") == expected_version, expected_version, payload.get("productversion")),
+]
+
+failed = False
+for name, passed, expected, actual in checks:
+    if passed:
+        print(f"OK: {label} {name} = {actual}")
+    else:
+        print(f"ERROR: {label} {name}: expected {expected!r}, got {actual!r}", file=sys.stderr)
+        failed = True
+
+if failed:
+    sys.exit(1)
+PY
+    then
+        ok "$label JSON invariants match expected production state"
+    else
+        error "$label JSON invariants failed"
+    fi
+}
+
+check_metrics() {
+    output="$tmp_dir/metrics.txt"
+
+    if [ -n "${METRICS_BEARER_TOKEN:-}" ]; then
+        metrics_result=0
+        curl -fsS -H "Authorization: Bearer ${METRICS_BEARER_TOKEN}" "$METRICS_URL" -o "$output" || metrics_result=$?
+    else
+        metrics_result=0
+        curl -fsS "$METRICS_URL" -o "$output" || metrics_result=$?
+    fi
+
+    if [ "$metrics_result" -eq 0 ]; then
+        if grep -Eq '^(# HELP|# TYPE|go_|process_)' "$output"; then
+            ok "Metrics endpoint returned Prometheus-formatted content"
+        else
+            error "Metrics endpoint responded, but content does not look like Prometheus metrics."
+        fi
+        return
+    fi
+
+    if is_true "$REQUIRE_METRICS"; then
+        error "Metrics endpoint is required but unavailable: $METRICS_URL"
+    else
+        warn "Metrics endpoint unavailable: $METRICS_URL"
+    fi
+}
+
+for command_name in docker curl python3 awk df; do
+    require_command "$command_name"
+done
+
+if [ ! -f "$COMPOSE_FILE" ]; then
+    error "Compose file not found: $COMPOSE_FILE"
+fi
+
+if [ -n "$ENV_FILE" ] && [ ! -f "$ENV_FILE" ]; then
+    error "Environment file not found: $ENV_FILE"
+fi
+
+if [ "$errors" -eq 0 ]; then
+    if compose config >/dev/null; then
+        ok "Docker Compose config renders successfully for $COMPOSE_FILE"
+    else
+        error "Docker Compose config failed for $COMPOSE_FILE"
+    fi
+fi
+
+if docker image inspect "$EXPECTED_IMAGE" >/dev/null 2>&1; then
+    ok "Expected image exists locally: $EXPECTED_IMAGE"
+else
+    error "Expected image is missing locally: $EXPECTED_IMAGE"
+fi
+
+if is_true "$VERIFY_IMAGE_VERSION"; then
+    version_output="$tmp_dir/image-version.txt"
+    if docker run --rm --entrypoint /usr/bin/qsfera "$EXPECTED_IMAGE" version --skip-services >"$version_output" 2>&1; then
+        if grep -q "Version: $EXPECTED_VERSION" "$version_output" && grep -q "Edition: $EXPECTED_EDITION" "$version_output"; then
+            ok "Image version metadata matches $EXPECTED_EDITION $EXPECTED_VERSION"
+        else
+            error "Image version metadata does not match expected $EXPECTED_EDITION $EXPECTED_VERSION."
+            sed 's/^/  /' "$version_output" >&2
+        fi
+    else
+        error "Cannot read image version metadata from $EXPECTED_IMAGE."
+        sed 's/^/  /' "$version_output" >&2
+    fi
+fi
+
+container_status="$(docker inspect -f '{{.State.Status}}' "$CONTAINER" 2>/dev/null || true)"
+if [ "$container_status" = "running" ]; then
+    ok "Container $CONTAINER is running"
+else
+    error "Container $CONTAINER status is '${container_status:-missing}', expected running."
+fi
+
+container_health="$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$CONTAINER" 2>/dev/null || true)"
+case "$container_health" in
+    healthy)
+        ok "Container $CONTAINER health is healthy"
+        ;;
+    none)
+        warn "Container $CONTAINER has no Docker healthcheck result."
+        ;;
+    *)
+        error "Container $CONTAINER health is '${container_health:-missing}', expected healthy."
+        ;;
+esac
+
+container_image="$(docker inspect -f '{{.Config.Image}}' "$CONTAINER" 2>/dev/null || true)"
+if [ "$container_image" = "$EXPECTED_IMAGE" ]; then
+    ok "Container image is $EXPECTED_IMAGE"
+else
+    error "Container image is '${container_image:-missing}', expected $EXPECTED_IMAGE."
+fi
+
+docker_root="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)"
+case "$docker_root" in
+    "$EXPECTED_DOCKER_ROOT_PREFIX"|"$EXPECTED_DOCKER_ROOT_PREFIX"/*)
+        ok "Docker root is under $EXPECTED_DOCKER_ROOT_PREFIX: $docker_root"
+        ;;
+    "")
+        error "Cannot read Docker root directory."
+        ;;
+    *)
+        if is_true "$STRICT_DOCKER_ROOT"; then
+            error "Docker root is $docker_root, expected path under $EXPECTED_DOCKER_ROOT_PREFIX."
+        else
+            warn "Docker root is $docker_root, not under $EXPECTED_DOCKER_ROOT_PREFIX; keep image/build retention active or move data-root during maintenance."
+        fi
+        ;;
+esac
+
+check_disk "$ROOT_PATH" "$MIN_ROOT_AVAILABLE_KB" "Root filesystem"
+check_disk "$DATA_PATH" "$MIN_DATA_AVAILABLE_KB" "Data filesystem"
+check_status_url "Local server status" "$LOCAL_STATUS_URL" "-kfsS"
+
+if [ -n "$PUBLIC_STATUS_URL" ]; then
+    check_status_url "Public server status" "$PUBLIC_STATUS_URL" "-fsS"
+fi
+
+check_metrics
+
+if [ "$errors" -gt 0 ]; then
+    echo "QSfera Raspberry Pi production verification failed: $errors error(s), $warnings warning(s)." >&2
+    exit 1
+fi
+
+echo "QSfera Raspberry Pi production verification passed: $warnings warning(s)."