Add Raspberry Pi production verification runbook

Server/docs/raspberry-pi-5-docker.md

@@ -21,6 +21,7 @@ Files used for this deployment:
   `QSFERA_RPI_DATA_PATH`.
 - Host-side backend port bound only to `127.0.0.1:9200`; external access should go
   through the reverse proxy and HTTPS domain.
+- Host-side proxy debug/metrics port bound only to `127.0.0.1:9205`.
 
 ## Prerequisites On Raspberry Pi 5
 
@@ -31,6 +32,9 @@ Files used for this deployment:
 5. Set `OC_URL` to the public HTTPS URL used by clients and the reverse proxy.
 6. Keep `QSFERA_EDITION=stable` and a concrete `QSFERA_VERSION` for production
    rollouts. Leaving those unset falls back to the defaults in the compose file.
+7. Keep Docker image and builder retention under control. The current Raspberry Pi
+   deployment stores QSfera data on `/mnt/data`; Docker's own `data-root` must
+   either be moved there during a maintenance window or pruned on a schedule.
 
 ## Start
 
@@ -48,6 +52,13 @@ docker compose -f docker-compose.rpi5.yml build
 docker run --rm --entrypoint /usr/bin/qsfera qsfera-cloud-server:7.0.0-rpi5-stable version --skip-services
 ```
 
+Run the production checker from the `Server` directory after every rollout:
+
+```bash
+PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \
+  ./scripts/verify-rpi-production.sh
+```
+
 ## Stop
 
 ```bash
@@ -80,6 +91,36 @@ External reverse-proxied status endpoint:
 curl https://qsfera.example.com/status.php
 ```
 
+Local Prometheus metrics endpoint:
+
+```bash
+curl http://127.0.0.1:9205/metrics
+```
+
+Repeatable production gate from the `Server` directory:
+
+```bash
+PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \
+  REQUIRE_METRICS=true \
+  ./scripts/verify-rpi-production.sh
+```
+
+The checker validates:
+
+- Docker Compose config rendering.
+- Local presence of the pinned image.
+- Image metadata from `qsfera version --skip-services`.
+- Container running state, Docker health status, and container image tag.
+- Docker root location, root filesystem free space, and data filesystem free space.
+- Local and optional public `/status.php` JSON invariants:
+  `installed=true`, `maintenance=false`, `needsDbUpgrade=false`,
+  `edition=stable`, and `productversion=7.0.0`.
+- Optional Prometheus metrics reachability.
+
+The default free-space thresholds are 5 GiB for `/` and 20 GiB for `/mnt/data`.
+Override them with `MIN_ROOT_AVAILABLE_KB` and `MIN_DATA_AVAILABLE_KB` when the
+host has a documented capacity plan.
+
 ## Data Location
 
 Docker bind mounts:
@@ -118,3 +159,94 @@ qsfera.kusoft.xyz {
     }
 }
 ```
+
+Do not proxy `127.0.0.1:9205` to the public Internet. If a remote Prometheus
+server needs to scrape it, connect through a private tunnel or bind it to an
+internal-only interface and set `PROXY_DEBUG_TOKEN`.
+
+## Monitoring And Alerts
+
+Minimum checks for the Raspberry Pi deployment:
+
+- Run `./scripts/verify-rpi-production.sh` from the `Server` directory by cron
+  or systemd timer every 5 minutes with `PUBLIC_STATUS_URL` set to the public
+  QSfera URL.
+- Alert when the script exits non-zero.
+- Alert on any `WARN:` line that persists for more than one day, especially
+  Docker root outside `/mnt/data` or unavailable metrics.
+- Scrape `http://127.0.0.1:9205/metrics` locally or through a private tunnel.
+- Keep public HTTPS probing separate from local loopback probing so reverse-proxy
+  and backend failures are distinguishable.
+- Keep `OC_LOG_LEVEL=info` for production; raise to `debug` only for a bounded
+  incident window.
+
+For larger deployments, the full compose example already contains tracing and
+metrics wiring in `devtools/deployments/qsfera_full/monitoring_tracing/`.
+
+## Backups And Restore Drill
+
+Back up the two bind-mounted paths from `.env`:
+
+- `QSFERA_RPI_CONFIG_PATH`
+- `QSFERA_RPI_DATA_PATH`
+
+Create an archive before every server image rollout:
+
+```bash
+stamp="$(date -u +%Y%m%d-%H%M%S)"
+backup="/mnt/data/qsfera/backups/cloud-server-${stamp}.tar.gz"
+sudo tar -C /mnt/data/qsfera/cloud-server -czf "$backup" config data
+sudo tar -tzf "$backup" >/dev/null
+```
+
+Run a restore drill after backup automation changes and at least monthly:
+
+```bash
+stamp="$(date -u +%Y%m%d-%H%M%S)"
+restore_root="/mnt/data/qsfera/restore-drills/${stamp}"
+sudo mkdir -p "$restore_root"
+sudo tar -C "$restore_root" -xzf "$backup"
+sudo test -d "$restore_root/config"
+sudo test -d "$restore_root/data"
+```
+
+For a full service restore test, copy `.env` to a drill-only file, point
+`QSFERA_RPI_CONFIG_PATH` and `QSFERA_RPI_DATA_PATH` at the extracted directories,
+bind `QSFERA_RPI_HOST_PORT` to an unused loopback port, run
+`docker compose -f docker-compose.rpi5.yml --env-file .env.restore config`, and
+only then start the drill container. Do not run a restore drill against the live
+config/data paths.
+
+## Docker Data-Root And Retention
+
+Check where Docker stores images, layers, and build cache:
+
+```bash
+docker info --format '{{.DockerRootDir}}'
+docker system df
+```
+
+Preferred production state on this Pi is Docker data-root under `/mnt/data`,
+because the QSfera bind-mounted data is already stored there. Move it only during
+a maintenance window:
+
+```bash
+sudo systemctl stop docker
+sudo mkdir -p /mnt/data/docker
+sudo rsync -aHAX --numeric-ids /var/lib/docker/ /mnt/data/docker/
+printf '{\n  "data-root": "/mnt/data/docker"\n}\n' | sudo tee /etc/docker/daemon.json
+sudo systemctl start docker
+docker info --format '{{.DockerRootDir}}'
+docker compose -f docker-compose.rpi5.yml ps
+```
+
+If moving Docker data-root is postponed, keep conservative retention active:
+
+```bash
+docker builder prune --filter 'until=168h'
+docker image prune --filter 'until=168h'
+docker system df
+```
+
+Do not run `docker system prune --volumes` on this host unless a current backup
+has been verified and the target volumes have been explicitly reviewed.