Add Raspberry Pi production verification runbook

This commit is contained in:
Курнат Андрей
2026-06-09 21:14:26 +03:00
parent 5e4dfc6642
commit 9323f976d8
5 changed files with 415 additions and 2 deletions
+4 -2
View File
@@ -40,6 +40,7 @@ Date: 2026-06-08.
- Windows/Desktop now has a configured HTTPS update feed at `https://qsfera.kusoft.xyz/desktop/updates/stable.xml`; the feed returned HTTP `200` on 2026-06-08 and contains the XML elements parsed by `Desktop/src/gui/updater/updateinfo.cpp`.
- Added `scripts/verify-production-state.ps1` as a repeatable live production endpoint gate for the public server status, Android Argus manifest/download, and Desktop update feed.
- `scripts/verify-production-state.ps1 -VerifyAndroidDownload` passed on 2026-06-08: server `edition=stable`/`productversion=7.0.0`, Android `1.3.7`/`35`, downloaded APK size `12220848`, APK SHA-256 `c9c044c1332f8d973c64502ad2e3227ad92398101e7354ce9dbe7f708e7e693b`, and valid Desktop no-update feed.
- Added `Server/scripts/verify-rpi-production.sh`, declared a loopback-only Pi metrics port in `Server/docker-compose.rpi5.yml`, and expanded the Raspberry Pi Docker runbook with local/public status checks, container/image checks, Prometheus metrics verification, backup/restore drill steps, and Docker data-root/retention planning.
## Raspberry Pi 5 Check
@@ -76,6 +77,7 @@ Date: 2026-06-08.
- Add Google Photos-style backup status UX: overall status, per-item status, queued/paused/permanent-failure states, and low-storage messaging.
- Add mobile-data controls comparable to Google Photos/Yandex Disk beyond the Wi-Fi-only default: mobile-data limit and no-video-over-mobile-data options.
- Store auto-upload sync cursors per source folder rather than one timestamp for all folders.
- Add production monitoring runbook and compose verification for metrics, tracing, alerts, backups, and restore drills.
- Move Docker build cache and/or Docker data-root planning for the Pi to `/mnt/data`, or add documented retention, so future server image rebuilds cannot fill the root filesystem again.
- Deploy the Raspberry Pi metrics endpoint change with `docker compose -f docker-compose.rpi5.yml up -d`, then run `Server/scripts/verify-rpi-production.sh` on the Pi with `PUBLIC_STATUS_URL=https://qsfera.kusoft.xyz/status.php` and `REQUIRE_METRICS=true`.
- Add an actual scheduler/alert target for `Server/scripts/verify-rpi-production.sh`; the runbook defines what to check, but no cron/systemd timer is committed yet.
- Decide whether to move Docker `data-root` to `/mnt/data/docker` or keep scheduled builder/image retention. The runbook documents both paths, but the host-level Docker daemon setting still needs an operator maintenance window.
- Verify sharing controls against Google Drive and Yandex Disk expectations: link expiration, password, disable downloads, organization-only access, and role inheritance.
+3
View File
@@ -3,6 +3,7 @@ QSFERA_VERSION=7.0.0
QSFERA_EDITION=stable
QSFERA_BUILD_DATE=20260608
QSFERA_RPI_HOST_PORT=9200
QSFERA_RPI_DEBUG_HOST_PORT=9205
QSFERA_RPI_CONFIG_PATH=/mnt/data/qsfera/cloud-server/config
QSFERA_RPI_DATA_PATH=/mnt/data/qsfera/cloud-server/data
@@ -10,4 +11,6 @@ OC_URL=https://qsfera.example.com
OC_LOG_LEVEL=info
OC_INSECURE=false
PROXY_TLS=true
PROXY_DEBUG_ADDR=0.0.0.0:9205
PROXY_DEBUG_TOKEN=
FRONTEND_CHECK_FOR_UPDATES=false
+3
View File
@@ -25,8 +25,11 @@ services:
OC_URL: ${OC_URL:?OC_URL must be set in .env}
OC_INSECURE: ${OC_INSECURE:-false}
PROXY_TLS: ${PROXY_TLS:-true}
PROXY_DEBUG_ADDR: ${PROXY_DEBUG_ADDR:-0.0.0.0:9205}
PROXY_DEBUG_TOKEN: ${PROXY_DEBUG_TOKEN:-}
ports:
- "127.0.0.1:${QSFERA_RPI_HOST_PORT:-9200}:9200"
- "127.0.0.1:${QSFERA_RPI_DEBUG_HOST_PORT:-9205}:9205"
volumes:
- ${QSFERA_RPI_CONFIG_PATH:?QSFERA_RPI_CONFIG_PATH must be set in .env}:/etc/qsfera
- ${QSFERA_RPI_DATA_PATH:?QSFERA_RPI_DATA_PATH must be set in .env}:/var/lib/qsfera
+132
View File
@@ -21,6 +21,7 @@ Files used for this deployment:
`QSFERA_RPI_DATA_PATH`.
- Host-side backend port bound only to `127.0.0.1:9200`; external access should go
through the reverse proxy and HTTPS domain.
- Host-side proxy debug/metrics port bound only to `127.0.0.1:9205`.
## Prerequisites On Raspberry Pi 5
@@ -31,6 +32,9 @@ Files used for this deployment:
5. Set `OC_URL` to the public HTTPS URL used by clients and the reverse proxy.
6. Keep `QSFERA_EDITION=stable` and a concrete `QSFERA_VERSION` for production
rollouts. Leaving those unset falls back to the defaults in the compose file.
7. Keep Docker image and builder retention under control. The current Raspberry Pi
deployment stores QSfera data on `/mnt/data`; Docker's own `data-root` must
either be moved there during a maintenance window or pruned on a schedule.
## Start
@@ -48,6 +52,13 @@ docker compose -f docker-compose.rpi5.yml build
docker run --rm --entrypoint /usr/bin/qsfera qsfera-cloud-server:7.0.0-rpi5-stable version --skip-services
```
Run the production checker from the `Server` directory after every rollout:
```bash
PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \
./scripts/verify-rpi-production.sh
```
## Stop
```bash
@@ -80,6 +91,36 @@ External reverse-proxied status endpoint:
curl https://qsfera.example.com/status.php
```
Local Prometheus metrics endpoint:
```bash
curl http://127.0.0.1:9205/metrics
```
Repeatable production gate from the `Server` directory:
```bash
PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \
REQUIRE_METRICS=true \
./scripts/verify-rpi-production.sh
```
The checker validates:
- Docker Compose config rendering.
- Local presence of the pinned image.
- Image metadata from `qsfera version --skip-services`.
- Container running state, Docker health status, and container image tag.
- Docker root location, root filesystem free space, and data filesystem free space.
- Local and optional public `/status.php` JSON invariants:
`installed=true`, `maintenance=false`, `needsDbUpgrade=false`,
`edition=stable`, and `productversion=7.0.0`.
- Optional Prometheus metrics reachability.
The default free-space thresholds are 5 GiB for `/` and 20 GiB for `/mnt/data`.
Override them with `MIN_ROOT_AVAILABLE_KB` and `MIN_DATA_AVAILABLE_KB` when the
host has a documented capacity plan.
## Data Location
Docker bind mounts:
@@ -118,3 +159,94 @@ qsfera.kusoft.xyz {
}
}
```
Do not proxy `127.0.0.1:9205` to the public Internet. If a remote Prometheus
server needs to scrape it, connect through a private tunnel or bind it to an
internal-only interface and set `PROXY_DEBUG_TOKEN`.
## Monitoring And Alerts
Minimum checks for the Raspberry Pi deployment:
- Run `./scripts/verify-rpi-production.sh` from the `Server` directory by cron
or systemd timer every 5 minutes with `PUBLIC_STATUS_URL` set to the public
QSfera URL.
- Alert when the script exits non-zero.
- Alert on any `WARN:` line that persists for more than one day, especially
Docker root outside `/mnt/data` or unavailable metrics.
- Scrape `http://127.0.0.1:9205/metrics` locally or through a private tunnel.
- Keep public HTTPS probing separate from local loopback probing so reverse-proxy
and backend failures are distinguishable.
- Keep `OC_LOG_LEVEL=info` for production; raise to `debug` only for a bounded
incident window.
For larger deployments, the full compose example already contains tracing and
metrics wiring in `devtools/deployments/qsfera_full/monitoring_tracing/`.
## Backups And Restore Drill
Back up the two bind-mounted paths from `.env`:
- `QSFERA_RPI_CONFIG_PATH`
- `QSFERA_RPI_DATA_PATH`
Create an archive before every server image rollout:
```bash
stamp="$(date -u +%Y%m%d-%H%M%S)"
backup="/mnt/data/qsfera/backups/cloud-server-${stamp}.tar.gz"
sudo tar -C /mnt/data/qsfera/cloud-server -czf "$backup" config data
sudo tar -tzf "$backup" >/dev/null
```
Run a restore drill after backup automation changes and at least monthly:
```bash
stamp="$(date -u +%Y%m%d-%H%M%S)"
restore_root="/mnt/data/qsfera/restore-drills/${stamp}"
sudo mkdir -p "$restore_root"
sudo tar -C "$restore_root" -xzf "$backup"
sudo test -d "$restore_root/config"
sudo test -d "$restore_root/data"
```
For a full service restore test, copy `.env` to a drill-only file, point
`QSFERA_RPI_CONFIG_PATH` and `QSFERA_RPI_DATA_PATH` at the extracted directories,
bind `QSFERA_RPI_HOST_PORT` to an unused loopback port, run
`docker compose -f docker-compose.rpi5.yml --env-file .env.restore config`, and
only then start the drill container. Do not run a restore drill against the live
config/data paths.
## Docker Data-Root And Retention
Check where Docker stores images, layers, and build cache:
```bash
docker info --format '{{.DockerRootDir}}'
docker system df
```
Preferred production state on this Pi is Docker data-root under `/mnt/data`,
because the QSfera bind-mounted data is already stored there. Move it only during
a maintenance window:
```bash
sudo systemctl stop docker
sudo mkdir -p /mnt/data/docker
sudo rsync -aHAX --numeric-ids /var/lib/docker/ /mnt/data/docker/
printf '{\n "data-root": "/mnt/data/docker"\n}\n' | sudo tee /etc/docker/daemon.json
sudo systemctl start docker
docker info --format '{{.DockerRootDir}}'
docker compose -f docker-compose.rpi5.yml ps
```
If moving Docker data-root is postponed, keep conservative retention active:
```bash
docker builder prune --filter 'until=168h'
docker image prune --filter 'until=168h'
docker system df
```
Do not run `docker system prune --volumes` on this host unless a current backup
has been verified and the target volumes have been explicitly reviewed.
+273
View File
@@ -0,0 +1,273 @@
#!/usr/bin/env sh
set -eu
COMPOSE_FILE="${COMPOSE_FILE:-docker-compose.rpi5.yml}"
ENV_FILE="${ENV_FILE:-.env}"
SERVICE="${SERVICE:-qsfera-cloud-server}"
CONTAINER="${CONTAINER:-qsfera-cloud-server}"
EXPECTED_IMAGE="${EXPECTED_IMAGE:-qsfera-cloud-server:7.0.0-rpi5-stable}"
EXPECTED_VERSION="${EXPECTED_VERSION:-7.0.0}"
EXPECTED_EDITION="${EXPECTED_EDITION:-stable}"
LOCAL_STATUS_URL="${LOCAL_STATUS_URL:-https://127.0.0.1:9200/status.php}"
PUBLIC_STATUS_URL="${PUBLIC_STATUS_URL:-}"
METRICS_URL="${METRICS_URL:-http://127.0.0.1:9205/metrics}"
REQUIRE_METRICS="${REQUIRE_METRICS:-false}"
VERIFY_IMAGE_VERSION="${VERIFY_IMAGE_VERSION:-true}"
ROOT_PATH="${ROOT_PATH:-/}"
DATA_PATH="${DATA_PATH:-/mnt/data}"
MIN_ROOT_AVAILABLE_KB="${MIN_ROOT_AVAILABLE_KB:-5242880}"
MIN_DATA_AVAILABLE_KB="${MIN_DATA_AVAILABLE_KB:-20971520}"
EXPECTED_DOCKER_ROOT_PREFIX="${EXPECTED_DOCKER_ROOT_PREFIX:-/mnt/data}"
STRICT_DOCKER_ROOT="${STRICT_DOCKER_ROOT:-false}"
errors=0
warnings=0
tmp_dir="$(mktemp -d)"
cleanup() {
rm -rf "$tmp_dir"
}
trap cleanup EXIT INT TERM
ok() {
echo "OK: $*"
}
warn() {
warnings=$((warnings + 1))
echo "WARN: $*" >&2
}
error() {
errors=$((errors + 1))
echo "ERROR: $*" >&2
}
is_true() {
case "$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" in
true|1|yes|on)
return 0
;;
*)
return 1
;;
esac
}
require_command() {
if command -v "$1" >/dev/null 2>&1; then
ok "Command available: $1"
else
error "Required command is missing: $1"
fi
}
compose() {
if [ -n "$ENV_FILE" ] && [ -f "$ENV_FILE" ]; then
docker compose --env-file "$ENV_FILE" -f "$COMPOSE_FILE" "$@"
else
docker compose -f "$COMPOSE_FILE" "$@"
fi
}
available_kb() {
df -Pk "$1" 2>/dev/null | awk 'NR == 2 { print $4 }'
}
format_gib() {
awk -v kb="$1" 'BEGIN { printf "%.1f", kb / 1048576 }'
}
check_disk() {
path="$1"
minimum_kb="$2"
label="$3"
current_kb="$(available_kb "$path")"
if [ -z "$current_kb" ]; then
error "Cannot read free disk space for $label at $path."
return
fi
if [ "$current_kb" -ge "$minimum_kb" ]; then
ok "$label free space $(format_gib "$current_kb") GiB >= $(format_gib "$minimum_kb") GiB"
else
error "$label free space $(format_gib "$current_kb") GiB < $(format_gib "$minimum_kb") GiB"
fi
}
check_status_url() {
label="$1"
url="$2"
curl_opts="$3"
output="$tmp_dir/status.json"
if curl $curl_opts "$url" -o "$output"; then
ok "$label returned a successful HTTP response"
else
error "$label request failed: $url"
return
fi
if python3 - "$output" "$EXPECTED_EDITION" "$EXPECTED_VERSION" "$label" <<'PY'
import json
import sys
path, expected_edition, expected_version, label = sys.argv[1:]
with open(path, "r", encoding="utf-8") as handle:
payload = json.load(handle)
checks = [
("installed", payload.get("installed") is True, True, payload.get("installed")),
("maintenance", payload.get("maintenance") is False, False, payload.get("maintenance")),
("needsDbUpgrade", payload.get("needsDbUpgrade") is False, False, payload.get("needsDbUpgrade")),
("edition", payload.get("edition") == expected_edition, expected_edition, payload.get("edition")),
("productversion", payload.get("productversion") == expected_version, expected_version, payload.get("productversion")),
]
failed = False
for name, passed, expected, actual in checks:
if passed:
print(f"OK: {label} {name} = {actual}")
else:
print(f"ERROR: {label} {name}: expected {expected!r}, got {actual!r}", file=sys.stderr)
failed = True
if failed:
sys.exit(1)
PY
then
ok "$label JSON invariants match expected production state"
else
error "$label JSON invariants failed"
fi
}
check_metrics() {
output="$tmp_dir/metrics.txt"
if [ -n "${METRICS_BEARER_TOKEN:-}" ]; then
metrics_result=0
curl -fsS -H "Authorization: Bearer ${METRICS_BEARER_TOKEN}" "$METRICS_URL" -o "$output" || metrics_result=$?
else
metrics_result=0
curl -fsS "$METRICS_URL" -o "$output" || metrics_result=$?
fi
if [ "$metrics_result" -eq 0 ]; then
if grep -Eq '^(# HELP|# TYPE|go_|process_)' "$output"; then
ok "Metrics endpoint returned Prometheus-formatted content"
else
error "Metrics endpoint responded, but content does not look like Prometheus metrics."
fi
return
fi
if is_true "$REQUIRE_METRICS"; then
error "Metrics endpoint is required but unavailable: $METRICS_URL"
else
warn "Metrics endpoint unavailable: $METRICS_URL"
fi
}
for command_name in docker curl python3 awk df; do
require_command "$command_name"
done
if [ ! -f "$COMPOSE_FILE" ]; then
error "Compose file not found: $COMPOSE_FILE"
fi
if [ -n "$ENV_FILE" ] && [ ! -f "$ENV_FILE" ]; then
error "Environment file not found: $ENV_FILE"
fi
if [ "$errors" -eq 0 ]; then
if compose config >/dev/null; then
ok "Docker Compose config renders successfully for $COMPOSE_FILE"
else
error "Docker Compose config failed for $COMPOSE_FILE"
fi
fi
if docker image inspect "$EXPECTED_IMAGE" >/dev/null 2>&1; then
ok "Expected image exists locally: $EXPECTED_IMAGE"
else
error "Expected image is missing locally: $EXPECTED_IMAGE"
fi
if is_true "$VERIFY_IMAGE_VERSION"; then
version_output="$tmp_dir/image-version.txt"
if docker run --rm --entrypoint /usr/bin/qsfera "$EXPECTED_IMAGE" version --skip-services >"$version_output" 2>&1; then
if grep -q "Version: $EXPECTED_VERSION" "$version_output" && grep -q "Edition: $EXPECTED_EDITION" "$version_output"; then
ok "Image version metadata matches $EXPECTED_EDITION $EXPECTED_VERSION"
else
error "Image version metadata does not match expected $EXPECTED_EDITION $EXPECTED_VERSION."
sed 's/^/ /' "$version_output" >&2
fi
else
error "Cannot read image version metadata from $EXPECTED_IMAGE."
sed 's/^/ /' "$version_output" >&2
fi
fi
container_status="$(docker inspect -f '{{.State.Status}}' "$CONTAINER" 2>/dev/null || true)"
if [ "$container_status" = "running" ]; then
ok "Container $CONTAINER is running"
else
error "Container $CONTAINER status is '${container_status:-missing}', expected running."
fi
container_health="$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$CONTAINER" 2>/dev/null || true)"
case "$container_health" in
healthy)
ok "Container $CONTAINER health is healthy"
;;
none)
warn "Container $CONTAINER has no Docker healthcheck result."
;;
*)
error "Container $CONTAINER health is '${container_health:-missing}', expected healthy."
;;
esac
container_image="$(docker inspect -f '{{.Config.Image}}' "$CONTAINER" 2>/dev/null || true)"
if [ "$container_image" = "$EXPECTED_IMAGE" ]; then
ok "Container image is $EXPECTED_IMAGE"
else
error "Container image is '${container_image:-missing}', expected $EXPECTED_IMAGE."
fi
docker_root="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)"
case "$docker_root" in
"$EXPECTED_DOCKER_ROOT_PREFIX"|"$EXPECTED_DOCKER_ROOT_PREFIX"/*)
ok "Docker root is under $EXPECTED_DOCKER_ROOT_PREFIX: $docker_root"
;;
"")
error "Cannot read Docker root directory."
;;
*)
if is_true "$STRICT_DOCKER_ROOT"; then
error "Docker root is $docker_root, expected path under $EXPECTED_DOCKER_ROOT_PREFIX."
else
warn "Docker root is $docker_root, not under $EXPECTED_DOCKER_ROOT_PREFIX; keep image/build retention active or move data-root during maintenance."
fi
;;
esac
check_disk "$ROOT_PATH" "$MIN_ROOT_AVAILABLE_KB" "Root filesystem"
check_disk "$DATA_PATH" "$MIN_DATA_AVAILABLE_KB" "Data filesystem"
check_status_url "Local server status" "$LOCAL_STATUS_URL" "-kfsS"
if [ -n "$PUBLIC_STATUS_URL" ]; then
check_status_url "Public server status" "$PUBLIC_STATUS_URL" "-fsS"
fi
check_metrics
if [ "$errors" -gt 0 ]; then
echo "QSfera Raspberry Pi production verification failed: $errors error(s), $warnings warning(s)." >&2
exit 1
fi
echo "QSfera Raspberry Pi production verification passed: $warnings warning(s)."