diff --git a/PRODUCTION_READINESS.md b/PRODUCTION_READINESS.md index 88f977a0..af2503a1 100644 --- a/PRODUCTION_READINESS.md +++ b/PRODUCTION_READINESS.md @@ -40,6 +40,7 @@ Date: 2026-06-08. - Windows/Desktop now has a configured HTTPS update feed at `https://qsfera.kusoft.xyz/desktop/updates/stable.xml`; the feed returned HTTP `200` on 2026-06-08 and contains the XML elements parsed by `Desktop/src/gui/updater/updateinfo.cpp`. - Added `scripts/verify-production-state.ps1` as a repeatable live production endpoint gate for the public server status, Android Argus manifest/download, and Desktop update feed. - `scripts/verify-production-state.ps1 -VerifyAndroidDownload` passed on 2026-06-08: server `edition=stable`/`productversion=7.0.0`, Android `1.3.7`/`35`, downloaded APK size `12220848`, APK SHA-256 `c9c044c1332f8d973c64502ad2e3227ad92398101e7354ce9dbe7f708e7e693b`, and valid Desktop no-update feed. +- Added `Server/scripts/verify-rpi-production.sh`, declared a loopback-only Pi metrics port in `Server/docker-compose.rpi5.yml`, and expanded the Raspberry Pi Docker runbook with local/public status checks, container/image checks, Prometheus metrics verification, backup/restore drill steps, and Docker data-root/retention planning. ## Raspberry Pi 5 Check @@ -76,6 +77,7 @@ Date: 2026-06-08. - Add Google Photos-style backup status UX: overall status, per-item status, queued/paused/permanent-failure states, and low-storage messaging. - Add mobile-data controls comparable to Google Photos/Yandex Disk beyond the Wi-Fi-only default: mobile-data limit and no-video-over-mobile-data options. - Store auto-upload sync cursors per source folder rather than one timestamp for all folders. -- Add production monitoring runbook and compose verification for metrics, tracing, alerts, backups, and restore drills. -- Move Docker build cache and/or Docker data-root planning for the Pi to `/mnt/data`, or add documented retention, so future server image rebuilds cannot fill the root filesystem again. +- Deploy the Raspberry Pi metrics endpoint change with `docker compose -f docker-compose.rpi5.yml up -d`, then run `Server/scripts/verify-rpi-production.sh` on the Pi with `PUBLIC_STATUS_URL=https://qsfera.kusoft.xyz/status.php` and `REQUIRE_METRICS=true`. +- Add an actual scheduler/alert target for `Server/scripts/verify-rpi-production.sh`; the runbook defines what to check, but no cron/systemd timer is committed yet. +- Decide whether to move Docker `data-root` to `/mnt/data/docker` or keep scheduled builder/image retention. The runbook documents both paths, but the host-level Docker daemon setting still needs an operator maintenance window. - Verify sharing controls against Google Drive and Yandex Disk expectations: link expiration, password, disable downloads, organization-only access, and role inheritance. diff --git a/Server/.env.example b/Server/.env.example index 01359aba..146c44d9 100644 --- a/Server/.env.example +++ b/Server/.env.example @@ -3,6 +3,7 @@ QSFERA_VERSION=7.0.0 QSFERA_EDITION=stable QSFERA_BUILD_DATE=20260608 QSFERA_RPI_HOST_PORT=9200 +QSFERA_RPI_DEBUG_HOST_PORT=9205 QSFERA_RPI_CONFIG_PATH=/mnt/data/qsfera/cloud-server/config QSFERA_RPI_DATA_PATH=/mnt/data/qsfera/cloud-server/data @@ -10,4 +11,6 @@ OC_URL=https://qsfera.example.com OC_LOG_LEVEL=info OC_INSECURE=false PROXY_TLS=true +PROXY_DEBUG_ADDR=0.0.0.0:9205 +PROXY_DEBUG_TOKEN= FRONTEND_CHECK_FOR_UPDATES=false diff --git a/Server/docker-compose.rpi5.yml b/Server/docker-compose.rpi5.yml index b00d1149..98e69a60 100644 --- a/Server/docker-compose.rpi5.yml +++ b/Server/docker-compose.rpi5.yml @@ -25,8 +25,11 @@ services: OC_URL: ${OC_URL:?OC_URL must be set in .env} OC_INSECURE: ${OC_INSECURE:-false} PROXY_TLS: ${PROXY_TLS:-true} + PROXY_DEBUG_ADDR: ${PROXY_DEBUG_ADDR:-0.0.0.0:9205} + PROXY_DEBUG_TOKEN: ${PROXY_DEBUG_TOKEN:-} ports: - "127.0.0.1:${QSFERA_RPI_HOST_PORT:-9200}:9200" + - "127.0.0.1:${QSFERA_RPI_DEBUG_HOST_PORT:-9205}:9205" volumes: - ${QSFERA_RPI_CONFIG_PATH:?QSFERA_RPI_CONFIG_PATH must be set in .env}:/etc/qsfera - ${QSFERA_RPI_DATA_PATH:?QSFERA_RPI_DATA_PATH must be set in .env}:/var/lib/qsfera diff --git a/Server/docs/raspberry-pi-5-docker.md b/Server/docs/raspberry-pi-5-docker.md index 2417a2ef..0163b9f5 100644 --- a/Server/docs/raspberry-pi-5-docker.md +++ b/Server/docs/raspberry-pi-5-docker.md @@ -21,6 +21,7 @@ Files used for this deployment: `QSFERA_RPI_DATA_PATH`. - Host-side backend port bound only to `127.0.0.1:9200`; external access should go through the reverse proxy and HTTPS domain. +- Host-side proxy debug/metrics port bound only to `127.0.0.1:9205`. ## Prerequisites On Raspberry Pi 5 @@ -31,6 +32,9 @@ Files used for this deployment: 5. Set `OC_URL` to the public HTTPS URL used by clients and the reverse proxy. 6. Keep `QSFERA_EDITION=stable` and a concrete `QSFERA_VERSION` for production rollouts. Leaving those unset falls back to the defaults in the compose file. +7. Keep Docker image and builder retention under control. The current Raspberry Pi + deployment stores QSfera data on `/mnt/data`; Docker's own `data-root` must + either be moved there during a maintenance window or pruned on a schedule. ## Start @@ -48,6 +52,13 @@ docker compose -f docker-compose.rpi5.yml build docker run --rm --entrypoint /usr/bin/qsfera qsfera-cloud-server:7.0.0-rpi5-stable version --skip-services ``` +Run the production checker from the `Server` directory after every rollout: + +```bash +PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \ + ./scripts/verify-rpi-production.sh +``` + ## Stop ```bash @@ -80,6 +91,36 @@ External reverse-proxied status endpoint: curl https://qsfera.example.com/status.php ``` +Local Prometheus metrics endpoint: + +```bash +curl http://127.0.0.1:9205/metrics +``` + +Repeatable production gate from the `Server` directory: + +```bash +PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \ + REQUIRE_METRICS=true \ + ./scripts/verify-rpi-production.sh +``` + +The checker validates: + +- Docker Compose config rendering. +- Local presence of the pinned image. +- Image metadata from `qsfera version --skip-services`. +- Container running state, Docker health status, and container image tag. +- Docker root location, root filesystem free space, and data filesystem free space. +- Local and optional public `/status.php` JSON invariants: + `installed=true`, `maintenance=false`, `needsDbUpgrade=false`, + `edition=stable`, and `productversion=7.0.0`. +- Optional Prometheus metrics reachability. + +The default free-space thresholds are 5 GiB for `/` and 20 GiB for `/mnt/data`. +Override them with `MIN_ROOT_AVAILABLE_KB` and `MIN_DATA_AVAILABLE_KB` when the +host has a documented capacity plan. + ## Data Location Docker bind mounts: @@ -118,3 +159,94 @@ qsfera.kusoft.xyz { } } ``` + +Do not proxy `127.0.0.1:9205` to the public Internet. If a remote Prometheus +server needs to scrape it, connect through a private tunnel or bind it to an +internal-only interface and set `PROXY_DEBUG_TOKEN`. + +## Monitoring And Alerts + +Minimum checks for the Raspberry Pi deployment: + +- Run `./scripts/verify-rpi-production.sh` from the `Server` directory by cron + or systemd timer every 5 minutes with `PUBLIC_STATUS_URL` set to the public + QSfera URL. +- Alert when the script exits non-zero. +- Alert on any `WARN:` line that persists for more than one day, especially + Docker root outside `/mnt/data` or unavailable metrics. +- Scrape `http://127.0.0.1:9205/metrics` locally or through a private tunnel. +- Keep public HTTPS probing separate from local loopback probing so reverse-proxy + and backend failures are distinguishable. +- Keep `OC_LOG_LEVEL=info` for production; raise to `debug` only for a bounded + incident window. + +For larger deployments, the full compose example already contains tracing and +metrics wiring in `devtools/deployments/qsfera_full/monitoring_tracing/`. + +## Backups And Restore Drill + +Back up the two bind-mounted paths from `.env`: + +- `QSFERA_RPI_CONFIG_PATH` +- `QSFERA_RPI_DATA_PATH` + +Create an archive before every server image rollout: + +```bash +stamp="$(date -u +%Y%m%d-%H%M%S)" +backup="/mnt/data/qsfera/backups/cloud-server-${stamp}.tar.gz" +sudo tar -C /mnt/data/qsfera/cloud-server -czf "$backup" config data +sudo tar -tzf "$backup" >/dev/null +``` + +Run a restore drill after backup automation changes and at least monthly: + +```bash +stamp="$(date -u +%Y%m%d-%H%M%S)" +restore_root="/mnt/data/qsfera/restore-drills/${stamp}" +sudo mkdir -p "$restore_root" +sudo tar -C "$restore_root" -xzf "$backup" +sudo test -d "$restore_root/config" +sudo test -d "$restore_root/data" +``` + +For a full service restore test, copy `.env` to a drill-only file, point +`QSFERA_RPI_CONFIG_PATH` and `QSFERA_RPI_DATA_PATH` at the extracted directories, +bind `QSFERA_RPI_HOST_PORT` to an unused loopback port, run +`docker compose -f docker-compose.rpi5.yml --env-file .env.restore config`, and +only then start the drill container. Do not run a restore drill against the live +config/data paths. + +## Docker Data-Root And Retention + +Check where Docker stores images, layers, and build cache: + +```bash +docker info --format '{{.DockerRootDir}}' +docker system df +``` + +Preferred production state on this Pi is Docker data-root under `/mnt/data`, +because the QSfera bind-mounted data is already stored there. Move it only during +a maintenance window: + +```bash +sudo systemctl stop docker +sudo mkdir -p /mnt/data/docker +sudo rsync -aHAX --numeric-ids /var/lib/docker/ /mnt/data/docker/ +printf '{\n "data-root": "/mnt/data/docker"\n}\n' | sudo tee /etc/docker/daemon.json +sudo systemctl start docker +docker info --format '{{.DockerRootDir}}' +docker compose -f docker-compose.rpi5.yml ps +``` + +If moving Docker data-root is postponed, keep conservative retention active: + +```bash +docker builder prune --filter 'until=168h' +docker image prune --filter 'until=168h' +docker system df +``` + +Do not run `docker system prune --volumes` on this host unless a current backup +has been verified and the target volumes have been explicitly reviewed. diff --git a/Server/scripts/verify-rpi-production.sh b/Server/scripts/verify-rpi-production.sh new file mode 100644 index 00000000..be4a5b4c --- /dev/null +++ b/Server/scripts/verify-rpi-production.sh @@ -0,0 +1,273 @@ +#!/usr/bin/env sh +set -eu + +COMPOSE_FILE="${COMPOSE_FILE:-docker-compose.rpi5.yml}" +ENV_FILE="${ENV_FILE:-.env}" +SERVICE="${SERVICE:-qsfera-cloud-server}" +CONTAINER="${CONTAINER:-qsfera-cloud-server}" +EXPECTED_IMAGE="${EXPECTED_IMAGE:-qsfera-cloud-server:7.0.0-rpi5-stable}" +EXPECTED_VERSION="${EXPECTED_VERSION:-7.0.0}" +EXPECTED_EDITION="${EXPECTED_EDITION:-stable}" +LOCAL_STATUS_URL="${LOCAL_STATUS_URL:-https://127.0.0.1:9200/status.php}" +PUBLIC_STATUS_URL="${PUBLIC_STATUS_URL:-}" +METRICS_URL="${METRICS_URL:-http://127.0.0.1:9205/metrics}" +REQUIRE_METRICS="${REQUIRE_METRICS:-false}" +VERIFY_IMAGE_VERSION="${VERIFY_IMAGE_VERSION:-true}" +ROOT_PATH="${ROOT_PATH:-/}" +DATA_PATH="${DATA_PATH:-/mnt/data}" +MIN_ROOT_AVAILABLE_KB="${MIN_ROOT_AVAILABLE_KB:-5242880}" +MIN_DATA_AVAILABLE_KB="${MIN_DATA_AVAILABLE_KB:-20971520}" +EXPECTED_DOCKER_ROOT_PREFIX="${EXPECTED_DOCKER_ROOT_PREFIX:-/mnt/data}" +STRICT_DOCKER_ROOT="${STRICT_DOCKER_ROOT:-false}" + +errors=0 +warnings=0 +tmp_dir="$(mktemp -d)" + +cleanup() { + rm -rf "$tmp_dir" +} +trap cleanup EXIT INT TERM + +ok() { + echo "OK: $*" +} + +warn() { + warnings=$((warnings + 1)) + echo "WARN: $*" >&2 +} + +error() { + errors=$((errors + 1)) + echo "ERROR: $*" >&2 +} + +is_true() { + case "$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" in + true|1|yes|on) + return 0 + ;; + *) + return 1 + ;; + esac +} + +require_command() { + if command -v "$1" >/dev/null 2>&1; then + ok "Command available: $1" + else + error "Required command is missing: $1" + fi +} + +compose() { + if [ -n "$ENV_FILE" ] && [ -f "$ENV_FILE" ]; then + docker compose --env-file "$ENV_FILE" -f "$COMPOSE_FILE" "$@" + else + docker compose -f "$COMPOSE_FILE" "$@" + fi +} + +available_kb() { + df -Pk "$1" 2>/dev/null | awk 'NR == 2 { print $4 }' +} + +format_gib() { + awk -v kb="$1" 'BEGIN { printf "%.1f", kb / 1048576 }' +} + +check_disk() { + path="$1" + minimum_kb="$2" + label="$3" + current_kb="$(available_kb "$path")" + + if [ -z "$current_kb" ]; then + error "Cannot read free disk space for $label at $path." + return + fi + + if [ "$current_kb" -ge "$minimum_kb" ]; then + ok "$label free space $(format_gib "$current_kb") GiB >= $(format_gib "$minimum_kb") GiB" + else + error "$label free space $(format_gib "$current_kb") GiB < $(format_gib "$minimum_kb") GiB" + fi +} + +check_status_url() { + label="$1" + url="$2" + curl_opts="$3" + output="$tmp_dir/status.json" + + if curl $curl_opts "$url" -o "$output"; then + ok "$label returned a successful HTTP response" + else + error "$label request failed: $url" + return + fi + + if python3 - "$output" "$EXPECTED_EDITION" "$EXPECTED_VERSION" "$label" <<'PY' +import json +import sys + +path, expected_edition, expected_version, label = sys.argv[1:] +with open(path, "r", encoding="utf-8") as handle: + payload = json.load(handle) + +checks = [ + ("installed", payload.get("installed") is True, True, payload.get("installed")), + ("maintenance", payload.get("maintenance") is False, False, payload.get("maintenance")), + ("needsDbUpgrade", payload.get("needsDbUpgrade") is False, False, payload.get("needsDbUpgrade")), + ("edition", payload.get("edition") == expected_edition, expected_edition, payload.get("edition")), + ("productversion", payload.get("productversion") == expected_version, expected_version, payload.get("productversion")), +] + +failed = False +for name, passed, expected, actual in checks: + if passed: + print(f"OK: {label} {name} = {actual}") + else: + print(f"ERROR: {label} {name}: expected {expected!r}, got {actual!r}", file=sys.stderr) + failed = True + +if failed: + sys.exit(1) +PY + then + ok "$label JSON invariants match expected production state" + else + error "$label JSON invariants failed" + fi +} + +check_metrics() { + output="$tmp_dir/metrics.txt" + + if [ -n "${METRICS_BEARER_TOKEN:-}" ]; then + metrics_result=0 + curl -fsS -H "Authorization: Bearer ${METRICS_BEARER_TOKEN}" "$METRICS_URL" -o "$output" || metrics_result=$? + else + metrics_result=0 + curl -fsS "$METRICS_URL" -o "$output" || metrics_result=$? + fi + + if [ "$metrics_result" -eq 0 ]; then + if grep -Eq '^(# HELP|# TYPE|go_|process_)' "$output"; then + ok "Metrics endpoint returned Prometheus-formatted content" + else + error "Metrics endpoint responded, but content does not look like Prometheus metrics." + fi + return + fi + + if is_true "$REQUIRE_METRICS"; then + error "Metrics endpoint is required but unavailable: $METRICS_URL" + else + warn "Metrics endpoint unavailable: $METRICS_URL" + fi +} + +for command_name in docker curl python3 awk df; do + require_command "$command_name" +done + +if [ ! -f "$COMPOSE_FILE" ]; then + error "Compose file not found: $COMPOSE_FILE" +fi + +if [ -n "$ENV_FILE" ] && [ ! -f "$ENV_FILE" ]; then + error "Environment file not found: $ENV_FILE" +fi + +if [ "$errors" -eq 0 ]; then + if compose config >/dev/null; then + ok "Docker Compose config renders successfully for $COMPOSE_FILE" + else + error "Docker Compose config failed for $COMPOSE_FILE" + fi +fi + +if docker image inspect "$EXPECTED_IMAGE" >/dev/null 2>&1; then + ok "Expected image exists locally: $EXPECTED_IMAGE" +else + error "Expected image is missing locally: $EXPECTED_IMAGE" +fi + +if is_true "$VERIFY_IMAGE_VERSION"; then + version_output="$tmp_dir/image-version.txt" + if docker run --rm --entrypoint /usr/bin/qsfera "$EXPECTED_IMAGE" version --skip-services >"$version_output" 2>&1; then + if grep -q "Version: $EXPECTED_VERSION" "$version_output" && grep -q "Edition: $EXPECTED_EDITION" "$version_output"; then + ok "Image version metadata matches $EXPECTED_EDITION $EXPECTED_VERSION" + else + error "Image version metadata does not match expected $EXPECTED_EDITION $EXPECTED_VERSION." + sed 's/^/ /' "$version_output" >&2 + fi + else + error "Cannot read image version metadata from $EXPECTED_IMAGE." + sed 's/^/ /' "$version_output" >&2 + fi +fi + +container_status="$(docker inspect -f '{{.State.Status}}' "$CONTAINER" 2>/dev/null || true)" +if [ "$container_status" = "running" ]; then + ok "Container $CONTAINER is running" +else + error "Container $CONTAINER status is '${container_status:-missing}', expected running." +fi + +container_health="$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$CONTAINER" 2>/dev/null || true)" +case "$container_health" in + healthy) + ok "Container $CONTAINER health is healthy" + ;; + none) + warn "Container $CONTAINER has no Docker healthcheck result." + ;; + *) + error "Container $CONTAINER health is '${container_health:-missing}', expected healthy." + ;; +esac + +container_image="$(docker inspect -f '{{.Config.Image}}' "$CONTAINER" 2>/dev/null || true)" +if [ "$container_image" = "$EXPECTED_IMAGE" ]; then + ok "Container image is $EXPECTED_IMAGE" +else + error "Container image is '${container_image:-missing}', expected $EXPECTED_IMAGE." +fi + +docker_root="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || true)" +case "$docker_root" in + "$EXPECTED_DOCKER_ROOT_PREFIX"|"$EXPECTED_DOCKER_ROOT_PREFIX"/*) + ok "Docker root is under $EXPECTED_DOCKER_ROOT_PREFIX: $docker_root" + ;; + "") + error "Cannot read Docker root directory." + ;; + *) + if is_true "$STRICT_DOCKER_ROOT"; then + error "Docker root is $docker_root, expected path under $EXPECTED_DOCKER_ROOT_PREFIX." + else + warn "Docker root is $docker_root, not under $EXPECTED_DOCKER_ROOT_PREFIX; keep image/build retention active or move data-root during maintenance." + fi + ;; +esac + +check_disk "$ROOT_PATH" "$MIN_ROOT_AVAILABLE_KB" "Root filesystem" +check_disk "$DATA_PATH" "$MIN_DATA_AVAILABLE_KB" "Data filesystem" +check_status_url "Local server status" "$LOCAL_STATUS_URL" "-kfsS" + +if [ -n "$PUBLIC_STATUS_URL" ]; then + check_status_url "Public server status" "$PUBLIC_STATUS_URL" "-fsS" +fi + +check_metrics + +if [ "$errors" -gt 0 ]; then + echo "QSfera Raspberry Pi production verification failed: $errors error(s), $warnings warning(s)." >&2 + exit 1 +fi + +echo "QSfera Raspberry Pi production verification passed: $warnings warning(s)."