QSfera/Server/docs/raspberry-pi-5-docker.md

# Raspberry Pi 5 Docker Run

This server repository contains a Dockerized QSfera path for Raspberry Pi 5 and other
64-bit ARM Linux hosts.

Files used for this deployment:

- `Dockerfile`
- `docker-compose.rpi5.yml`
- `.env.example`

## What This Runs

- QSfera server from this `Server` source tree.
- A pinned production image tag by default: `qsfera-cloud-server:7.0.0-rpi5-stable`.
- Build metadata injected into the binary through Docker build args:
  `QSFERA_VERSION`, `QSFERA_EDITION`, and `QSFERA_BUILD_DATE`.
- QSfera config persisted under `/etc/qsfera` inside the container.
- QSfera data persisted under `/var/lib/qsfera` inside the container.
- Host-side config and data paths controlled by `QSFERA_RPI_CONFIG_PATH` and
  `QSFERA_RPI_DATA_PATH`.
- Host-side backend port bound only to `127.0.0.1:9200`; external access should go
  through the reverse proxy and HTTPS domain.
- Host-side proxy debug/metrics port bound only to `127.0.0.1:9205`.

## Prerequisites On Raspberry Pi 5

1. Install Docker Engine using the official Docker docs: `https://docs.docker.com/engine/install/debian/`.
2. Install the Docker Compose plugin using the official Docker docs: `https://docs.docker.com/compose/install/`.
3. Use a 64-bit OS. The compose file sets `platform: linux/arm64`, and the Dockerfile builds with `TARGETARCH=arm64`.
4. Put `QSFERA_RPI_CONFIG_PATH` and `QSFERA_RPI_DATA_PATH` on durable storage, not on a nearly full SD-card root filesystem.
5. Set `OC_URL` to the public HTTPS URL used by clients and the reverse proxy.
6. Keep `QSFERA_EDITION=stable` and a concrete `QSFERA_VERSION` for production
   rollouts. Leaving those unset falls back to the defaults in the compose file.
7. Keep Docker image and builder retention under control. The current Raspberry Pi
   deployment stores QSfera data on `/mnt/data`; Docker's own `data-root` must
   either be moved there during a maintenance window or pruned on a schedule.

## Start

Run from the `Server` directory:

```bash
cp .env.example .env
docker compose -f docker-compose.rpi5.yml up -d --build
```

Verify the image metadata before rollout:

```bash
docker compose -f docker-compose.rpi5.yml build
docker run --rm --entrypoint /usr/bin/qsfera qsfera-cloud-server:7.0.0-rpi5-stable version --skip-services
```

Run the production checker from the `Server` directory after every rollout:

```bash
PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \
  ./scripts/verify-rpi-production.sh
```

## Stop

```bash
docker compose -f docker-compose.rpi5.yml down
```

## Logs

```bash
docker compose -f docker-compose.rpi5.yml logs -f qsfera-cloud-server
```

## Update After Pulling New Code

```bash
docker compose -f docker-compose.rpi5.yml up -d --build
```

## Verify

Local backend status endpoint:

```bash
curl -k https://127.0.0.1:9200/status.php
```

External reverse-proxied status endpoint:

```bash
curl https://qsfera.example.com/status.php
```

Local Prometheus metrics endpoint:

```bash
curl http://127.0.0.1:9205/metrics
```

Repeatable production gate from the `Server` directory:

```bash
PUBLIC_STATUS_URL=https://qsfera.example.com/status.php \
  REQUIRE_METRICS=true \
  ./scripts/verify-rpi-production.sh
```

The checker validates:

- Docker Compose config rendering.
- Local presence of the pinned image.
- Image metadata from `qsfera version --skip-services`.
- Container running state, Docker health status, and container image tag.
- Docker root location, root filesystem free space, and data filesystem free space.
- Local and optional public `/status.php` JSON invariants:
  `installed=true`, `maintenance=false`, `needsDbUpgrade=false`,
  `edition=stable`, and `productversion=7.0.0`.
- Optional Prometheus metrics reachability.

The default free-space thresholds are 5 GiB for `/` and 20 GiB for `/mnt/data`.
Override them with `MIN_ROOT_AVAILABLE_KB` and `MIN_DATA_AVAILABLE_KB` when the
host has a documented capacity plan.

## Data Location

Docker bind mounts:

- host config: `${QSFERA_RPI_CONFIG_PATH}`
- container config: `/etc/qsfera`
- host data: `${QSFERA_RPI_DATA_PATH}`
- container data: `/var/lib/qsfera`

For the current Raspberry Pi deployment checked on 2026-06-09 after the metrics
rollout, the existing container uses:

- host config: `/mnt/data/qsfera/cloud-server/config`
- host data: `/mnt/data/qsfera/cloud-server/data`
- image: `qsfera-cloud-server:7.0.0-rpi5-stable`
- public URL: `https://qsfera.kusoft.xyz`
- loopback port: `127.0.0.1:9200`
- loopback metrics port: `127.0.0.1:9205`
- Docker data-root: `/mnt/data/docker`

## Reverse Proxy

The server is intentionally bound to loopback. Put Caddy, Nginx, Traefik, or another
reverse proxy in front of it for external HTTPS access.

The inspected Raspberry Pi uses Caddy with this QSfera route:

```caddyfile
qsfera.kusoft.xyz {
    encode zstd gzip
    handle {
        reverse_proxy https://127.0.0.1:9200 {
            transport http {
                tls_insecure_skip_verify
                versions 1.1
            }
        }
    }
}
```

Do not proxy `127.0.0.1:9205` to the public Internet. If a remote Prometheus
server needs to scrape it, connect through a private tunnel or bind it to an
internal-only interface and set `PROXY_DEBUG_TOKEN`.

## Monitoring And Alerts

Install the production checker timer from the `Server` directory:

```bash
sudo ./scripts/install-rpi-production-monitoring.sh
```

The installer writes:

- `/etc/systemd/system/qsfera-rpi-production-check.service`
- `/etc/systemd/system/qsfera-rpi-production-check.timer`
- `/etc/systemd/system/qsfera-rpi-production-check-alert@.service`
- `/etc/qsfera/rpi-production-check.env`

The timer runs `./scripts/verify-rpi-production.sh` every 5 minutes. The service
uses `OnFailure=qsfera-rpi-production-check-alert@%n.service`; the alert service
logs the failed unit and recent journal output through `systemd-cat`. To forward
alerts off-host, set `QSFERA_ALERT_WEBHOOK_URL` in
`/etc/qsfera/rpi-production-check.env`; `alert-rpi-production-check-failure.sh`
sends a JSON `POST` when that variable is set.

For the current Raspberry Pi deployment checked on 2026-06-09, the timer was
installed under `/etc/systemd/system`, `systemctl is-enabled` returned
`enabled`, `systemctl is-active` returned `active`, and the checker service
reported `Result=success`, `ExecMainStatus=0`, and `ActiveState=inactive` after
the initial run. `systemctl list-timers qsfera-rpi-production-check.timer`
reported the next run 5 minutes after the previous run.

Verify the timer:

```bash
systemctl list-timers qsfera-rpi-production-check.timer --no-pager
systemctl status qsfera-rpi-production-check.timer --no-pager
systemctl show qsfera-rpi-production-check.service -p Result -p ExecMainStatus -p ActiveState
journalctl -u qsfera-rpi-production-check.service -n 80 --no-pager
```

Minimum monitoring expectations:

- Keep `qsfera-rpi-production-check.timer` enabled and active.
- Investigate any non-zero `ExecMainStatus` from
  `qsfera-rpi-production-check.service`.
- Investigate any `WARN:` line that persists for more than one day, especially
  Docker root outside `/mnt/data` or unavailable metrics.
- Configure `QSFERA_ALERT_WEBHOOK_URL` when off-host alert delivery is required;
  without it, failures are still routed to the systemd failure service and
  journal, but not to an external notification endpoint.
- Scrape `http://127.0.0.1:9205/metrics` locally or through a private tunnel.
- Keep public HTTPS probing separate from local loopback probing so reverse-proxy
  and backend failures are distinguishable.
- Keep `OC_LOG_LEVEL=info` for production; raise to `debug` only for a bounded
  incident window.

For larger deployments, the full compose example already contains tracing and
metrics wiring in `devtools/deployments/qsfera_full/monitoring_tracing/`.

## Backups And Restore Drill

Back up the two bind-mounted paths from `.env`:

- `QSFERA_RPI_CONFIG_PATH`
- `QSFERA_RPI_DATA_PATH`

Create an archive before every server image rollout:

```bash
stamp="$(date -u +%Y%m%d-%H%M%S)"
backup="/mnt/data/qsfera/backups/cloud-server-${stamp}.tar.gz"
sudo tar -C /mnt/data/qsfera/cloud-server -czf "$backup" config data
sudo tar -tzf "$backup" >/dev/null
```

Run a restore drill after backup automation changes and at least monthly:

```bash
stamp="$(date -u +%Y%m%d-%H%M%S)"
restore_root="/mnt/data/qsfera/restore-drills/${stamp}"
sudo mkdir -p "$restore_root"
sudo tar -C "$restore_root" -xzf "$backup"
sudo test -d "$restore_root/config"
sudo test -d "$restore_root/data"
```

For a full service restore test, copy `.env` to a drill-only file, point
`QSFERA_RPI_CONFIG_PATH` and `QSFERA_RPI_DATA_PATH` at the extracted directories,
bind `QSFERA_RPI_HOST_PORT` to an unused loopback port, run
`docker compose -f docker-compose.rpi5.yml --env-file .env.restore config`, and
only then start the drill container. Do not run a restore drill against the live
config/data paths.

## Docker Data-Root And Retention

Check where Docker stores images, layers, and build cache:

```bash
docker info --format '{{.DockerRootDir}}'
docker system df
```

Preferred production state on this Pi is Docker data-root under `/mnt/data`,
because the QSfera bind-mounted data is already stored there. Move it only during
a maintenance window:

```bash
sudo systemctl stop docker
sudo mkdir -p /mnt/data/docker
sudo rsync -aHAX --numeric-ids /var/lib/docker/ /mnt/data/docker/
printf '{\n  "data-root": "/mnt/data/docker"\n}\n' | sudo tee /etc/docker/daemon.json
sudo systemctl start docker
docker info --format '{{.DockerRootDir}}'
docker compose -f docker-compose.rpi5.yml ps
```

If moving Docker data-root is postponed, keep conservative retention active:

```bash
docker builder prune --filter 'until=168h'
docker image prune --filter 'until=168h'
docker system df
```

Do not run `docker system prune --volumes` on this host unless a current backup
has been verified and the target volumes have been explicitly reviewed.