Implement background media ingestion and Pi deployment
This commit is contained in:
Курнат Андрей
+1
-11
@@ -649,17 +649,7 @@ def buildWebCache(ctx):
|
||||
|
||||
def testQsferaAndUploadResults(ctx):
|
||||
unit_pipeline = testQsfera(ctx)
|
||||
|
||||
######################################################################
|
||||
# The triggers have been disabled for now, since the govulncheck can #
|
||||
# not silence single, acceptable vulnerabilities. #
|
||||
# See https://github.com/owncloud/ocis/issues/9527 for more details. #
|
||||
# FIXME: RE-ENABLE THIS ASAP!!! #
|
||||
######################################################################
|
||||
|
||||
#security_scan = scanQsfera(ctx)
|
||||
#return [security_scan] + unit_pipeline + [scan_result_upload]
|
||||
return unit_pipeline
|
||||
return scanQsfera(ctx) + unit_pipeline
|
||||
|
||||
def testPipelines(ctx):
|
||||
pipelines = []
|
||||
|
||||
@@ -48,6 +48,19 @@ LABEL maintainer="QSfera" \
|
||||
org.opencontainers.image.documentation="" \
|
||||
org.opencontainers.image.source=""
|
||||
|
||||
RUN addgroup -g 1000 -S qsfera-group && \
|
||||
adduser -S --ingroup qsfera-group --uid 1000 qsfera-user --home /var/lib/qsfera && \
|
||||
mkdir -p /var/lib/qsfera/web/assets/apps /etc/qsfera && \
|
||||
chown -R qsfera-user:qsfera-group /var/lib/qsfera /etc/qsfera && \
|
||||
chmod -R 751 /var/lib/qsfera /etc/qsfera
|
||||
|
||||
VOLUME ["/var/lib/qsfera", "/etc/qsfera"]
|
||||
WORKDIR /var/lib/qsfera
|
||||
|
||||
USER 1000
|
||||
|
||||
EXPOSE 9200/tcp
|
||||
|
||||
ENTRYPOINT ["/usr/bin/qsfera"]
|
||||
CMD ["server"]
|
||||
|
||||
|
||||
+2
-2
@@ -210,11 +210,11 @@ protobuf:
|
||||
|
||||
.PHONY: golangci-lint
|
||||
golangci-lint: $(GOLANGCI_LINT)
|
||||
$(GOLANGCI_LINT) run --modules-download-mode vendor --timeout 15m0s --issues-exit-code 0 --out-format checkstyle > checkstyle.xml
|
||||
$(GOLANGCI_LINT) run --modules-download-mode vendor --timeout 15m0s --issues-exit-code 1 --out-format checkstyle > checkstyle.xml
|
||||
|
||||
.PHONY: ci-golangci-lint
|
||||
ci-golangci-lint:
|
||||
$(GOLANGCI_LINT) run --modules-download-mode vendor --timeout 15m0s --issues-exit-code 0 --out-format checkstyle > checkstyle.xml
|
||||
$(GOLANGCI_LINT) run --modules-download-mode vendor --timeout 15m0s --issues-exit-code 1 --out-format checkstyle > checkstyle.xml
|
||||
|
||||
.PHONY: golangci-lint-fix
|
||||
golangci-lint-fix: $(GOLANGCI_LINT)
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# Use an immutable release tag or digest that contains linux/arm64.
|
||||
QSFERA_IMAGE=
|
||||
|
||||
# Public URL used by the Android, desktop and web clients.
|
||||
QSFERA_URL=
|
||||
|
||||
# Bind to localhost when a reverse proxy runs on the Pi.
|
||||
QSFERA_BIND_IP=127.0.0.1
|
||||
QSFERA_PROXY_TLS=true
|
||||
QSFERA_INSECURE=false
|
||||
QSFERA_ENABLE_BASIC_AUTH=false
|
||||
QSFERA_LOG_LEVEL=info
|
||||
|
||||
# Do not reuse the example or demo password.
|
||||
QSFERA_ADMIN_PASSWORD=
|
||||
|
||||
# Absolute directories on persistent storage. An external SSD is recommended.
|
||||
QSFERA_CONFIG_DIR=
|
||||
QSFERA_DATA_DIR=
|
||||
|
||||
# Set after measuring RAM used by the OS and other services, for example 2g.
|
||||
QSFERA_MEMORY_LIMIT=
|
||||
@@ -0,0 +1,96 @@
|
||||
# Raspberry Pi deployment
|
||||
|
||||
This profile runs the single-container QSfera server on a 64-bit Raspberry Pi
|
||||
OS and persists configuration and user data outside the container. It expects a
|
||||
separate reverse proxy for a browser-trusted TLS certificate; by default port
|
||||
9200 is therefore published only on `127.0.0.1`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- `uname -m` must report `aarch64` or `arm64`.
|
||||
- Docker Engine with the Compose plugin must be installed.
|
||||
- The selected QSfera image tag or digest must contain `linux/arm64`.
|
||||
- The configuration and data directories must be on persistent storage and
|
||||
writable by UID/GID `1000:1000`, which is the non-root user in the image.
|
||||
|
||||
For an external disk mounted at `/mnt/qsfera`, create the directories with:
|
||||
|
||||
```sh
|
||||
sudo install -d -o 1000 -g 1000 -m 0750 /mnt/qsfera/config /mnt/qsfera/data
|
||||
findmnt /mnt/qsfera
|
||||
```
|
||||
|
||||
`findmnt` must show the expected external filesystem before the service starts.
|
||||
This prevents an unavailable disk from silently placing data on the Pi's root
|
||||
filesystem.
|
||||
|
||||
## Configuration
|
||||
|
||||
```sh
|
||||
cd Server/deployments/raspberry-pi
|
||||
cp .env.example .env
|
||||
chmod 600 .env
|
||||
```
|
||||
|
||||
Fill every empty value in `.env`. The Compose file deliberately has no default
|
||||
for the image, public URL, admin password, persistent paths or memory limit.
|
||||
`docker compose config` fails when any of these values is empty.
|
||||
|
||||
Choose `QSFERA_MEMORY_LIMIT` from measurements on the target Pi:
|
||||
|
||||
```text
|
||||
QSFERA_MEMORY_LIMIT = total RAM - OS reserve - other services reserve
|
||||
```
|
||||
|
||||
Use `free -h` for total/current host memory and `docker stats` for the other
|
||||
containers. Leave enough headroom for the kernel, filesystem cache and reverse
|
||||
proxy. QSfera's Go runtime then derives its own default `GOMEMLIMIT` from the
|
||||
container limit with a 0.9 ratio.
|
||||
|
||||
Keep `QSFERA_BIND_IP=127.0.0.1` when the reverse proxy runs on the same Pi. Set
|
||||
it to a reachable interface only when access to port 9200 is protected by an
|
||||
equivalent network and TLS design.
|
||||
|
||||
## Start and verify
|
||||
|
||||
```sh
|
||||
sh ./smoke-test.sh
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
The smoke test validates the Compose file, starts the service and waits for the
|
||||
container health check. Its default 180-second startup budget can be changed
|
||||
with `SMOKE_TIMEOUT_SECONDS`.
|
||||
|
||||
## Backup
|
||||
|
||||
The backup script stops QSfera, archives both persistent directories, writes a
|
||||
SHA-256 checksum and starts the service again:
|
||||
|
||||
```sh
|
||||
sh ./backup.sh /mnt/backups/qsfera
|
||||
```
|
||||
|
||||
Copy the resulting `.tar.gz` and `.sha256` files to storage that is independent
|
||||
of the Pi and its data disk.
|
||||
|
||||
## Restore
|
||||
|
||||
Restore replaces the current configuration and data. Check the selected archive
|
||||
and its checksum first, then run:
|
||||
|
||||
```sh
|
||||
RESTORE_CONFIRM=restore sh ./restore.sh /mnt/backups/qsfera/qsfera-TIMESTAMP.tar.gz
|
||||
sh ./smoke-test.sh
|
||||
```
|
||||
|
||||
The restore script verifies the checksum when the matching `.sha256` file is
|
||||
present and stops the service during replacement. It starts QSfera only after a
|
||||
successful extraction; after an error the service remains stopped so a partial
|
||||
data set is never served automatically.
|
||||
|
||||
## References
|
||||
|
||||
- [Raspberry Pi external storage documentation](https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#external-storage)
|
||||
- [Docker Compose service configuration](https://docs.docker.com/reference/compose-file/services/)
|
||||
- [Go garbage collector guide](https://go.dev/doc/gc-guide)
|
||||
@@ -0,0 +1,38 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
backup_root=${1:-./backups}
|
||||
timestamp=$(date -u +%Y%m%dT%H%M%SZ)
|
||||
archive="$backup_root/qsfera-$timestamp.tar.gz"
|
||||
temporary="$archive.incomplete"
|
||||
archive_directory=$(dirname "$archive")
|
||||
archive_name=$(basename "$archive")
|
||||
|
||||
umask 077
|
||||
mkdir -p "$backup_root"
|
||||
|
||||
compose() {
|
||||
docker compose "$@"
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
rm -f "$temporary"
|
||||
compose start qsfera >/dev/null
|
||||
}
|
||||
|
||||
compose config --quiet
|
||||
compose stop qsfera
|
||||
trap cleanup EXIT
|
||||
trap 'exit 1' HUP INT TERM
|
||||
|
||||
compose run --rm --no-deps -T --entrypoint /bin/sh qsfera -ec \
|
||||
'tar -C / -czf - etc/qsfera var/lib/qsfera' >"$temporary"
|
||||
mv "$temporary" "$archive"
|
||||
(cd "$archive_directory" && sha256sum "$archive_name" >"$archive_name.sha256")
|
||||
|
||||
compose start qsfera >/dev/null
|
||||
trap - EXIT HUP INT TERM
|
||||
|
||||
printf 'Backup: %s\nChecksum: %s\n' "$archive" "$archive.sha256"
|
||||
@@ -0,0 +1,47 @@
|
||||
name: qsfera-pi
|
||||
|
||||
services:
|
||||
qsfera:
|
||||
image: ${QSFERA_IMAGE:?Set QSFERA_IMAGE to a pinned ARM64 image tag}
|
||||
platform: linux/arm64
|
||||
init: true
|
||||
entrypoint:
|
||||
- /bin/sh
|
||||
command:
|
||||
- -ec
|
||||
- |
|
||||
if [ ! -f /etc/qsfera/qsfera.yaml ]; then
|
||||
qsfera init
|
||||
fi
|
||||
exec qsfera server
|
||||
environment:
|
||||
OC_BASE_DATA_PATH: /var/lib/qsfera
|
||||
OC_CONFIG_DIR: /etc/qsfera
|
||||
OC_URL: ${QSFERA_URL:?Set QSFERA_URL to the public server URL}
|
||||
OC_LOG_LEVEL: ${QSFERA_LOG_LEVEL:-info}
|
||||
OC_LOG_COLOR: "false"
|
||||
OC_LOG_PRETTY: "false"
|
||||
OC_INSECURE: "${QSFERA_INSECURE:-false}"
|
||||
PROXY_TLS: "${QSFERA_PROXY_TLS:-true}"
|
||||
PROXY_ENABLE_BASIC_AUTH: "${QSFERA_ENABLE_BASIC_AUTH:-false}"
|
||||
IDM_ADMIN_PASSWORD: ${QSFERA_ADMIN_PASSWORD:?Set a strong QSFERA_ADMIN_PASSWORD}
|
||||
IDM_CREATE_DEMO_USERS: "false"
|
||||
ports:
|
||||
- "${QSFERA_BIND_IP:-127.0.0.1}:9200:9200"
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ${QSFERA_CONFIG_DIR:?Set QSFERA_CONFIG_DIR to a persistent directory}
|
||||
target: /etc/qsfera
|
||||
- type: bind
|
||||
source: ${QSFERA_DATA_DIR:?Set QSFERA_DATA_DIR to a persistent directory}
|
||||
target: /var/lib/qsfera
|
||||
mem_limit: "${QSFERA_MEMORY_LIMIT:?Set QSFERA_MEMORY_LIMIT after measuring available RAM}"
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
- 'scheme=https; [ "$${PROXY_TLS}" = "false" ] && scheme=http; curl --fail --silent --show-error --insecure "$${scheme}://127.0.0.1:9200/status.php" >/dev/null'
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
logging:
|
||||
driver: local
|
||||
restart: unless-stopped
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
archive=${1:-}
|
||||
if [ -z "$archive" ] || [ ! -f "$archive" ]; then
|
||||
echo "Usage: RESTORE_CONFIRM=restore $0 /path/to/qsfera-TIMESTAMP.tar.gz" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [ "${RESTORE_CONFIRM:-}" != "restore" ]; then
|
||||
echo "Restore replaces the current QSfera configuration and data." >&2
|
||||
echo "Re-run with RESTORE_CONFIRM=restore after checking the archive path." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [ -f "$archive.sha256" ]; then
|
||||
archive_directory=$(dirname "$archive")
|
||||
archive_name=$(basename "$archive")
|
||||
(cd "$archive_directory" && sha256sum --check "$archive_name.sha256")
|
||||
fi
|
||||
|
||||
compose() {
|
||||
docker compose "$@"
|
||||
}
|
||||
|
||||
restore_completed=false
|
||||
|
||||
finish_restore() {
|
||||
if [ "$restore_completed" = "true" ]; then
|
||||
compose start qsfera >/dev/null
|
||||
else
|
||||
echo "Restore failed; QSfera remains stopped to protect the data set." >&2
|
||||
fi
|
||||
}
|
||||
|
||||
compose config --quiet
|
||||
compose stop qsfera
|
||||
trap finish_restore EXIT
|
||||
trap 'exit 1' HUP INT TERM
|
||||
|
||||
compose run --rm --no-deps -T --entrypoint /bin/sh qsfera -ec \
|
||||
'find /etc/qsfera -mindepth 1 -maxdepth 1 -exec rm -rf {} +; find /var/lib/qsfera -mindepth 1 -maxdepth 1 -exec rm -rf {} +; tar -C / -xzf -' \
|
||||
<"$archive"
|
||||
|
||||
restore_completed=true
|
||||
compose start qsfera >/dev/null
|
||||
trap - EXIT HUP INT TERM
|
||||
|
||||
echo "Restore completed. Run sh ./smoke-test.sh to verify startup and health."
|
||||
@@ -0,0 +1,53 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
case "$(uname -m)" in
|
||||
aarch64|arm64) ;;
|
||||
*)
|
||||
echo "This profile requires a 64-bit ARM OS; uname -m returned $(uname -m)." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
timeout_seconds=${SMOKE_TIMEOUT_SECONDS:-180}
|
||||
started_at=$(date +%s)
|
||||
|
||||
docker compose config --quiet
|
||||
docker compose up -d
|
||||
container_id=$(docker compose ps -q qsfera)
|
||||
|
||||
if [ -z "$container_id" ]; then
|
||||
echo "QSfera container was not created." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
while :; do
|
||||
state=$(docker inspect --format '{{.State.Status}}' "$container_id")
|
||||
if [ "$state" != "running" ]; then
|
||||
docker compose logs --tail=100 qsfera >&2
|
||||
echo "QSfera container state is $state." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
health=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}missing{{end}}' "$container_id")
|
||||
case "$health" in
|
||||
healthy)
|
||||
echo "QSfera container is healthy."
|
||||
exit 0
|
||||
;;
|
||||
unhealthy)
|
||||
docker compose logs --tail=100 qsfera >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
now=$(date +%s)
|
||||
if [ $((now - started_at)) -ge "$timeout_seconds" ]; then
|
||||
docker compose logs --tail=100 qsfera >&2
|
||||
echo "QSfera did not become healthy within ${timeout_seconds}s." >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
@@ -15,7 +15,7 @@ To configure which registry to use, you have to set the environment variable `MI
|
||||
|
||||
## Memory limits
|
||||
|
||||
КуСфера will automatically set the go native `GOMEMLIMIT` to `0.9`. To disable the limit set `AUTOMEMEMLIMIT=off`. For more information take a look at the official [Guide to the Go Garbage Collector](https://go.dev/doc/gc-guide).
|
||||
КуСфера will automatically set the go native `GOMEMLIMIT` to `0.9`. To disable the limit set `AUTOMEMLIMIT=off`. For more information take a look at the official [Guide to the Go Garbage Collector](https://go.dev/doc/gc-guide).
|
||||
|
||||
## CLI Commands
|
||||
|
||||
|
||||
Reference in New Issue
Block a user