feat: improve library and audiobook workflow

server/audiobook/README.md

@@ -1,49 +1,61 @@
-# Aletheia CMP audiobook receiver
+# Aletheia audiobook receiver for VoiceStudio on CMP
 
-The receiver accepts EPUB/FB2 files, queues one synthesis job at a time, reports progress, and returns an AAC/M4A file. The HTTP process starts with no Qwen model in memory. `Qwen3TTSModel` and `torch` are imported only when a job reaches the synthesis stage; the model is deleted and the CUDA cache is cleared before M4A encoding.
+The receiver accepts EPUB and FB2, stores persistent jobs and device credentials,
+extracts text, and sends it to the existing VoiceStudio `/audiobook` API.
+VoiceStudio runs on three GPUs and uses the saved Russian narrator `db7928a3`.
+The receiver does not load TTS models or manage other inference services.
 
-## Deployed layout
+## Deployment
 
-- CMP host: `192.168.0.112`, service root `/home/sevenhill/apps/aletheia-audiobook`
-- API: `http://0.0.0.0:8765`
-- jobs and SQLite state: `/home/sevenhill/apps/aletheia-audiobook/audiobook-jobs`
-- API token: `/home/sevenhill/apps/aletheia-audiobook/service/service-token.txt`
-- public route: `https://argus.kusoft.xyz/aletheia-tts/`
-- voice: Qwen3-TTS 0.6B CustomVoice, `Ryan`, Russian
+- Receiver: `/home/sevenhill/apps/aletheia-audiobook/service`
+- Python: `/home/sevenhill/apps/VoiceStudio/.venv/bin/python`
+- State: `/home/sevenhill/apps/aletheia-audiobook/audiobook-jobs/jobs.db3`
+- VoiceStudio: `http://127.0.0.1:3900`, outputs in `~/voicestudio-data/outputs`
+- Public API: `https://argus.kusoft.xyz/aletheia-tts/`
+- User unit: `aletheia-audiobook-cmp.service`, port 8765
 
-The token is intentionally not stored in this repository. The Android build reads it from `%USERPROFILE%\.aletheia\audiobook-api-token.txt` and embeds it in `BuildConfig` for this private installation.
+Copy service Python files and the unit, then run `systemctl --user daemon-reload`
+and `systemctl --user enable --now aletheia-audiobook-cmp`. User lingering enables
+startup without an SSH session. Back up the SQLite database before upgrades.
+The Pi Caddy route strips `/aletheia-tts/` and proxies to CMP port 8765.
 
-## Runtime
+`aletheia-audio-cleanup.service` runs `audio_cleanup.py` independently, so it can
+be deployed while a book is rendering. Android acknowledges delivery with DELETE
+only after saving the M4A; a SQLite trigger retains that acknowledgement across
+restarts. The cleaner removes the corresponding VoiceStudio M4B, work directory
+and chapter WAVs, plus old disposable segment cache when all renderers are idle.
+Models, voice profiles and other finished books are retained. New jobs record the
+exact VoiceStudio output before becoming ready; an already-running legacy worker
+is matched by SHA-256 while its output is being delivered. Unknown matches are
+retained rather than guessing which book to delete.
 
-The CMP receiver uses `/home/sevenhill/apps/qwen3-tts-venv`. It starts as a lightweight FastAPI process and loads the 0.6B Qwen model only after a queued audiobook reaches the synthesis stage. The model is deleted and CUDA cache cleared before M4A encoding.
+The health endpoint checks VoiceStudio availability. Device enrollment is required:
+run `manage_devices.py pair` with the service Python, then enter the single-use code
+in Aletheia Settings → «Подключить аудиосервис». Codes normally expire in ten minutes.
+Credentials are independent per device; job status, file and deletion check ownership.
+Use `manage_devices.py list` and `manage_devices.py revoke DEVICE_ID` for revocation.
 
-The `systemd --user` service starts only the receiver process. It does not preload Qwen or reserve GPU memory; the model is loaded by an audiobook request. After a job finishes, systemd replaces the receiver process so Qwen and Triton CUDA contexts are fully released before the receiver waits for the next request.
+## Processing and progress
 
-When an audiobook enters synthesis, the receiver temporarily stops and runtime-masks Ollama, then loads one tested 0.6B Qwen worker on each of GPU 0, 1, and 2. Up to three independent chunks are generated in parallel. After the job, the receiver exits so systemd can release all CUDA contexts; its fresh idle process restores Ollama. Existing WAV chunks are never regenerated during resume.
+Android checks availability before resetting a book and uploading it. The receiver
+consumes VoiceStudio chapter/assembly events and exposes progress through job status.
+Long EPUB/FB2 sections are split at sentence/word boundaries into parts of at most
+1600 characters. Each free GPU immediately takes the next part from a shared queue;
+it never waits for the other two GPUs to finish a batch. Progress events are sent
+after each part and counted by completed text length. Repeated events from GPU
+rendering and assembly are counted once. The deployed scheduler and regression
+test are mirrored in `cmp/` and installed in VoiceStudio's `backend/services/`
+and `cmp/` directories respectively.
+VoiceStudio's M4B MP4/AAC container is copied as M4A, preserving chapter metadata.
+The app downloads it to persistent storage and reports 100% only after saving it.
 
-Per-batch timings are appended to `audiobook-jobs\performance.jsonl`. The log separates the autoregressive talker, speech-tokenizer decoder, and wrapper time so runtime optimizations can be benchmarked without changing the generated audio path.
+Jobs interrupted by a receiver restart are requeued; VoiceStudio reuses its chapter
+cache. Cancellation is checked between stream events; a chapter already rendering
+can finish before the cancellation is observed. VoiceStudio keeps its own output/cache.
 
-The CMP firewall must permit TCP 8765 only from the Raspberry Pi address. Caddy uses a path handler that removes the public prefix before proxying:
+## Verification
 
-```caddyfile
-argus.kusoft.xyz {
-    encode zstd gzip
-    handle_path /aletheia-tts/* {
-        reverse_proxy 192.168.0.112:8765
-    }
-    handle {
-        reverse_proxy 127.0.0.1:5105
-    }
-}
-```
-
-Validate with `caddy validate` before reloading Caddy. `/health` is public and reports `modelLoaded`; all `/v1/audiobooks` endpoints require the bearer token.
-
-## API
-
-- `POST /v1/audiobooks` — multipart fields `book`, `title`, `author`
-- `GET /v1/audiobooks/{id}` — status, stage, processed/total characters, chapter, duration and chapter markers
-- `GET /v1/audiobooks/{id}/file` — completed M4A
-- `DELETE /v1/audiobooks/{id}` — cancel an active job or remove a completed job
-
-Interrupted receiver processes requeue unfinished jobs on the next start. Existing WAV chunks are reused, so synthesis resumes at the first missing chunk.
+Run `python -m unittest discover -s server/audiobook -p 'test_*.py' -v`.
+`smoke_live.py`, run on CMP, performs real EPUB upload, VoiceStudio synthesis and
+public M4A download with a disposable device, then revokes it. Set
+`AUDIOBOOK_SMOKE_FORMAT=fb2` for the same test with FB2. Unit tests mock synthesis.

server/audiobook/aletheia-audio-cleanup.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Aletheia delivered audiobook cleanup
+After=aletheia-audiobook-cmp.service
+
+[Service]
+Type=simple
+WorkingDirectory=/home/sevenhill/apps/aletheia-audiobook/service
+Environment=AUDIOBOOK_SERVICE_ROOT=/home/sevenhill/apps/aletheia-audiobook
+ExecStart=/home/sevenhill/apps/VoiceStudio/.venv/bin/python audio_cleanup.py
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

server/audiobook/aletheia-audiobook-cmp.service

@@ -7,12 +7,8 @@ Wants=network-online.target
 Type=simple
 WorkingDirectory=/home/sevenhill/apps/aletheia-audiobook/service
 Environment=AUDIOBOOK_SERVICE_ROOT=/home/sevenhill/apps/aletheia-audiobook
-Environment=HF_HOME=/home/sevenhill/apps/qwen3-audiobook-test/hf
-Environment=QWEN_TTS_MODEL_ID=Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice
-Environment=QWEN_TTS_SPEAKER=Ryan
-Environment=QWEN_TTS_GPU_IDS=0,1,2
-Environment=CPATH=/home/sevenhill/apps/qwen3-audiobook-test/sysdeps/extracted/usr/include/python3.12:/home/sevenhill/apps/qwen3-audiobook-test/sysdeps/extracted/usr/include
-ExecStart=/home/sevenhill/apps/qwen3-tts-venv/bin/python -m uvicorn audiobook_service:app --host 0.0.0.0 --port 8765 --workers 1
+Environment=VOICESTUDIO_VOICE=f24f004e
+ExecStart=/home/sevenhill/apps/VoiceStudio/.venv/bin/python -m uvicorn audiobook_service:app --host 0.0.0.0 --port 8765 --workers 1
 Restart=always
 RestartSec=5
 

server/audiobook/audio_cleanup.py

@@ -0,0 +1,161 @@
+"""Durable post-delivery cleanup. A ready-job DELETE is the phone's acknowledgement.
+
+Runs separately so it can be deployed without interrupting an active renderer.
+Never deletes voice profiles, models, or another book's finished output.
+"""
+import hashlib
+import json
+import logging
+import os
+from pathlib import Path
+import re
+import shutil
+import sqlite3
+import time
+
+ROOT = Path(os.environ.get('AUDIOBOOK_SERVICE_ROOT', '/home/sevenhill/apps/aletheia-audiobook'))
+OUTPUTS = Path(os.environ.get('VOICESTUDIO_OUTPUTS', '/home/sevenhill/voicestudio-data/outputs'))
+DATABASE = ROOT / 'audiobook-jobs/jobs.db3'
+VOICE_DATABASE = OUTPUTS.parent / 'omnivoice.db'
+LOG = logging.getLogger(__name__)
+
+
+def connect():
+    connection = sqlite3.connect(DATABASE, timeout=30)
+    connection.row_factory = sqlite3.Row
+    return connection
+
+
+def initialize():
+    with connect() as db:
+        db.executescript('''
+            CREATE TABLE IF NOT EXISTS delivered_audio_cleanup (
+                id TEXT PRIMARY KEY, output_path TEXT NOT NULL,
+                acknowledged INTEGER NOT NULL DEFAULT 0, cutoff REAL,
+                artifact TEXT, completed INTEGER NOT NULL DEFAULT 0
+            );
+            CREATE TRIGGER IF NOT EXISTS acknowledge_delivered_audio
+            BEFORE DELETE ON jobs WHEN OLD.status = 'ready'
+            BEGIN
+                INSERT INTO delivered_audio_cleanup(id, output_path, acknowledged, cutoff)
+                VALUES(OLD.id, OLD.output_path, 1, unixepoch())
+                ON CONFLICT(id) DO UPDATE SET acknowledged=1, cutoff=unixepoch();
+            END;
+        ''')
+
+
+def remember_output(destination, source):
+    """Persist outside the job directory, before the result becomes downloadable."""
+    source = Path(source).resolve()
+    if source.parent != OUTPUTS.resolve() or not re.fullmatch(r'audiobook_[a-zA-Z0-9_-]+\.m4b', source.name):
+        raise ValueError('Unexpected VoiceStudio output')
+    with connect() as db:
+        db.execute('''INSERT INTO delivered_audio_cleanup(id, output_path, artifact)
+            VALUES(?, ?, ?) ON CONFLICT(id) DO UPDATE SET artifact=excluded.artifact''',
+            (Path(destination).parent.name, str(destination), source.name))
+
+
+def digest(path):
+    with path.open('rb') as stream:
+        return hashlib.file_digest(stream, 'sha256').digest()
+
+
+def capture_legacy_outputs():
+    """Also covers a worker that was already running when cleanup was installed."""
+    with connect() as db:
+        rows = db.execute('''SELECT j.id, j.output_path FROM jobs j
+            LEFT JOIN delivered_audio_cleanup c ON c.id=j.id
+            WHERE j.status IN ('encoding','ready') AND c.artifact IS NULL''').fetchall()
+    for row in rows:
+        local = Path(row['output_path'])
+        if not local.is_file() or not local.stat().st_size:
+            continue
+        candidates = [p for p in OUTPUTS.glob('audiobook_*.m4b')
+                      if p.stat().st_size == local.stat().st_size]
+        if not candidates:
+            continue
+        expected = digest(local)
+        matches = [p for p in candidates if digest(p) == expected]
+        if len(matches) == 1:
+            remember_output(local, matches[0])
+
+
+def renderers_idle():
+    with connect() as db:
+        if db.execute("SELECT 1 FROM jobs WHERE status IN ('queued','extracting','synthesizing','encoding') LIMIT 1").fetchone():
+            return False
+    # VoiceStudio's own jobs may belong to a different client.
+    if not VOICE_DATABASE.is_file():
+        return False
+    with sqlite3.connect(f'file:{VOICE_DATABASE}?mode=ro', uri=True) as db:
+        return not db.execute("SELECT 1 FROM jobs WHERE status IN ('queued','running','pending') LIMIT 1").fetchone()
+
+
+def remove_artifact(name):
+    if not re.fullmatch(r'audiobook_[a-zA-Z0-9_-]+\.m4b', name):
+        raise ValueError('Unsafe artifact name')
+    root = OUTPUTS.resolve()
+    output = (root / name).resolve()
+    work = (root / Path(name).stem).resolve()
+    if output.parent != root or work.parent != root:
+        raise ValueError('Artifact escapes output directory')
+    cache = (root / 'longform_cache').resolve()
+    concat = work / 'concat.txt'
+    removed = 0
+    if concat.is_file():
+        for line in concat.read_text().splitlines():
+            match = re.fullmatch(r"file '([^']+)'", line)
+            if not match:
+                raise ValueError('Unexpected concat entry')
+            wav = Path(match[1]).resolve()
+            if wav.parent != cache or not re.fullmatch(r'[a-f0-9]{20}\.wav', wav.name):
+                raise ValueError('Unsafe cache path')
+            if wav.is_file():
+                removed += wav.stat().st_size
+                wav.unlink()
+    if output.is_file():
+        removed += output.stat().st_size
+        output.unlink()
+    if work.exists():
+        shutil.rmtree(work)
+    return removed
+
+
+def sweep():
+    capture_legacy_outputs()
+    with connect() as db:
+        # Serialize against receiver job creation/state changes. A new render
+        # cannot begin between the idle check and the last unlink.
+        db.execute('BEGIN IMMEDIATE')
+        pending = db.execute('''SELECT * FROM delivered_audio_cleanup
+            WHERE acknowledged=1 AND completed=0 AND artifact IS NOT NULL''').fetchall()
+        if not pending or not VOICE_DATABASE.is_file():
+            return
+        with sqlite3.connect(VOICE_DATABASE, timeout=30) as voice_db:
+            voice_db.execute('BEGIN IMMEDIATE')
+            if not renderers_idle():
+                return
+            for row in pending:
+                removed = remove_artifact(row['artifact'])
+                # Segment WAVs are a disposable shared cache. Only prune old
+                # entries while idle; newly created files are never candidates.
+                segment_root = OUTPUTS / 'longform_cache/segments'
+                for path in segment_root.glob('*.wav'):
+                    if path.is_symlink() or not re.fullmatch(r'[a-f0-9]{20}\.wav', path.name):
+                        continue
+                    if path.stat().st_mtime < row['cutoff']:
+                        removed += path.stat().st_size
+                        path.unlink()
+                db.execute('UPDATE delivered_audio_cleanup SET completed=1 WHERE id=?', (row['id'],))
+                LOG.info('Delivered audiobook %s cleaned: %d bytes freed', row['id'], removed)
+
+
+if __name__ == '__main__':
+    logging.basicConfig(level=logging.INFO)
+    initialize()
+    while True:
+        try:
+            sweep()
+        except Exception:
+            LOG.exception('Cleanup deferred; durable acknowledgement retained')
+        time.sleep(5)

server/audiobook/audiobook_service.py

@@ -1,13 +1,11 @@
 from __future__ import annotations
 
-import gc
-import hashlib
-import hmac
+import sys
+from device_auth import DeviceStore
 import html
 import json
 import os
 import re
-import secrets
 import shutil
 import sqlite3
 import subprocess
@@ -15,68 +13,24 @@ import threading
 import time
 import uuid
 import zipfile
-from concurrent.futures import ThreadPoolExecutor
 from contextlib import contextmanager
 from pathlib import Path, PurePosixPath
 from typing import TYPE_CHECKING, Annotated, Any, Iterator
 from xml.etree import ElementTree
 
-import imageio_ffmpeg
-import soundfile as sf
 from fastapi import Depends, FastAPI, File, Form, Header, HTTPException, UploadFile
 from fastapi.responses import FileResponse
 from num2words import num2words
 
-if TYPE_CHECKING:
-    from qwen_tts import Qwen3TTSModel
-
-
 ROOT = Path(os.environ.get("AUDIOBOOK_SERVICE_ROOT", "/home/sevenhill/apps/aletheia-audiobook"))
 SERVICE_ROOT = ROOT / "service"
 DATA_ROOT = ROOT / "audiobook-jobs"
-TOKEN_FILE = SERVICE_ROOT / "service-token.txt"
 DATABASE = DATA_ROOT / "jobs.db3"
-PERFORMANCE_LOG = DATA_ROOT / "performance.jsonl"
-MODEL_ID = os.environ.get("QWEN_TTS_MODEL_ID", "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice")
 MAX_UPLOAD_BYTES = 200 * 1024 * 1024
-GPU_IDS = tuple(
-    int(value.strip())
-    for value in os.environ.get("QWEN_TTS_GPU_IDS", "0,1,2").split(",")
-    if value.strip()
-)
-SPEAKER = os.environ.get("QWEN_TTS_SPEAKER", "Ryan")
-VOICE_INSTRUCTION = (
-    "Read in a calm, clear, natural audiobook style with steady pacing and distinct diction."
-)
-
 SERVICE_ROOT.mkdir(parents=True, exist_ok=True)
 DATA_ROOT.mkdir(parents=True, exist_ok=True)
-OLLAMA_PAUSE_MARKER = SERVICE_ROOT / "ollama-paused-for-audiobook"
-if not TOKEN_FILE.exists():
-    TOKEN_FILE.write_text(secrets.token_urlsafe(48), encoding="ascii")
-API_TOKEN = TOKEN_FILE.read_text(encoding="ascii").strip()
 
-
-def pause_ollama_for_audiobook() -> None:
-    """Release every CMP GPU before loading the three Qwen workers."""
-    subprocess.run(["systemctl", "--user", "stop", "ollama.service"], check=True)
-    subprocess.run(["systemctl", "--user", "mask", "--runtime", "ollama.service"], check=True)
-    OLLAMA_PAUSE_MARKER.write_text("1", encoding="ascii")
-
-
-def restore_ollama_after_audiobook() -> None:
-    if not OLLAMA_PAUSE_MARKER.exists():
-        return
-    subprocess.run(["systemctl", "--user", "unmask", "--runtime", "ollama.service"], check=True)
-    subprocess.run(["systemctl", "--user", "start", "ollama.service"], check=True)
-    OLLAMA_PAUSE_MARKER.unlink(missing_ok=True)
-
-
-# The worker terminates after every job to release CUDA/Triton contexts. The
-# next lightweight receiver process restores Ollama before accepting new work.
-restore_ollama_after_audiobook()
-
-app = FastAPI(title="Aletheia CMP Audiobook Service", version="1.1.0")
+app = FastAPI(title="Aletheia CMP Audiobook Service", version="2.0.0")
 queue_condition = threading.Condition()
 queued_jobs: list[str] = []
 worker_state = {"activeJobId": None, "modelLoaded": False}
@@ -118,6 +72,9 @@ def initialize_database() -> None:
             )
             """
         )
+        if "owner_id" not in {row[1] for row in connection.execute("PRAGMA table_info(jobs)")}:
+            connection.execute("ALTER TABLE jobs ADD COLUMN owner_id TEXT")
+        connection.execute("CREATE INDEX IF NOT EXISTS idx_jobs_owner ON jobs(owner_id)")
         connection.execute(
             "UPDATE jobs SET status='queued', stage='queued' "
             "WHERE status IN ('extracting','synthesizing','encoding')"
@@ -128,10 +85,31 @@ def initialize_database() -> None:
     queued_jobs.extend(row["id"] for row in rows)
 
 
-def require_token(authorization: Annotated[str | None, Header()] = None) -> None:
-    expected = f"Bearer {API_TOKEN}"
-    if not authorization or not hmac.compare_digest(authorization, expected):
-        raise HTTPException(status_code=401, detail="Invalid API token")
+devices = DeviceStore(DATABASE)
+
+
+def require_token(authorization: Annotated[str | None, Header()] = None) -> str:
+    try:
+        if not authorization or not authorization.startswith("Bearer "):
+            raise PermissionError()
+        return devices.authenticate(authorization[7:])
+    except PermissionError:
+        raise HTTPException(status_code=401, detail="Invalid device credential")
+
+
+@app.post("/v1/devices/enroll")
+def enroll_device(payload: dict) -> dict:
+    try:
+        return devices.enroll(payload.get("code", ""), payload.get("label", "Android"))
+    except PermissionError:
+        raise HTTPException(status_code=403, detail="Invalid or expired pairing code")
+
+
+def owned_job(job_id: str, owner: str) -> sqlite3.Row:
+    row = get_job(job_id)
+    if row["owner_id"] != owner:
+        raise HTTPException(status_code=404, detail="Audiobook job not found")
+    return row
 
 
 def now_ms() -> int:
@@ -170,17 +148,22 @@ def job_payload(row: sqlite3.Row) -> dict:
 
 @app.get("/health")
 def health() -> dict:
+    from voicestudio_client import check_available
+    try:
+        check_available()
+    except Exception:
+        raise HTTPException(status_code=503, detail="VoiceStudio unavailable")
     return {
         "status": "ok",
-        "workerMode": "lazy-model",
-        "activeJobId": worker_state["activeJobId"],
+        "workerMode": "voicestudio",
         "modelLoaded": worker_state["modelLoaded"],
     }
 
 
-@app.post("/v1/audiobooks", dependencies=[Depends(require_token)])
+@app.post("/v1/audiobooks")
 async def create_audiobook(
     book: Annotated[UploadFile, File()],
+    owner: Annotated[str, Depends(require_token)],
     title: Annotated[str, Form()],
     author: Annotated[str, Form()] = "",
 ) -> dict:
@@ -207,8 +190,8 @@ async def create_audiobook(
         connection.execute(
             """
             INSERT INTO jobs (
-                id,title,author,source_name,input_path,output_path,status,stage,created_at,updated_at
-            ) VALUES (?,?,?,?,?,?,?,?,?,?)
+                id,title,author,source_name,input_path,output_path,status,stage,created_at,updated_at,owner_id
+            ) VALUES (?,?,?,?,?,?,?,?,?,?,?)
             """,
             (
                 job_id,
@@ -221,6 +204,7 @@ async def create_audiobook(
                 "queued",
                 timestamp,
                 timestamp,
+                owner,
             ),
         )
     with queue_condition:
@@ -229,14 +213,14 @@ async def create_audiobook(
     return job_payload(get_job(job_id))
 
 
-@app.get("/v1/audiobooks/{job_id}", dependencies=[Depends(require_token)])
-def audiobook_status(job_id: str) -> dict:
-    return job_payload(get_job(job_id))
+@app.get("/v1/audiobooks/{job_id}")
+def audiobook_status(job_id: str, owner: Annotated[str, Depends(require_token)]) -> dict:
+    return job_payload(owned_job(job_id, owner))
 
 
-@app.get("/v1/audiobooks/{job_id}/file", dependencies=[Depends(require_token)])
-def audiobook_file(job_id: str) -> FileResponse:
-    row = get_job(job_id)
+@app.get("/v1/audiobooks/{job_id}/file")
+def audiobook_file(job_id: str, owner: Annotated[str, Depends(require_token)]) -> FileResponse:
+    row = owned_job(job_id, owner)
     output = Path(row["output_path"])
     if row["status"] != "ready" or not output.is_file():
         raise HTTPException(status_code=409, detail="Audiobook is not ready")
@@ -244,9 +228,9 @@ def audiobook_file(job_id: str) -> FileResponse:
     return FileResponse(output, media_type="audio/mp4", filename=f"{safe_title}.m4a")
 
 
-@app.delete("/v1/audiobooks/{job_id}", dependencies=[Depends(require_token)])
-def cancel_or_delete(job_id: str) -> dict:
-    row = get_job(job_id)
+@app.delete("/v1/audiobooks/{job_id}")
+def cancel_or_delete(job_id: str, owner: Annotated[str, Depends(require_token)]) -> dict:
+    row = owned_job(job_id, owner)
     if row["status"] == "queued":
         with queue_condition:
             queued_jobs[:] = [queued_id for queued_id in queued_jobs if queued_id != job_id]
@@ -394,273 +378,61 @@ def cancellation_requested(job_id: str) -> bool:
     return row is None or bool(row["cancel_requested"])
 
 
-def load_model(device_id: int) -> "Qwen3TTSModel":
-    import torch
-    from qwen_tts import Qwen3TTSModel
-
-    if not torch.cuda.is_available():
-        raise RuntimeError("CUDA is unavailable")
-    torch.set_float32_matmul_precision("high")
-    model = Qwen3TTSModel.from_pretrained(
-        MODEL_ID,
-        device_map=f"cuda:{device_id}",
-        dtype=torch.bfloat16,
-        attn_implementation="sdpa",
-    )
-    return model
-
-
-def load_models() -> list["Qwen3TTSModel"]:
-    if not GPU_IDS:
-        raise RuntimeError("No Qwen GPU IDs are configured")
-    models = [load_model(device_id) for device_id in GPU_IDS]
-    worker_state["modelLoaded"] = True
-    return models
-
-
-def generate_custom_voice_with_metrics(
-    model: Any,
-    batch_chunks: list[dict],
-    device_id: int,
-) -> tuple[list[Any], int]:
-    """Measure Qwen talker and codec stages without changing generated audio."""
-    import torch
-
-    timings: dict[str, float] = {}
-    original_generate = model.model.generate
-    speech_tokenizer = model.model.speech_tokenizer
-    original_decode = speech_tokenizer.decode
-
-    def synchronize() -> None:
-        if torch.cuda.is_available():
-            torch.cuda.synchronize(device_id)
-
-    def timed_generate(*args: Any, **kwargs: Any) -> Any:
-        synchronize()
-        started = time.perf_counter()
-        try:
-            return original_generate(*args, **kwargs)
-        finally:
-            synchronize()
-            timings["talkerGenerateSeconds"] = time.perf_counter() - started
-
-    def timed_decode(*args: Any, **kwargs: Any) -> Any:
-        synchronize()
-        started = time.perf_counter()
-        try:
-            return original_decode(*args, **kwargs)
-        finally:
-            synchronize()
-            timings["codecDecodeSeconds"] = time.perf_counter() - started
-
-    model.model.generate = timed_generate
-    speech_tokenizer.decode = timed_decode
-    started = time.perf_counter()
-    try:
-        wavs, sample_rate = model.generate_custom_voice(
-            text=[chunk["text"] for chunk in batch_chunks],
-            language=["Russian"] * len(batch_chunks),
-            speaker=[SPEAKER] * len(batch_chunks),
-            instruct=[VOICE_INSTRUCTION] * len(batch_chunks),
-        )
-        synchronize()
-    finally:
-        model.model.generate = original_generate
-        speech_tokenizer.decode = original_decode
-
-    total_seconds = time.perf_counter() - started
-    talker_seconds = timings.get("talkerGenerateSeconds", 0.0)
-    codec_seconds = timings.get("codecDecodeSeconds", 0.0)
-    record = {
-        "timestampMs": now_ms(),
-        "batchSize": len(batch_chunks),
-        "characters": sum(len(chunk["text"]) for chunk in batch_chunks),
-        "totalSeconds": round(total_seconds, 4),
-        "talkerGenerateSeconds": round(talker_seconds, 4),
-        "codecDecodeSeconds": round(codec_seconds, 4),
-        "wrapperSeconds": round(max(0.0, total_seconds - talker_seconds - codec_seconds), 4),
-        "cudaPeakAllocatedMiB": round(
-            torch.cuda.max_memory_allocated(device_id) / (1024 * 1024), 1
-        ),
-    }
-    with PERFORMANCE_LOG.open("a", encoding="utf-8") as output:
-        output.write(json.dumps(record, ensure_ascii=False) + "\n")
-    return wavs, sample_rate
-
-
-def unload_models(models: list[Any]) -> None:
-    if models:
-        models.clear()
-        import torch
-
-        gc.collect()
-        if torch.cuda.is_available():
-            for device_id in GPU_IDS:
-                torch.cuda.empty_cache()
-    else:
-        gc.collect()
-    worker_state["modelLoaded"] = False
-
-
-def encode_m4a(chunk_files: list[Path], output: Path, job_root: Path) -> None:
-    concat = job_root / "concat.txt"
-    concat.write_text(
-        "\n".join(f"file '{path.as_posix()}'" for path in chunk_files),
-        encoding="utf-8",
-    )
-    command = [
-        imageio_ffmpeg.get_ffmpeg_exe(),
-        "-y",
-        "-f",
-        "concat",
-        "-safe",
-        "0",
-        "-i",
-        str(concat),
-        "-vn",
-        "-c:a",
-        "aac",
-        "-b:a",
-        "96k",
-        "-movflags",
-        "+faststart",
-        str(output),
-    ]
-    completed = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", errors="replace")
-    if completed.returncode != 0:
-        raise RuntimeError(completed.stderr[-2000:] or "FFmpeg failed")
-
-
 def run_job(job_id: str) -> None:
+    from voicestudio_client import render_audiobook, split_sections
     row = get_job(job_id)
     job_root = Path(row["input_path"]).parent
-    chunks_root = job_root / "chunks"
-    chunks_root.mkdir(exist_ok=True)
-    models: list[Any] = []
     try:
         if cancellation_requested(job_id):
             finish_cancellation(job_id, job_root)
             return
         update_job(job_id, status="extracting", stage="extracting", error_message=None)
-        input_path = Path(row["input_path"])
-        sections = extract_epub(input_path) if input_path.suffix.lower() == ".epub" else extract_fb2(input_path)
-        chunks = split_chunks(sections)
-        if not chunks:
-            raise RuntimeError("В книге не найден русский текст для озвучивания")
-        total = sum(len(chunk["text"]) for chunk in chunks)
-        manifest_path = job_root / "chunks.json"
-        manifest_path.write_text(json.dumps(chunks, ensure_ascii=False, indent=2), encoding="utf-8")
-        update_job(job_id, status="synthesizing", stage="loading_model", total_characters=total)
+        source = Path(row["input_path"])
+        sections = extract_epub(source) if source.suffix.lower() == ".epub" else extract_fb2(source)
+        sections = split_sections(sections)
+        if not sections:
+            raise RuntimeError("В книге не найден текст для озвучивания")
+        total = sum(len(text) for _, text in sections)
+        update_job(job_id, status="synthesizing", stage="synthesizing", total_characters=total,
+                   processed_characters=0, duration_ms=0, chapters_json="[]",
+                   current_chapter="Проверка сохранённых фрагментов и озвучивание недостающих…")
+        completed = {}
+        verified = set()
+
+        def on_event(event):
+            if event.get("type") == "chapter":
+                index = int(event["index"])
+                completed[index] = event
+                duration = 0
+                markers = []
+                for _, chapter in sorted(completed.items()):
+                    markers.append({"title": chapter.get("title", "Раздел"), "startMs": duration})
+                    duration += round(float(chapter.get("duration_s", 0)) * 1000)
+                processed = sum(len(sections[i][1]) for i in completed if 0 <= i < len(sections))
+                label = event.get("title")
+                if event.get("stage") != "gpu_render":
+                    verified.add(index)
+                    label = f"Проверка готовых фрагментов: {len(verified)} из {len(sections)}"
+                update_job(job_id, processed_characters=processed,
+                           current_chapter=label, duration_ms=duration,
+                           chapters_json=json.dumps(markers, ensure_ascii=False))
+            elif event.get("type") == "started":
+                update_job(job_id, current_chapter="Проверка готовых фрагментов перед сборкой M4A…")
+            elif event.get("type") == "assembling":
+                update_job(job_id, status="encoding", stage="encoding", current_chapter=None)
+
+        result = render_audiobook(sections, row["title"], row["author"], Path(row["output_path"]),
+                                 on_event, lambda: cancellation_requested(job_id))
         if cancellation_requested(job_id):
             finish_cancellation(job_id, job_root)
             return
-        pause_ollama_for_audiobook()
-        models = load_models()
-        import torch
-
-        processed = 0
-        duration_ms = 0
-        chapter_markers: list[dict] = []
-        last_chapter: str | None = None
-        chunk_files: list[Path] = []
-        index = 0
-        while index < len(chunks):
-            if cancellation_requested(job_id):
-                finish_cancellation(job_id, job_root)
-                return
-            output = chunks_root / f"{index:06d}.wav"
-            completed_indices: list[int]
-            if output.is_file():
-                completed_indices = [index]
-            else:
-                batch_indices: list[int] = []
-                cursor = index
-                while cursor < len(chunks) and len(batch_indices) < len(models):
-                    candidate = chunks_root / f"{cursor:06d}.wav"
-                    if candidate.is_file():
-                        break
-                    batch_indices.append(cursor)
-                    cursor += 1
-                with ThreadPoolExecutor(max_workers=len(batch_indices)) as executor:
-                    futures = [
-                        executor.submit(
-                            generate_custom_voice_with_metrics,
-                            model,
-                            [chunks[batch_index]],
-                            device_id,
-                        )
-                        for batch_index, model, device_id in zip(
-                            batch_indices,
-                            models[: len(batch_indices)],
-                            GPU_IDS[: len(batch_indices)],
-                            strict=True,
-                        )
-                    ]
-                    rendered = [future.result() for future in futures]
-                if cancellation_requested(job_id):
-                    finish_cancellation(job_id, job_root)
-                    return
-                for batch_index, (wavs, sample_rate) in zip(batch_indices, rendered, strict=True):
-                    sf.write(
-                        chunks_root / f"{batch_index:06d}.wav",
-                        wavs[0],
-                        sample_rate,
-                        subtype="PCM_16",
-                    )
-                completed_indices = batch_indices
-
-            for completed_index in completed_indices:
-                completed_chunk = chunks[completed_index]
-                completed_output = chunks_root / f"{completed_index:06d}.wav"
-                if completed_chunk["chapter"] != last_chapter:
-                    chapter_markers.append({"title": completed_chunk["chapter"], "startMs": duration_ms})
-                    last_chapter = completed_chunk["chapter"]
-                info = sf.info(completed_output)
-                duration_ms += round(info.frames * 1000 / info.samplerate)
-                chunk_files.append(completed_output)
-                processed += len(completed_chunk["text"])
-                update_job(
-                    job_id,
-                    status="synthesizing",
-                    stage="synthesizing",
-                    processed_characters=processed,
-                    current_chapter=completed_chunk["chapter"],
-                    duration_ms=duration_ms,
-                    chapters_json=json.dumps(chapter_markers, ensure_ascii=False),
-                )
-            index = completed_indices[-1] + 1
-
-        unload_models(models)
-        models = []
-        if cancellation_requested(job_id):
-            finish_cancellation(job_id, job_root)
-            return
-        update_job(job_id, status="encoding", stage="encoding", current_chapter=None)
-        output = Path(row["output_path"])
-        encode_m4a(chunk_files, output, job_root)
-        if cancellation_requested(job_id):
-            finish_cancellation(job_id, job_root)
-            return
-        if not output.is_file() or output.stat().st_size == 0:
-            raise RuntimeError("Итоговый M4A-файл не создан")
-        update_job(
-            job_id,
-            status="ready",
-            stage="ready",
-            processed_characters=total,
-            current_chapter=None,
-            chapters_json=json.dumps(chapter_markers, ensure_ascii=False),
-        )
+        update_job(job_id, status="ready", stage="ready", processed_characters=total,
+                   duration_ms=round(float(result["duration_s"]) * 1000), current_chapter=None)
     except Exception as error:
-        update_job(
-            job_id,
-            status="failed",
-            stage="failed",
-            error_message=f"{type(error).__name__}: {error}",
-        )
-    finally:
-        unload_models(models)
-
+        if cancellation_requested(job_id):
+            finish_cancellation(job_id, job_root)
+        else:
+            update_job(job_id, status="failed", stage="failed", error_message=f"{type(error).__name__}: {error}")
 
 def worker_loop() -> None:
     while True:
@@ -670,19 +442,25 @@ def worker_loop() -> None:
             job_id = queued_jobs.pop(0)
         worker_state["activeJobId"] = job_id
         try:
-            try:
-                run_job(job_id)
-            except HTTPException as error:
-                if error.status_code != 404:
-                    raise
+            env = dict(os.environ, AUDIOBOOK_WORKER_PROCESS="1")
+            process = subprocess.Popen([sys.executable, str(Path(__file__).with_name("worker_runner.py")), job_id], env=env)
+            while process.poll() is None:
+                try:
+                    worker_state["modelLoaded"] = get_job(job_id)["status"] == "synthesizing"
+                except HTTPException:
+                    pass
+                time.sleep(0.5)
+            if process.returncode:
+                try:
+                    update_job(job_id, status="failed", stage="failed", error_message="Audiobook worker stopped unexpectedly")
+                except HTTPException:
+                    pass
         finally:
             worker_state["activeJobId"] = None
-            # Qwen/Triton can retain a CUDA context after Python references are
-            # released. Let systemd start a clean idle receiver so CMP VRAM is
-            # fully available between audiobook requests. Unfinished jobs are
-            # restored from SQLite by initialize_database() after restart.
-            os._exit(0)
+            worker_state["modelLoaded"] = False
 
 
-initialize_database()
-threading.Thread(target=worker_loop, name="audiobook-worker", daemon=True).start()
+if os.environ.get("AUDIOBOOK_WORKER_PROCESS") != "1":
+    initialize_database()
+    if os.environ.get("AUDIOBOOK_DISABLE_WORKER") != "1":
+        threading.Thread(target=worker_loop, name="audiobook-worker", daemon=True).start()

server/audiobook/cmp/cmp_multigpu.py

@@ -0,0 +1,62 @@
+"""CMP opt-in: one continuous chapter worker per GPU, with immediate progress."""
+import asyncio
+import json
+import logging
+import httpx
+
+logger = logging.getLogger(__name__)
+_lock = asyncio.Lock()
+
+
+async def warm_chapters(script, count, endpoints, params, disconnected=None):
+    if count <= 0:
+        return
+    if not endpoints:
+        raise ValueError('No CMP GPU endpoints configured')
+    async with _lock:
+        if disconnected and await disconnected():
+            raise asyncio.CancelledError()
+        async with httpx.AsyncClient(timeout=httpx.Timeout(7200, connect=15), trust_env=False) as client:
+            indices = iter(range(count))
+            results = asyncio.Queue()
+
+            async def worker(endpoint):
+                try:
+                    for index in indices:
+                        payload = dict(params, text=script, chapter_index=index)
+                        response = await client.post(endpoint + '/audiobook/preview', json=payload)
+                        response.raise_for_status()
+                        result = response.json()
+                        logger.info('CMP_GPU_CHAPTER gpu=%s chapter=%s cached=%s duration=%s',
+                                    endpoint, index, result.get('cached'), result.get('duration_s'))
+                        event = dict(type='chapter', index=index, total=count,
+                                     title=result.get('title', ''), duration_s=result['duration_s'],
+                                     cached=result.get('cached', False), stage='gpu_render')
+                        await results.put(('chapter', event))
+                except Exception as error:
+                    await results.put(('error', error))
+                finally:
+                    await results.put(('finished', None))
+
+            tasks = [asyncio.create_task(worker(endpoint)) for endpoint in endpoints[:count]]
+            finished = 0
+            try:
+                while finished < len(tasks):
+                    if disconnected and await disconnected():
+                        raise asyncio.CancelledError()
+                    try:
+                        kind, value = await asyncio.wait_for(results.get(), timeout=5)
+                    except asyncio.TimeoutError:
+                        yield ': cmp GPU rendering\n\n'
+                        continue
+                    if kind == 'error':
+                        raise value
+                    if kind == 'finished':
+                        finished += 1
+                    else:
+                        yield 'data: ' + json.dumps(value) + '\n\n'
+            finally:
+                for task in tasks:
+                    if not task.done():
+                        task.cancel()
+                await asyncio.gather(*tasks, return_exceptions=True)

server/audiobook/cmp/fix_longform_encode.py

@@ -0,0 +1,35 @@
+"""Apply the duration-aware export fix to the installed VoiceStudio router."""
+from pathlib import Path
+import sys
+
+path = Path(sys.argv[1])
+source = path.read_text(encoding='utf-8')
+old = '''        await run_ffmpeg(
+            build_render_cmd('''
+new = '''        # Long audiobooks can take hours to encode on the CMP CPU.
+        # Keep a bounded timeout proportional to the prepared audio duration.
+        encode_timeout = min(21600.0, max(1800.0,
+            sum(d for _, d in chapters_meta) / 1000.0 * 0.5))
+        returncode, _, encode_error = await run_ffmpeg(
+            build_render_cmd('''
+tail = '''            job_id=job_id,
+        )
+
+        if job_store is not None:'''
+replacement = '''            job_id=job_id, timeout=encode_timeout,
+        )
+        if returncode != 0:
+            logger.error("[%s] ffmpeg export failed: %s", job_id,
+                         (encode_error or b"").decode(errors="replace")[-2000:])
+            raise RuntimeError("Audio export failed")
+
+        if job_store is not None:'''
+if 'encode_timeout = min(21600.0' not in source:
+    assert source.count(old) == 1
+    assert source.count(tail) == 1
+    backup = path.with_suffix('.py.before-duration-timeout')
+    if not backup.exists():
+        backup.write_text(source, encoding='utf-8')
+    source = source.replace(old, new).replace(tail, replacement)
+    compile(source, str(path), 'exec')
+    path.write_text(source, encoding='utf-8')

server/audiobook/cmp/preserve_prepared_cache.py

@@ -0,0 +1,24 @@
+"""Idempotent VoiceStudio patch: CMP's warm pass already owns the active cache."""
+from pathlib import Path
+import sys
+
+
+def patch_router(path):
+    path = Path(path)
+    source = path.read_text()
+    old = '    prune_cache_dir(cache_dir)  # bound disk before this job adds its chapters'
+    new = '''    # CMP has already rendered the chapters on all GPUs. Pruning here
+    # deletes this job's inputs and silently repeats synthesis on GPU 0.
+    # Aletheia removes delivered audio through its acknowledgement cleaner.
+    if not os.environ.get("CMP_AUDIOBOOK_GPUS"):
+        prune_cache_dir(cache_dir)  # bound disk before this job adds its chapters'''
+    if new in source:
+        return
+    if source.count(old) != 1:
+        raise RuntimeError('Unexpected VoiceStudio router; patch requires review')
+    path.with_suffix('.py.before-cmp-cache-fix').write_text(source)
+    path.write_text(source.replace(old, new))
+
+
+if __name__ == '__main__':
+    patch_router(sys.argv[1])

server/audiobook/cmp/test_dynamic_queue.py

@@ -0,0 +1,82 @@
+import asyncio
+import json
+import sys
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'backend'))
+from services.cmp_multigpu import warm_chapters
+
+
+class DynamicQueueTests(unittest.IsolatedAsyncioTestCase):
+    async def test_free_gpu_continues_and_reports_before_slow_gpu_finishes(self):
+        slow = asyncio.Event()
+        dispatched = []
+        active = set()
+
+        class Response:
+            def raise_for_status(self): pass
+            def json(self): return {'duration_s': 1, 'cached': False}
+
+        class Client:
+            def __init__(self, **kwargs): pass
+            async def __aenter__(self): return self
+            async def __aexit__(self, *args): pass
+            async def post(self, url, json):
+                self_outer.assertNotIn(url, active)
+                active.add(url)
+                index = json['chapter_index']
+                dispatched.append((url, index))
+                try:
+                    if index == 0:
+                        await slow.wait()
+                    else:
+                        await asyncio.sleep(.001)
+                    return Response()
+                finally:
+                    active.remove(url)
+
+        self_outer = self
+        with patch('services.cmp_multigpu.httpx.AsyncClient', Client):
+            stream = warm_chapters('text', 6, ['gpu0', 'gpu1', 'gpu2'], {})
+            try:
+                first = await asyncio.wait_for(anext(stream), .3)
+                self.assertTrue(first.startswith('data:'))
+                self.assertFalse(slow.is_set())
+                while len(dispatched) < 6:
+                    await asyncio.wait_for(anext(stream), .3)
+                self.assertTrue(any(index >= 3 and url != 'gpu0/audiobook/preview' for url, index in dispatched))
+                slow.set()
+                remaining = [event async for event in stream]
+                self.assertTrue(any('"index": 0' in event for event in remaining))
+                self.assertEqual(list(range(6)), sorted(index for _, index in dispatched))
+            finally:
+                slow.set()
+                await stream.aclose()
+        self.assertFalse(active)
+
+    async def test_closing_stream_cancels_inflight_requests(self):
+        stopped = asyncio.Event()
+        class Response:
+            def raise_for_status(self): pass
+            def json(self): return {'duration_s': 1}
+        class Client:
+            def __init__(self, **kwargs): pass
+            async def __aenter__(self): return self
+            async def __aexit__(self, *args): pass
+            async def post(self, url, json):
+                if json['chapter_index'] == 0:
+                    try: await asyncio.Event().wait()
+                    finally: stopped.set()
+                return Response()
+        with patch('services.cmp_multigpu.httpx.AsyncClient', Client):
+            stream = warm_chapters('text', 2, ['gpu0', 'gpu1'], {})
+            try:
+                await asyncio.wait_for(anext(stream), .3)
+            finally:
+                await stream.aclose()
+            self.assertTrue(stopped.is_set())
+
+
+if __name__ == '__main__': unittest.main()

server/audiobook/cmp/test_prepared_cache.py

@@ -0,0 +1,49 @@
+"""Execute VoiceStudio's real assembly generator against a cache over its cap."""
+import os
+from pathlib import Path
+import tempfile
+import types
+import unittest
+from unittest.mock import AsyncMock, patch
+
+from api.routers import audiobook as router
+from services.audiobook import parse_audiobook_script
+from services.longform_render import prune_cache_dir
+
+
+class PreparedCacheTests(unittest.IsolatedAsyncioTestCase):
+    async def test_cmp_assembly_keeps_warmed_audio_even_over_cache_limit(self):
+        await self.check_assembly(0)
+
+    async def test_failed_encoder_does_not_report_done(self):
+        await self.check_assembly(1)
+
+    async def check_assembly(self, returncode):
+        with tempfile.TemporaryDirectory() as root:
+            wav = Path(root) / 'longform_cache/fixture.wav'
+            wav.parent.mkdir()
+            wav.write_bytes(b'prepared chapter larger than configured cap')
+            async def cached_chapter(*args, **kwargs):
+                self.assertTrue(wav.is_file(), 'assembly evicted an already rendered chapter')
+                return str(wav), 79213.0, True, None
+            job_store = types.SimpleNamespace(**{n: lambda *a, **k: None for n in
+                ['create', 'mark_running', 'append_event', 'mark_done', 'mark_failed']})
+            with patch.dict(os.environ, {'CMP_AUDIOBOOK_GPUS': 'gpu0,gpu1,gpu2'}), \
+                 patch('core.config.OUTPUTS_DIR', root), \
+                 patch('core.job_store', job_store, create=True), \
+                 patch('services.ffmpeg_utils.find_ffmpeg', return_value='ffmpeg'), \
+                 patch('services.ffmpeg_utils.run_ffmpeg', new=AsyncMock(return_value=(returncode, b'', b'encode failed'))) as encode, \
+                 patch('services.gpu_gateway.decide'), \
+                 patch.object(router, '_resolve_default_language', return_value='ru'), \
+                 patch.object(router, '_run_chapter', side_effect=cached_chapter), \
+                 patch.object(router, 'prune_cache_dir', side_effect=lambda p: prune_cache_dir(p, max_bytes=1)):
+                events = [e async for e in router._render_longform_sse(
+                    parse_audiobook_script('# Chapter\nText'), default_voice=None)]
+            self.assertTrue(wav.exists())
+            self.assertEqual(encode.call_args.kwargs['timeout'], 21600.0)
+            self.assertEqual(any('"type": "done"' in e for e in events), returncode == 0, events)
+            self.assertEqual(any('"type": "error"' in e for e in events), returncode != 0, events)
+
+
+if __name__ == '__main__':
+    unittest.main()

server/audiobook/device_auth.py

@@ -0,0 +1,63 @@
+"""Device enrollment and revocation. No credential is embedded in the APK."""
+import hashlib
+import secrets
+import sqlite3
+import time
+import uuid
+from contextlib import contextmanager
+
+
+class DeviceStore:
+    def __init__(self, path):
+        self.path = str(path)
+        with self.connect() as db:
+            db.execute("CREATE TABLE IF NOT EXISTS devices (id TEXT PRIMARY KEY, token_hash TEXT UNIQUE NOT NULL, label TEXT NOT NULL, revoked INTEGER NOT NULL DEFAULT 0, created_at INTEGER NOT NULL)")
+            db.execute("CREATE TABLE IF NOT EXISTS pairing_codes (code_hash TEXT PRIMARY KEY, expires_at INTEGER NOT NULL, used INTEGER NOT NULL DEFAULT 0)")
+
+    @contextmanager
+    def connect(self):
+        connection = sqlite3.connect(self.path, timeout=30)
+        try:
+            with connection:
+                yield connection
+        finally:
+            connection.close()
+
+    @staticmethod
+    def digest(value):
+        return hashlib.sha256(value.encode('utf-8')).hexdigest()
+
+    def create_code(self, lifetime=600):
+        code = secrets.token_urlsafe(24)
+        with self.connect() as db:
+            db.execute("DELETE FROM pairing_codes WHERE expires_at < ? OR used=1", (int(time.time()),))
+            db.execute("INSERT INTO pairing_codes VALUES (?, ?, 0)", (self.digest(code), int(time.time()) + lifetime))
+        return code
+
+    def enroll(self, code, label):
+        if not isinstance(code, str) or not 20 <= len(code) <= 128:
+            raise PermissionError('Invalid pairing code')
+        token = secrets.token_urlsafe(48)
+        device_id = uuid.uuid4().hex
+        with self.connect() as db:
+            db.execute('BEGIN IMMEDIATE')
+            updated = db.execute('UPDATE pairing_codes SET used=1 WHERE code_hash=? AND used=0 AND expires_at>=?',
+                                 (self.digest(code), int(time.time())))
+            if updated.rowcount != 1:
+                raise PermissionError('Pairing code expired or already used')
+            db.execute('INSERT INTO devices VALUES (?, ?, ?, 0, ?)',
+                       (device_id, self.digest(token), str(label)[:100], int(time.time())))
+        return {'deviceId': device_id, 'token': token}
+
+    def authenticate(self, token):
+        if not token or len(token) > 256:
+            raise PermissionError('Missing device credential')
+        with self.connect() as db:
+            row = db.execute('SELECT id FROM devices WHERE token_hash=? AND revoked=0', (self.digest(token),)).fetchone()
+        if row is None:
+            raise PermissionError('Invalid device credential')
+        return row[0]
+
+    def revoke(self, device_id):
+        with self.connect() as db:
+            db.execute('UPDATE devices SET revoked=1 WHERE id=?', (device_id,))

server/audiobook/manage_devices.py

@@ -0,0 +1,25 @@
+"""Run via SSH: manage_devices.py pair | list | revoke DEVICE_ID.
+
+Pairing codes expire in ten minutes and must be entered in Android Settings.
+"""
+import argparse
+import os
+from pathlib import Path
+from device_auth import DeviceStore
+
+parser = argparse.ArgumentParser()
+parser.add_argument('action', choices=['pair', 'list', 'revoke'])
+parser.add_argument('device_id', nargs='?')
+args = parser.parse_args()
+root = Path(os.environ.get('AUDIOBOOK_SERVICE_ROOT', '/home/sevenhill/apps/aletheia-audiobook'))
+store = DeviceStore(root / 'audiobook-jobs/jobs.db3')
+if args.action == 'pair':
+    print(store.create_code())
+elif args.action == 'revoke':
+    if not args.device_id:
+        parser.error('device_id is required')
+    store.revoke(args.device_id)
+else:
+    with store.connect() as db:
+        for row in db.execute('SELECT id,label,revoked,created_at FROM devices ORDER BY created_at'):
+            print('\t'.join(map(str, row)))

server/audiobook/requirements.txt

@@ -1,4 +1,4 @@
-imageio-ffmpeg==0.6.0
+fastapi>=0.115,<1
+uvicorn>=0.30,<1
 num2words==0.5.14
-python-multipart==0.0.32
-
+python-multipart>=0.0.18,<1

server/audiobook/smoke_live.py

@@ -0,0 +1,80 @@
+"""Real GPU/API smoke test, run on CMP. Uses and revokes a disposable device."""
+import hashlib
+import io
+import json
+import os
+from pathlib import Path
+import time
+import zipfile
+import httpx
+from device_auth import DeviceStore
+
+root = Path(os.environ.get('AUDIOBOOK_SERVICE_ROOT', '/home/sevenhill/apps/aletheia-audiobook'))
+base = 'https://argus.kusoft.xyz/aletheia-tts'
+with httpx.Client(timeout=60) as client:
+    for attempt in range(30):
+        try:
+            health = client.get(base + '/health')
+            if health.status_code == 200:
+                break
+        except httpx.TransportError:
+            pass
+        time.sleep(2)
+    else:
+        raise RuntimeError('Public receiver health endpoint is unavailable')
+    store = DeviceStore(root / 'audiobook-jobs/jobs.db3')
+    response = client.post(base + '/v1/devices/enroll', json={'code': store.create_code(), 'label': 'Deployment smoke test'})
+    response.raise_for_status()
+    device = response.json()
+    headers = {'Authorization': 'Bearer ' + device['token']}
+    job = None
+    try:
+        text = 'Это проверка изготовления аудиокниги. Приёмник получил книгу от приложения и читает её вслух. Готовая запись возвращается в библиотеку.'
+        book_format = os.environ.get('AUDIOBOOK_SMOKE_FORMAT', 'epub')
+        if book_format == 'epub':
+            archive = io.BytesIO()
+            with zipfile.ZipFile(archive, 'w') as epub:
+                epub.writestr('mimetype', 'application/epub+zip')
+                epub.writestr('META-INF/container.xml', '<container xmlns="urn:oasis:names:tc:opendocument:xmlns:container" version="1.0"><rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles></container>')
+                chapters = int(os.environ.get('AUDIOBOOK_SMOKE_CHAPTERS', '1'))
+                manifest = ''.join(f'<item id="c{i}" href="chapter{i}.xhtml" media-type="application/xhtml+xml"/>' for i in range(chapters))
+                spine = ''.join(f'<itemref idref="c{i}"/>' for i in range(chapters))
+                epub.writestr('OEBPS/content.opf', '<package xmlns="http://www.idpf.org/2007/opf" version="2.0" unique-identifier="bookid"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:identifier id="bookid">cmp-smoke</dc:identifier><dc:title>Проверка CMP</dc:title><dc:language>ru</dc:language></metadata><manifest>' + manifest + '</manifest><spine>' + spine + '</spine></package>')
+                for i in range(chapters):
+                    body = text if chapters == 1 else text + f' Раздел номер {i + 1}.'
+                    epub.writestr(f'OEBPS/chapter{i}.xhtml', '<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Проверка</title></head><body><h1>Проверка ' + str(i + 1) + '</h1><p>' + body + '</p></body></html>')
+            book = archive.getvalue()
+        else:
+            book = ('<?xml version="1.0" encoding="UTF-8"?><FictionBook><body><section><title><p>Проверка</p></title><p>' + text + '</p></section></body></FictionBook>').encode()
+        response = client.post(base + '/v1/audiobooks', headers=headers, data={'title': 'Проверка CMP', 'author': 'Aletheia'}, files={'book': ('test.' + book_format, book, 'application/epub+zip' if book_format == 'epub' else 'application/xml')})
+        response.raise_for_status()
+        job = response.json()['id']
+        path = base + '/v1/audiobooks/' + job
+        assert client.get(path).status_code == 401
+        deadline = time.monotonic() + 1200
+        previous = None
+        while time.monotonic() < deadline:
+            response = client.get(path, headers=headers)
+            response.raise_for_status()
+            status = response.json()
+            state = (status['status'], status['stage'], status['processedCharacters'], status['totalCharacters'])
+            if state != previous:
+                print(json.dumps({'state': state}, ensure_ascii=False), flush=True)
+                previous = state
+            if status['status'] == 'ready':
+                response = client.get(path + '/file', headers=headers)
+                response.raise_for_status()
+                assert b'ftyp' in response.content[:32] and status['durationMs'] > 0
+                output = root / 'smoke-audiobook.m4a'
+                output.write_bytes(response.content)
+                print(json.dumps({'result': 'PASS', 'bytes': len(response.content), 'durationMs': status['durationMs'], 'sha256': hashlib.sha256(response.content).hexdigest()}), flush=True)
+                break
+            if status['status'] in ('failed', 'cancelled'):
+                raise RuntimeError(status)
+            time.sleep(3)
+        else:
+            raise TimeoutError('Synthesis exceeded smoke test timeout')
+    finally:
+        if job:
+            client.delete(base + '/v1/audiobooks/' + job, headers=headers)
+        store.revoke(device['deviceId'])

server/audiobook/test_api.py

@@ -0,0 +1,63 @@
+import os
+import tempfile
+import unittest
+from unittest.mock import patch
+from pathlib import Path
+
+# A separate temporary service root and disabled worker prevent any GPU/service action.
+_root = tempfile.TemporaryDirectory()
+os.environ['AUDIOBOOK_SERVICE_ROOT'] = _root.name
+os.environ['AUDIOBOOK_DISABLE_WORKER'] = '1'
+import audiobook_service as service
+from fastapi.testclient import TestClient
+
+
+class ApiOwnershipTests(unittest.TestCase):
+    def setUp(self):
+        self.health = patch('voicestudio_client.check_available')
+        self.health.start()
+        self.addCleanup(self.health.stop)
+        self.client = TestClient(service.app)
+        self.one = self.enroll('first')
+        self.two = self.enroll('second')
+
+    def tearDown(self):
+        self.client.close()
+
+    def enroll(self, name):
+        response = self.client.post('/v1/devices/enroll', json={'code': service.devices.create_code(), 'label': name})
+        self.assertEqual(200, response.status_code)
+        return {'Authorization': 'Bearer ' + response.json()['token']}
+
+    def test_owner_is_required_for_status_download_and_delete(self):
+        response = self.client.post('/v1/audiobooks', headers=self.one,
+            data={'title': 'Fixture', 'author': 'Test'}, files={'book': ('fixture.fb2', b'<FictionBook/>', 'application/xml')})
+        self.assertEqual(200, response.status_code, response.text)
+        job_id = response.json()['id']
+        path = '/v1/audiobooks/' + job_id
+        self.assertEqual(401, self.client.get(path).status_code)
+        self.assertEqual(200, self.client.get(path, headers=self.one).status_code)
+        self.assertEqual(404, self.client.get(path, headers=self.two).status_code)
+        self.assertEqual(404, self.client.get(path + '/file', headers=self.two).status_code)
+        self.assertEqual(404, self.client.delete(path, headers=self.two).status_code)
+        self.assertEqual(200, self.client.delete(path, headers=self.one).status_code)
+        self.assertEqual(404, self.client.get(path, headers=self.one).status_code)
+
+    def test_health_does_not_disclose_job_id_and_legacy_token_is_rejected(self):
+        self.assertNotIn('activeJobId', self.client.get('/health').json())
+        response = self.client.post('/v1/audiobooks', headers={'Authorization': 'Bearer legacy-shared-key'},
+            data={'title': 'Test'}, files={'book': ('a.fb2', b'<FictionBook/>')})
+        self.assertEqual(401, response.status_code)
+
+    def test_enrollment_code_cannot_be_replayed(self):
+        code = service.devices.create_code()
+        self.assertEqual(200, self.client.post('/v1/devices/enroll', json={'code': code}).status_code)
+        self.assertEqual(403, self.client.post('/v1/devices/enroll', json={'code': code}).status_code)
+
+
+def tearDownModule():
+    _root.cleanup()
+
+
+if __name__ == '__main__':
+    unittest.main()

server/audiobook/test_audio_cleanup.py

@@ -0,0 +1,98 @@
+import os
+from pathlib import Path
+import sqlite3
+import tempfile
+import unittest
+from unittest.mock import patch
+import audio_cleanup as cleanup
+
+
+class CleanupTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name)
+        self.outputs = self.root / 'outputs'
+        self.outputs.mkdir()
+        self.db = self.root / 'jobs.db3'
+        self.voice_db = self.root / 'voice.db3'
+        for name, value in [('DATABASE', self.db), ('OUTPUTS', self.outputs), ('VOICE_DATABASE', self.voice_db)]:
+            p = patch.object(cleanup, name, value)
+            p.start()
+            self.addCleanup(p.stop)
+        with sqlite3.connect(self.db) as db:
+            db.execute('CREATE TABLE jobs(id TEXT, output_path TEXT, status TEXT)')
+        with sqlite3.connect(self.voice_db) as db:
+            db.execute('CREATE TABLE jobs(status TEXT)')
+        cleanup.initialize()
+        self.local = self.root / 'job/audiobook.m4a'
+        self.local.parent.mkdir()
+        self.local.write_bytes(b'complete mp4')
+        self.output = self.outputs / 'audiobook_test.m4b'
+        self.output.write_bytes(self.local.read_bytes())
+        self.work = self.outputs / 'audiobook_test'
+        self.work.mkdir()
+        self.wav = self.outputs / ('longform_cache/' + 'a' * 20 + '.wav')
+        self.wav.parent.mkdir()
+        self.wav.write_bytes(b'chapter')
+        self.segment = self.wav.parent / ('segments/' + 'b' * 20 + '.wav')
+        self.segment.parent.mkdir()
+        self.segment.write_bytes(b'segment')
+        os.utime(self.segment, (1, 1))
+        (self.work / 'concat.txt').write_text("file '" + str(self.wav) + "'\n")
+        with cleanup.connect() as db:
+            db.execute('INSERT INTO jobs VALUES(?,?,?)', ('job', str(self.local), 'ready'))
+
+    def acknowledge(self):
+        with cleanup.connect() as db:
+            db.execute("DELETE FROM jobs WHERE id='job'")
+
+    def test_keeps_everything_until_phone_acknowledges_then_cleans(self):
+        other = self.outputs / 'audiobook_other.m4b'
+        other.write_bytes(b'other book')
+        cleanup.remember_output(self.local, self.output)
+        cleanup.sweep()
+        self.assertTrue(self.output.exists())
+        self.assertTrue(self.wav.exists())
+        self.acknowledge()
+        cleanup.sweep()
+        for path in [self.output, self.work, self.wav, self.segment]:
+            self.assertFalse(path.exists(), path)
+        self.assertTrue(other.exists())
+        self.assertTrue(self.local.exists())
+        cleanup.sweep()  # restart/retry is idempotent
+
+    def test_active_generation_defers_cleanup(self):
+        cleanup.remember_output(self.local, self.output)
+        self.acknowledge()
+        with cleanup.connect() as db:
+            db.execute("INSERT INTO jobs VALUES('next', '', 'synthesizing')")
+        cleanup.sweep()
+        self.assertTrue(self.output.exists())
+        with cleanup.connect() as db:
+            db.execute("DELETE FROM jobs WHERE id='next'")
+        cleanup.sweep()
+        self.assertFalse(self.output.exists())
+
+    def test_legacy_worker_is_matched_by_content_not_title(self):
+        cleanup.capture_legacy_outputs()
+        self.acknowledge()
+        self.local.unlink()
+        cleanup.sweep()
+        self.assertFalse(self.output.exists())
+
+    def test_rejects_path_escape_and_retains_acknowledgement_on_error(self):
+        cleanup.remember_output(self.local, self.output)
+        self.acknowledge()
+        outside = self.root / 'keep.wav'
+        outside.write_bytes(b'keep')
+        (self.work / 'concat.txt').write_text("file '" + str(outside) + "'\n")
+        with self.assertRaises(ValueError):
+            cleanup.sweep()
+        self.assertTrue(outside.exists())
+        with cleanup.connect() as db:
+            self.assertEqual(0, db.execute('SELECT completed FROM delivered_audio_cleanup').fetchone()[0])
+
+
+if __name__ == '__main__':
+    unittest.main()

server/audiobook/test_device_auth.py

@@ -0,0 +1,43 @@
+import tempfile
+import unittest
+from pathlib import Path
+from device_auth import DeviceStore
+
+
+class DeviceAuthTests(unittest.TestCase):
+    def setUp(self):
+        self.root = tempfile.TemporaryDirectory()
+        self.store = DeviceStore(Path(self.root.name) / 'auth.db')
+
+    def tearDown(self):
+        self.root.cleanup()
+
+    def test_pairing_is_single_use_and_devices_have_independent_credentials(self):
+        code = self.store.create_code()
+        first = self.store.enroll(code, 'one')
+        with self.assertRaises(PermissionError):
+            self.store.enroll(code, 'replay')
+        second = self.store.enroll(self.store.create_code(), 'two')
+        self.assertNotEqual(first['token'], second['token'])
+        self.assertEqual(first['deviceId'], self.store.authenticate(first['token']))
+        self.store.revoke(first['deviceId'])
+        with self.assertRaises(PermissionError):
+            self.store.authenticate(first['token'])
+        self.assertEqual(second['deviceId'], self.store.authenticate(second['token']))
+
+    def test_expired_pairing_and_unknown_tokens_are_rejected(self):
+        with self.assertRaises(PermissionError):
+            self.store.enroll(self.store.create_code(lifetime=-1), 'expired')
+        with self.assertRaises(PermissionError):
+            self.store.authenticate('old-shared-apk-token')
+
+    def test_database_contains_hashes_not_bearer_tokens(self):
+        enrolled = self.store.enroll(self.store.create_code(), 'phone')
+        with self.store.connect() as db:
+            value = db.execute('SELECT token_hash FROM devices').fetchone()[0]
+        self.assertNotEqual(enrolled['token'], value)
+        self.assertEqual(64, len(value))
+
+
+if __name__ == '__main__':
+    unittest.main()

server/audiobook/test_voicestudio_client.py

@@ -0,0 +1,39 @@
+import io
+import json
+from pathlib import Path
+import tempfile
+import unittest
+from unittest.mock import patch
+import voicestudio_client as adapter
+
+
+class VoiceStudioTests(unittest.TestCase):
+    def test_bounded_parts_preserve_text_order(self):
+        text = ('Это длинный раздел книги. ' * 2000).strip()
+        parts = adapter.split_sections([('Глава', text)])
+        self.assertGreater(len(parts), 20)
+        self.assertTrue(all(0 < len(part) <= 1600 for _, part in parts))
+        self.assertEqual(text, ' '.join(part for _, part in parts))
+        self.assertEqual([], adapter.split_sections([('Пустая', '  ')]))
+
+    def test_stream_progress_and_completed_container(self):
+        with tempfile.TemporaryDirectory() as root:
+            outputs = Path(root)
+            audio = b'\x00\x00\x00\x18ftypM4A ' + b'\0' * 128
+            (outputs / 'result.m4b').write_bytes(audio)
+            events = [{'type': 'chapter', 'index': 0, 'total': 1},
+                      {'type': 'assembling'},
+                      {'type': 'done', 'output': 'result.m4b', 'duration_s': 1, 'failed_chapters': []}]
+            stream = io.BytesIO(b''.join(('data: ' + json.dumps(e) + '\n\n').encode() for e in events))
+            seen = []
+            with patch('audio_cleanup.remember_output') as remember, patch.object(adapter, 'check_available'), patch.object(adapter, 'OUTPUTS', outputs), patch.object(adapter, 'urlopen', return_value=stream):
+                adapter.render_audiobook([('Глава', 'Текст')], 'Книга', 'Автор', outputs / 'book.m4a', seen.append, lambda: False)
+            remember.assert_called_once_with(outputs / 'book.m4a', outputs / 'result.m4b')
+            self.assertEqual(events, seen)
+            self.assertEqual(audio, (outputs / 'book.m4a').read_bytes())
+
+    def test_failed_chapters_never_become_ready(self):
+        stream = io.BytesIO(b'data: {"type":"done","failed_chapters":[1]}\n')
+        with patch.object(adapter, 'check_available'), patch.object(adapter, 'urlopen', return_value=stream):
+            with self.assertRaises(RuntimeError):
+                adapter.render_audiobook([('Chapter', 'Text')], 'Book', '', Path('unused.m4a'), lambda event: None, lambda: False)

server/audiobook/voicestudio_client.py

@@ -0,0 +1,77 @@
+"""Adapter to the existing local VoiceStudio audiobook API; no local TTS model."""
+import json
+import os
+import re
+from pathlib import Path
+import shutil
+from urllib.request import Request, urlopen
+
+BASE = os.environ.get('VOICESTUDIO_URL', 'http://127.0.0.1:3900').rstrip('/')
+VOICE = os.environ.get('VOICESTUDIO_VOICE', 'db7928a3')
+OUTPUTS = Path(os.environ.get('VOICESTUDIO_OUTPUTS', '/home/sevenhill/voicestudio-data/outputs'))
+
+
+def split_sections(sections, limit=1600):
+    """Bound GPU tasks while preserving all non-whitespace text and its order."""
+    result = []
+    for heading, text in sections:
+        text = re.sub(r'\s+', ' ', text).strip()
+        parts = []
+        while len(text) > limit:
+            cut = max((match.end() for match in re.finditer(r'[.!?…]\s+', text[:limit])
+                       if match.end() >= limit // 2), default=0)
+            if not cut:
+                cut = text.rfind(' ', 0, limit + 1)
+            if cut <= 0:
+                cut = limit
+            parts.append(text[:cut].strip())
+            text = text[cut:].lstrip()
+        if text:
+            parts.append(text)
+        for index, part in enumerate(parts, 1):
+            label = heading if len(parts) == 1 else f'{heading} · {index}/{len(parts)}'
+            result.append((label, part))
+    return result
+
+
+def check_available():
+    with urlopen(BASE + '/health', timeout=3) as response:
+        if response.status != 200:
+            raise RuntimeError('VoiceStudio unavailable')
+
+
+def render_audiobook(sections, title, author, destination, on_event, cancelled):
+    check_available()
+    payload = {
+        'text': '\n\n'.join('# ' + heading.replace('\n', ' ') + '\n' + text for heading, text in sections),
+        'default_voice': VOICE, 'language': 'ru', 'format': 'm4b',
+        'metadata': {'title': title, 'author': author},
+    }
+    request = Request(BASE + '/audiobook', data=json.dumps(payload).encode(),
+                      headers={'Content-Type': 'application/json'})
+    # Assembly may take up to six hours on CMP, with no SSE events meanwhile.
+    with urlopen(request, timeout=28800) as response:
+        for line in response:
+            if cancelled():
+                raise RuntimeError('Изготовление отменено')
+            if not line.startswith(b'data:'):
+                continue
+            event = json.loads(line[5:].strip())
+            if event.get('type') == 'error':
+                raise RuntimeError(event.get('message') or event.get('error') or str(event))
+            on_event(event)
+            if event.get('type') == 'done':
+                if event.get('failed_chapters'):
+                    raise RuntimeError('VoiceStudio не озвучил все главы')
+                source = (OUTPUTS / event['output']).resolve()
+                if not source.is_relative_to(OUTPUTS.resolve()) or not source.is_file():
+                    raise RuntimeError('VoiceStudio вернул недоступный файл')
+                # M4B and M4A are the same MP4/AAC container; preserve embedded chapters.
+                with source.open('rb') as audio:
+                    if b'ftyp' not in audio.read(32):
+                        raise RuntimeError('VoiceStudio вернул повреждённый аудиофайл')
+                shutil.copyfile(source, destination)
+                from audio_cleanup import remember_output
+                remember_output(destination, source)
+                return event
+    raise RuntimeError('VoiceStudio завершил поток без готовой аудиокниги')

server/audiobook/worker_runner.py

@@ -0,0 +1,12 @@
+"""Disposable CUDA process; the API continues serving uploads and downloads."""
+import os
+import sys
+os.environ['AUDIOBOOK_WORKER_PROCESS'] = '1'
+from audiobook_service import run_job, HTTPException
+
+if __name__ == '__main__':
+    try:
+        run_job(sys.argv[1])
+    except HTTPException as error:
+        if error.status_code != 404:
+            raise