238 lines
12 KiB
Markdown
238 lines
12 KiB
Markdown
# Manual Homelab Deployment
|
||
|
||
## Scope
|
||
The next milestone is one manual **collection -> raw files -> PostgreSQL import**
|
||
job on the homelab. Development stays on the laptop; build and run from a repository
|
||
checkout on the existing Debian VM, alongside Gitea, its runner, and PostgreSQL 17.
|
||
Use the existing database and role, currently both `car_tracker`.
|
||
Do not provision another production PostgreSQL server or recreate those objects.
|
||
|
||
`ops/app/compose.yaml` is the one-shot application stack, separate from
|
||
`ops/ci-runner`. This section describes direct manual operation, with no scheduling
|
||
or restart loop. For the optional Gitea button → restricted SSH → host-run flow,
|
||
see [SSH_RUN.md](SSH_RUN.md). Migrations remain manual in both modes.
|
||
Run commands from the homelab checkout's root with Docker Engine and Compose
|
||
available. Review the revision and `searches.toml`; do not run these in CI.
|
||
After these changes are published, update the homelab checkout before building.
|
||
|
||
## Private Runtime Configuration
|
||
Use an external, nonsecret Compose env file, for example
|
||
`/srv/car-tracker/app.env`. These absolute paths are examples, not secrets.
|
||
Inspect existing files first: never overwrite an existing env file or `.pgpass`.
|
||
Use `ops/app/.env.example` as the reference when creating a new file locally:
|
||
|
||
```dotenv
|
||
APP_UID=1000
|
||
APP_GID=1000
|
||
RAW_DATA_DIR=/srv/car-tracker/raw
|
||
DB_CONFIG_DIR=/srv/car-tracker/db
|
||
PGHOST=postgres.your-lan.example
|
||
PGPORT=5432
|
||
PGDATABASE=car_tracker
|
||
PGUSER=car_tracker
|
||
PGSSLMODE=require
|
||
APP_SUBNET=172.30.17.0/28
|
||
```
|
||
|
||
Replace the hostname with the existing VM's reachable LAN/VPN hostname and choose
|
||
the numeric `APP_UID`/`APP_GID` that will own the mounted files. Keep the application
|
||
user non-root. Both bind directories must already exist; Compose refuses to create
|
||
missing source directories. After inspecting `/srv` and any existing application
|
||
directories, create only missing directories and apply permissions deliberately:
|
||
|
||
```sh
|
||
sudo install -d -m 0700 -o 1000 -g 1000 /srv/car-tracker/raw /srv/car-tracker/db
|
||
```
|
||
|
||
Use your chosen UID/GID instead of `1000` in all ownership commands. Both
|
||
`RAW_DATA_DIR` and `DB_CONFIG_DIR` must have mode `0700` and the correct ownership.
|
||
The raw directory must be writable by the container user; the DB directory must be
|
||
traversable by that user even though its container mount is read-only.
|
||
|
||
Create or carefully edit `/srv/car-tracker/db/pgpass` locally, using a private
|
||
editor/session. Its matching entry has this form (do not use the placeholder):
|
||
|
||
```text
|
||
postgres.your-lan.example:5432:car_tracker:car_tracker:<password entered locally>
|
||
```
|
||
|
||
Escape `:` and `\` in the password according to libpq password-file rules.
|
||
Do not put passwords in commands, the env file, Git, logs, or chat. Do not send
|
||
tokens or credentials to anyone to complete this setup. Preserve existing password
|
||
files and entries rather than replacing them blindly. Protect the dedicated file:
|
||
|
||
```sh
|
||
sudo chown 1000:1000 /srv/car-tracker/db/pgpass
|
||
sudo chmod 0600 /srv/car-tracker/db/pgpass
|
||
```
|
||
|
||
The password is supplied only through `/run/db/pgpass`, mounted from
|
||
`DB_CONFIG_DIR`. Incorrect ownership prevents reads; libpq ignores a password file
|
||
with overly permissive permissions. Keep this directory dedicated to app DB files,
|
||
not unrelated credentials.
|
||
|
||
Compose interpolates the external env file but passes only the runtime whitelist:
|
||
`PGHOST`, `PGPORT`, `PGDATABASE`, `PGUSER`, and `PGSSLMODE`, plus fixed
|
||
`PGPASSFILE=/run/db/pgpass` and `PGSSLROOTCERT=/run/db/root.crt`.
|
||
It does not pass through `DATABASE_URL`, `PGPASSWORD`, or arbitrary host variables.
|
||
Shell variables can override Compose env-file interpolation; unset stale matching
|
||
variables before using this configuration. Always specify `--env-file` explicitly.
|
||
|
||
## TLS And Network Access
|
||
`PGSSLMODE` defaults to `require`: require encryption, but do not treat this as
|
||
server identity verification. Prefer `verify-full` with the trusted CA certificate
|
||
at `/srv/car-tracker/db/root.crt`, readable by the application user. The existing
|
||
directory mount exposes it as `/run/db/root.crt`; `PGHOST` must match the server
|
||
certificate. Configure PostgreSQL server TLS before running. Never fix production
|
||
connectivity by disabling TLS or adding `trust` authentication.
|
||
|
||
`localhost` inside the container is not the VM's PostgreSQL server. Prefer the VM's
|
||
LAN/VPN hostname. Compose also provides `host.docker.internal` through
|
||
`extra_hosts: host-gateway`, but PostgreSQL must listen on that gateway address if
|
||
you use it. A listener only on loopback or the LAN address does not necessarily
|
||
accept gateway connections. Certificate hostname matching still applies.
|
||
|
||
The app has a dedicated bridge subnet, default `172.30.17.0/28`. Check existing
|
||
Docker, LAN, and VPN routes for conflicts before starting; configure `APP_SUBNET`
|
||
to a non-overlapping range if needed. Coordinate any network changes with the
|
||
existing runner and services. Do not expose PostgreSQL publicly.
|
||
|
||
PostgreSQL may see a container client address different from the laptop's address.
|
||
The laptop's working HBA rule is therefore not proof this job can connect. Inspect
|
||
the actual client address in PostgreSQL logs and existing firewall/HBA rules. A new
|
||
narrow rule may be needed for the default app subnet:
|
||
|
||
```text
|
||
hostssl car_tracker car_tracker 172.30.17.0/28 scram-sha-256
|
||
```
|
||
|
||
Use the configured subnet and actual observed addressing, not a blindly copied
|
||
rule. HBA rules are first-match; preserve unrelated rules and permit only the
|
||
necessary source/port through the firewall. Check `pg_hba_file_rules` for errors
|
||
and reload authentication changes with `SELECT pg_reload_conf();` as an admin.
|
||
Changing `listen_addresses` requires a PostgreSQL restart: coordinate downtime for
|
||
the shared VM's clients rather than restarting casually. See
|
||
[PostgreSQL troubleshooting](POSTGRESQL.md#connection-troubleshooting) for details.
|
||
Use the existing `car_tracker` database and role on this VM.
|
||
|
||
## Build And Smoke Run
|
||
Validate without printing interpolated configuration, then build on the homelab:
|
||
|
||
```sh
|
||
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml config --quiet
|
||
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml build app
|
||
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml run --rm app car-tracker-db check
|
||
```
|
||
|
||
`check` is read-only connectivity testing, not a schema or migration-privilege
|
||
check. The pipeline additionally checks access to `silver.collection_runs` before
|
||
opening the browser. No new migrations are required for this milestone. If the
|
||
existing packaged migrations have not yet been applied, apply them explicitly:
|
||
|
||
```sh
|
||
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml run --rm app car-tracker-db migrate
|
||
```
|
||
|
||
Migrations are not automatic. Once DB access/schema are ready, run the image's
|
||
default bounded headed pipeline: Saab 900, one page per selected search, at most
|
||
one unique detail request for the run, with a randomized 5–7.5-second request delay
|
||
by default.
|
||
|
||
```sh
|
||
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml run --rm app
|
||
```
|
||
|
||
Inspect the reported manifest and import outcome before expanding. After a usable
|
||
smoke run, this is the explicit larger manual command:
|
||
|
||
```sh
|
||
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml run --rm app \
|
||
car-tracker-pipeline --headed --config /app/searches.toml --output /data \
|
||
--family saab-900 --max-pages 3 --max-details 5
|
||
```
|
||
|
||
Limits can intentionally produce partial coverage. Do not loop or retry a blocked
|
||
source automatically; stop and investigate access failures before another scrape.
|
||
|
||
## Outcomes And Recovery
|
||
The pipeline prints `Manifest: /data/<date>/<run-id>/manifest.json`, reports a JSON
|
||
summary after successful import, and imports
|
||
that exact finalized run, including partial or failed collection runs. Host files
|
||
are under `/srv/car-tracker/raw/<date>/<run-id>/`. Import success does not imply
|
||
complete source coverage. Keep finalized raw files and manifests immutable.
|
||
|
||
| Pipeline exit | Meaning |
|
||
| --- | --- |
|
||
| 0 | Complete collection imported, without rejected records |
|
||
| 1 | Incomplete collection imported, without rejected records |
|
||
| 2 | Import or replay has rejected records; inspect validation issues |
|
||
| 3 | Operational, configuration, locking, or database failure |
|
||
| 130 | Cancellation/interruption |
|
||
|
||
`pipeline.json` is a separate operational sidecar, not a rewrite of raw coverage.
|
||
It records `collection_status`, `import_status`, and an import summary on success.
|
||
An interrupted import or lost DB response can leave `import_status: unknown`:
|
||
the transaction may already have committed. Recover by idempotently replaying the
|
||
exact manifest with the DB CLI, not by rerunning the scrape. Substitute the exact
|
||
date/run ID printed by the pipeline:
|
||
|
||
```sh
|
||
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml run --rm app \
|
||
car-tracker-db import '/data/<date>/<run-id>/manifest.json'
|
||
```
|
||
|
||
An identical replay reports `already_imported` without duplicating observations.
|
||
The standalone DB CLI does not update `pipeline.json`; retain its replay result
|
||
separately. Its exit codes differ; see [import documentation](POSTGRESQL.md#migrate-and-import).
|
||
Do not alter raw files to force a replay past integrity checks.
|
||
|
||
`browser_cleanup.json` may record a nonfatal browser teardown error after raw
|
||
finalization; the pipeline can still import that run. Allow graceful cancellation
|
||
to finish. A hard kill, OOM, or expired stop grace can leave a `running` manifest,
|
||
no sidecar, or stale `running` import state. A `running` manifest is not imported;
|
||
investigate rather than marking it complete or assuming a DB rollback occurred.
|
||
|
||
The output-root `.pipeline.lock` covers the whole manual pipeline job, including
|
||
collection and import. Do not delete an active lock file; the file can remain after
|
||
exit and its presence alone does not mean a job is active. Standalone collection
|
||
and DB CLIs do not honor this lock. Never overlap them with a pipeline or other
|
||
manual job using the same output root, including during recovery.
|
||
|
||
## Runtime Boundaries
|
||
|
||
Compose runs non-root with a read-only root filesystem, all capabilities dropped,
|
||
and `no-new-privileges`. There is no Docker socket mount or published port.
|
||
Runtime limits are 1.5 CPUs, 2 GiB memory, 256 PIDs, 512 MiB shared memory,
|
||
512 MiB `/tmp` tmpfs, a 1 MiB X11 socket tmpfs, and a 30-second stop grace period. These do not constrain
|
||
image builds. Keep load low on the VM shared with Gitea, its runner, and PostgreSQL.
|
||
|
||
The image installs Chromium matching the locked Playwright version. Its custom
|
||
Python entrypoint starts Xvfb for headed operation, forwards SIGTERM to the child,
|
||
and waits for child cleanup before removing remaining processes and the display.
|
||
No external GUI or noVNC service is needed.
|
||
|
||
Chromium's sandbox remains disabled by Playwright's default launch behavior; an
|
||
explicit sandbox trial failed under host security restrictions. These container
|
||
settings are not a claim of secure browser isolation. Do not add privileged mode,
|
||
`SYS_ADMIN`, or `seccomp=unconfined` to work around this. The browser container can
|
||
read mounted DB credentials and shares the VM kernel. Prefer a separate VM for
|
||
future browser isolation rather than weakening the shared host's protections.
|
||
|
||
## Verification And Retention
|
||
|
||
Live source access from this homelab has not been verified here. All 109 tests
|
||
passed inside the built non-root image with its read-only filesystem and resource
|
||
limits, using isolated PostgreSQL 17 and synthetic browser responses. An actual
|
||
Docker stop also allowed the child to finish cleanup and exit 130 before the
|
||
display was removed. These tests do not guarantee live source access.
|
||
For optional testing, see [PostgreSQL tests](POSTGRESQL.md#tests) and
|
||
`tests/test_browser_pipeline.py`. Browser tests require `RUN_BROWSER_TESTS=1` and
|
||
`TEST_DATABASE_URL`; use only a disposable database explicitly ending in `_test`,
|
||
never homelab production. Tests truncate data. Default CI skips browser tests.
|
||
|
||
Retention is still manual: start with 30 days for raw files and private database
|
||
copies, including descriptions and raw payloads, and account for relevant backups.
|
||
Deleting raw files does not delete DB copies. No retention or backup automation is
|
||
provided here; arrange and verify private backups manually before relying on the
|
||
data. Keep raw files, DB copies, and backups private and outside Git/CI artifacts.
|