car-tracker/docs/GITOPS.md

Gitea-triggered application jobs

Retired: the active Run app workflow now uses restricted SSH. Follow SSH_RUN.md, keep the reconcile timer disabled, and do not use the installation/activation steps below. Retained for historical recovery context.

Flow

The app is a one-shot collector/importer, not a continuously running service.

1. Push to master (the actual default branch). CI runs lint, offline tests, disposable PostgreSQL 17 integration tests and package checks.
2. After tests pass, CI builds and publishes gitea.vandeplanque.com/tonio/car-tracker:<full-commit-SHA>. Reruns skip an existing SHA tag. Tags are still mutable at the registry; requests pin a digest.
3. Choose Actions → Run app → Run workflow, select workflow branch master, and enter a published full SHA or a Git tag pointing to it. Branch names and latest are not release selectors. The release must be in master's history.
4. The workflow pulls the image on the builder VM, verifies its revision label, and commits deploy/request.json to the dedicated gitops branch. It creates that branch from master on the first request. Every dispatch gets a new UUID, even for the same image. Concurrent non-fast-forward pushes fail; no force push.
5. A production user-systemd timer reads the request history using a separate bare Git checkout and runs requests serially, once each. It uses the image digest and its bounded default command. Searches come from that image, not a host checkout.

Green Run app means queued, not successfully collected/imported. The production journal, local request state and raw manifest/sidecar report the actual outcome. There is no SSH deployment, incoming deployment API, automatic migration, daily collection schedule, or automatic retry after collection starts.

Runner and network configuration

ops/ci-runner/compose.yaml and config.yaml match the new setup at ~/containers/gitea-runner/: project gitea-runner, ./data bind, python-ci:docker://node:22-bookworm, capacity 1, timeout 45 minutes, and container.docker_host: "" for job socket access. The build job installs Docker CLI/Buildx; the daemon runs on the builder VM. No privileged jobs or arbitrary workflow host bind mounts are enabled.

Socket access grants jobs root-equivalent control of the builder VM, including other jobs and runner registration. Run only trusted workflows. A label is routing, not a security boundary. No production passwords, backups or raw data on that VM. GitOps avoids exposing production credentials to CI, but does not make arbitrary images trustworthy: the approved image can read the mounted DB password in production.

Keep runner configuration deployment manual. Inspect before copying repository files over deployed files; preserve data/.runner. Do not copy the old runner's named volume, unregister the working new runner, or use down -v during migration.

Runner/job extra_hosts map Gitea to 192.168.1.56 with HTTPS verification intact. Docker Engine also needs working registry DNS on the builder VM; container extra_hosts does not affect that daemon. If DNS still returns the wrong public address, carefully add this entry to the builder VM's /etc/hosts:

192.168.1.56 gitea.vandeplanque.com

Reserve that LAN address in DHCP. Production Docker also needs a working registry path. Never configure an insecure registry or disable TLS verification.

Gitea Actions secrets

Create these repository secrets in Settings → Actions → Secrets:

Secret	Purpose
REGISTRY_USER	tonio: owner of the current package namespace
REGISTRY_TOKEN	Token with write:package (includes package reads)
GITOPS_USER	Account allowed to write the repository's gitops branch
GITOPS_TOKEN	Separate token with write:repository

For the current user-owned tonio/car-tracker package, publishing must authenticate as tonio. A bot added as a repository collaborator cannot publish to that user's package namespace. Use separate narrowly scoped tokens; Gitea 1.23 token scopes are category-wide, not restricted to one repository. Private user-owned packages also require the owner's identity for reads; production uses a separate read:package token. Public user profiles can expose their packages publicly, regardless of linked repository visibility.

Gitea 1.23 does not enforce GitHub-style job permissions as a security boundary: set token scopes and account/branch access in Gitea itself. Protect master; restrict the state writer to gitops through branch protection where possible. Never rewrite the gitops branch's audit log/queue. Verify the package owner's visibility; packages do not inherit repository access controls.

Enter tokens only in the Gitea UI, not chat, commands, committed files or logs. The workflows use stdin for registry login and a temporary Git askpass helper, not tokens in command arguments or URLs. The built-in Actions token is not assumed to have write privileges on this older release. No production password or SSH deployment key belongs in Actions secrets.

Install the production consumer (manual, no run yet)

Follow host runtime configuration first: private directories, app.env, pgpass, TLS and HBA rule must exist. Migrations stay manual. The chosen stack root here is ~/containers/car-tracker, not the /srv example.

Disable the old SHA-polling agent before installing this version. It checked out releases in the development repository and retried failed scrapes. Neither behavior is used here. The obsolete deploy/prod.tag is removed; do not activate the old agent. It will see no desired tag after these changes are published.

systemctl --user disable --now car-tracker-reconcile.timer
systemctl --user status car-tracker-reconcile.service

If a service/job is active, let it finalize before replacing it; investigate uncertain imports. From the updated repository root, inspect destination files first, then:

install -m 0700 ops/gitops/reconcile.py ~/containers/car-tracker/reconcile.py
install -m 0600 ops/app/compose.runtime.yaml ~/containers/car-tracker/compose.runtime.yaml
mkdir -p ~/.config/systemd/user
install -m 0644 ops/gitops/car-tracker-reconcile.service ~/.config/systemd/user/
install -m 0644 ops/gitops/car-tracker-reconcile.timer ~/.config/systemd/user/
systemctl --user daemon-reload

Create a separate bare checkout only if desired.git doesn't already exist:

git clone --bare ssh://git@debian.tailf52e5b.ts.net:2222/tonio/car-tracker.git \
  ~/containers/car-tracker/desired.git
chmod 0700 ~/containers/car-tracker/desired.git

Prefer a read-only repository deploy key. Verify SSH host trust/access interactively before enabling the timer. The agent never checks out or resets the development repo.

For private registry pulls, run docker login gitea.vandeplanque.com interactively on production with a separate read-only package token, not the publisher token. Protect Docker's credential file or use a credential helper. Credentials remain local.

Both Compose stacks use car-tracker-app and the same subnet/storage. If the old pre-rename car-traker-app network occupies the subnet, remove it through the old project only when no job is active; do not delete unrelated networks.

Before dispatch, validate with a real published digest (never this placeholder):

CAR_TRACKER_IMAGE='gitea.vandeplanque.com/tonio/car-tracker@sha256:<published-digest>' \
  docker compose --env-file ~/containers/car-tracker/app.env \
  -f ~/containers/car-tracker/compose.runtime.yaml config --quiet

When ready:

sudo loginctl enable-linger tonio
systemctl --user enable --now car-tracker-reconcile.timer

Review gitops history first: enabling can execute all outstanding requests. If the branch does not exist yet, fetching fails and no app starts; later polls try again. Repository updates do not automatically replace deployed agent/Compose files.

Outcomes and recovery

The timer polls approximately every minute after the preceding cycle finishes. All request commits are processed in order, so two button presses between polls are not silently reduced to the latest image.

systemctl --user list-timers car-tracker-reconcile.timer
journalctl --user -u car-tracker-reconcile.service -n 100 --no-pager
ls ~/containers/car-tracker/state/requests/

Each state/requests/<uuid>.json records a request and pipeline exit code. finished means exited, not complete source coverage. See pipeline exit meanings. Raw files and pipeline.json stay private on production, never CI artifacts.

An attempt is atomically recorded before launching the collector. It is never rerun automatically, regardless of exit code. A crash/timeout may leave status: attempted, exit unknown. This blocks further requests until investigated. This favors a missed run over duplicate scraping; it is not an exactly-once guarantee.

Inspect car-tracker-run-<uuid>, the raw manifest and database outcome. Recover an uncertain import by replaying the exact finalized manifest, not scraping again. After investigation, retain the record and explicitly set its local status to resolved to unblock future requests. Do not delete state to force retries. Back up state/requests, raw files and the DB: losing the ledger can replay old jobs.

Registry/config failures before recording an attempt can retry because no collection started. No retries after the marker. Never overlap standalone CLI or recovery operations with a pipeline on the same output root.

For an intentional repeat or an older image, dispatch Run app again: a new UUID is recorded. Reverting a state commit containing a consumed UUID does not repeat it. "Rollback" here selects an older version for a new run; it does not undo DB data.

Build/image retention remains manual on the builder VM. Check docker system df and prune unused build cache deliberately; never prune volumes to free build space. No live collection is performed while implementing/testing these files.