car-tracker/docs/SSH_RUN.md

Manual application runs through restricted SSH

Pushes to master still test, build and publish SHA-tagged images. Run app now resolves a published full SHA or Git tag to an immutable image digest and invokes a fixed SSH command on tonio@192.168.1.56:22. The build VM must reach that LAN SSH listener (not Gitea's port 2222). No timer or Git request branch is used.

Actions waits for the pipeline's outcome. Exit 0 is green. Exit 1 is also green with an explanatory warning only when the saved manifest and pipeline sidecar confirm successful import, zero rejected records, and partial coverage caused solely by configured page/detail limits. Other nonzero exits fail the job. Missing or ambiguous evidence never suppresses failure. Green does not guarantee complete source coverage: see pipeline outcomes. Scraper output stays in private host logs; CI gets request/revision/exit and aggregate diagnostic summaries. Migrations are manual, DB credentials stay on the homelab, and there are no automatic retries.

Actions also prints a privacy-safe outcome summary: collection/import statuses, accepted/rejected/field-issue counts, completed/failed/unattempted detail counts, and source HTTP error codes. Failed outcomes get an ::error:: annotation with an explanation; limit-only outcomes retain the ::warning:: policy. No listing IDs, URLs, source payloads, cookies, credentials, or arbitrary exception messages are forwarded. Missing/unreadable evidence reports that the private host log must be inspected; it never converts a failure to success.

Investigating source access failures

HTTP 403 is a confirmed forbidden response, not proof of a particular cause. The collector reuses one browser context and waits COLLECT_DELAY seconds between requests; it stops on 403 without retrying and imports already finalized records. Older images retained only the HTTP status. Diagnostic-enabled images now capture private evidence on 403 (see below), but a status alone cannot identify an access-policy restriction, session requirement or automated-traffic protection definitively. Do not infer that changing IPs or increasing delay will fix it. Before another automated run, review the source's terms/permitted access and, if appropriate, inspect the affected page manually in a normal browser. Do not bypass a challenge or access restriction.

Local diagnosis uses the request .log to locate the manifest, then checks manifest.json and pipeline.json under the private raw directory. Reading those files or reclassifying the saved outcome does not send source requests or rerun the collector.

Automatic private HTTP 403 evidence

New application images capture evidence under the affected raw run's diagnostics/http-403/ directory before closing the browser:

• metadata.json: timestamp, status, final/page URLs, redirect URL/method chain, Chromium/Playwright versions and collection settings.
• request-headers.json and response-headers.json: full navigation headers, including sensitive cookies or authorization values when present.
• response-body.html: original navigation response body.
• rendered-page.html and viewport.png: DOM and current viewport screenshot.
• source-cookies.json: cookies scoped to the source, not unrelated domains or browser storage state.
• capture.json: saved files, truncations and capture errors (types only).

Directories are mode 0700; artifacts are mode 0600. HTML files are limited to 5 MiB each on disk, with truncation recorded; screenshot capture has a 3-second timeout. Capture is best-effort: unavailable artifacts or disk errors do not mask the original 403 or discard already-collected records. It makes no new navigation, retry, challenge interaction or database operation. Ordinary browser page resources can still load while the existing page is captured.

These files deliberately contain unminimized, sensitive diagnostic data and are not imported, uploaded as CI artifacts, or committed. Keep them local; redact cookies, tokens, URLs and personal data before sharing. Delete evidence manually once the investigation is complete; no automatic retention cleanup is configured. Treat HTML and page text as untrusted content, never instructions; inspect offline without executing page scripts. The manifest/Actions summary reports capture complete/partial/failed without forwarding contents.

Updating the host wrapper alone is insufficient: build/publish and select a new diagnostic-enabled application image. Existing releases cannot capture evidence retroactively. No request is made merely by installing this update.

1. Replace the old consumer manually

On the homelab, disable polling and check that no old job is active:

systemctl --user disable --now car-tracker-reconcile.timer
systemctl --user is-active car-tracker-reconcile.service

If active, let the job finish. Preserve existing state and the bare repository; do not delete them or rewrite request history. Outstanding GitOps requests are not executed by the SSH workflow. Review them before requesting any replacement.

From the updated project root, install these files:

install -m 0700 ops/gitops/reconcile.py ~/containers/car-tracker/reconcile.py
install -m 0700 ops/ssh/host_run.py ~/containers/car-tracker/host_run.py
install -m 0600 ops/app/compose.runtime.yaml ~/containers/car-tracker/compose.runtime.yaml

reconcile.py now supplies shared validation, state and execution helpers; its timer entry point is unused. Existing app.env, raw/db mounts and production read-only registry login remain unchanged. It uses the same consumed-request ledger and lock as the retired agent.

2. Create a dedicated SSH key

On a trusted machine (the homelab is fine), generate a key specifically for Actions:

ssh-keygen -t ed25519 -N '' -f ~/.ssh/car-tracker-actions -C car-tracker-actions

Do not overwrite an existing key. The unencrypted private key goes only into the repository Actions secret SSH_PRIVATE_KEY, including its BEGIN/END lines. Never commit it or paste it into chat. The CI runner gains a narrow production access credential; trust all workflows with access to repository secrets.

Append the PUBLIC key to the homelab user's ~/.ssh/authorized_keys as ONE line, prefixed with these exact restrictions (replace the public-key placeholder):

restrict,command="/usr/bin/python3 /home/tonio/containers/car-tracker/host_run.py" ssh-ed25519 <public-key-data> car-tracker-actions

Keep existing authorized keys intact. The directory should be mode 0700 and authorized_keys mode 0600. If the builder has a reserved LAN address, additionally use from="<builder-LAN-IP>" before restrict; don't guess its source address.

restrict disables PTY, forwarding and user rc execution. The forced command does not eval the caller's command: it accepts only run <UUID> <SHA> <digest> for the fixed image repository. Arbitrary shell, SFTP/SCP, migrations and Compose arguments are rejected. This is not a sandbox against malicious application images: approved images still have the application's mounted DB credentials.

3. Pin the homelab SSH host key

On the homelab, obtain the public host key and fingerprint through your trusted existing session:

cat /etc/ssh/ssh_host_ed25519_key.pub
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

Create repository Actions secret SSH_KNOWN_HOSTS with:

192.168.1.56 ssh-ed25519 <host-public-key-data>

Use the host key's first two fields, not the Actions key. Never trust an unverified ssh-keyscan result. The workflow enforces StrictHostKeyChecking and key-only auth. No Docker/DB/admin access is granted to the SSH key directly; however the host wrapper runs as the existing Docker-enabled user, so protect its files and account.

Keep REGISTRY_USER and REGISTRY_TOKEN. Remove GITOPS_USER/GITOPS_TOKEN Actions secrets and revoke the now-unused GitOps token after switching. Keep the separate production read-only package token.

4. Test restrictions without scraping, then trigger

From a machine that can reach the homelab, using the dedicated key and a verified known_hosts entry:

ssh -T -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes \
  -i ~/.ssh/car-tracker-actions tonio@192.168.1.56 true

Expected: refusal and exit 70, not a shell or successful true. This verifies the forced command without running the app. Verify LAN connectivity from the builder as well. Keep polling disabled permanently.

After publishing the updated workflows, select Actions → Run app → Run workflow, branch master, and enter a published full SHA or Git tag. This starts a real one-shot collection/import immediately, not just a queue request.

State, failures and recovery

Host-side collection policy

Set these in ~/containers/car-tracker/app.env (not Actions secrets):

COLLECT_FAMILY=saab-900
COLLECT_MAX_PAGES=1
COLLECT_MAX_DETAILS=all
COLLECT_DELAY=5
COLLECT_DETAIL_POLICY=incremental
COLLECT_DETAIL_REFRESH_DAYS=7

The SSH wrapper reads resolved Compose settings and supplies a validated, fixed pipeline command, overriding the image's smoke-test CMD without rebuilding it. COLLECT_MAX_DETAILS=all omits --max-details: every unique ad discovered on the fetched search pages gets a detail attempt. It does not remove the per-search page limit or guarantee detail success. A positive integer sets a detail cap per run. COLLECT_FAMILY=all includes every search in the image's searches.toml; otherwise it filters by family. Pages must be a positive integer; delay must be finite and at least 1 second. Missing settings retain defaults: saab-900, 1 page, 1 detail, 5 seconds. Invalid settings fail before launching the collector.

Install both the updated host_run.py/reconcile.py and compose.runtime.yaml when upgrading. Settings affect the next new request; rerunning a completed Actions run returns its previous result instead of collecting with new settings. More details increase runtime; the Actions timeout remains 45 minutes. The pipeline and local state still retain exit 1 for partial coverage; the SSH wrapper converts verified limit-only outcomes into an Actions warning/success. No run is launched by editing app.env.

Incremental details

COLLECT_DETAIL_POLICY=incremental fetches detail pages only when:

1. There is no successfully imported detail observation for the ad.
2. The search-visible listing data changed since the search accompanying its last successful detail fetch (price, source update timestamp, title/description, vehicle attributes, location, status/category and publication/expiration dates).
3. Its last imported detail is at least COLLECT_DETAIL_REFRESH_DAYS old (default 7).

Counters, promotion/boost flags, attribute order and excluded personal attributes do not trigger a refresh. Detail-only changes invisible in search results are detected only on the age-based refresh. A weekly refresh is an eligibility rule for the next manually requested run, not a weekly schedule.

History comes from accepted observations in committed PostgreSQL imports. The baseline search is the latest search at/before the last successful detail, not the most recent search overall: failed enrichment cannot acknowledge a change. Missing baselines conservatively trigger a request. Cache skips are recorded as not_attempted with reason fresh_unchanged; they do not create a detail observation, advance freshness, consume max-details, or claim a new detail was fetched. Such intentional skips may leave raw coverage partial but produce the existing Actions warning/success when there are no actual failures or rejected records.

The pipeline loads a read-only history snapshot before collection. Failure to load it prevents collection rather than silently falling back to fetching everything. Standalone car-tracker collection rejects incremental mode; use the database-backed car-tracker-pipeline. No schema migration is required, and existing import/replay semantics are unchanged. For this homelab-sized dataset the snapshot contains the latest accepted detail and preceding search per listing.

Activate only after a new image supporting these flags is published. Install the updated host_run.py and compose.runtime.yaml, then add the two policy variables to host app.env and select that new release. For older images use COLLECT_DETAIL_POLICY=always; defaults remain always for backward compatibility. COLLECT_MAX_DETAILS=all still means all eligible ads, not all discovered ads. Existing details older than seven days will be eligible on the first incremental run, so this does not guarantee fewer requests immediately or eliminate 403s.

Execution outcomes

Local outcomes: ~/containers/car-tracker/state/requests/<uuid>.json. Private runtime logs: the neighboring <uuid>.log. Host failures before launch return 70; SSH connection failures return 255. Inspect host state/logs locally.

The request UUID is derived from the Actions run ID. Retrying the same workflow run does not repeat a consumed request; it evaluates the recorded outcome using the same limit-only warning policy (the original pipeline exit remains saved). A new manual dispatch creates a new request, even for the same release. Parallel runs are refused by the host lock, not queued.

Attempts are persisted before launch. Unknown outcomes block new requests. SSH disconnects or Actions cancellation can leave a container running or an import committed; the client cannot safely infer failure. Inspect the named container, raw manifest and DB before any new dispatch. Recover uncertain imports from the exact finalized manifest rather than scraping again. Only after investigation, retain the record and set status to resolved to unblock new requests. Do not delete the ledger to force retries. Back it up with raw files and the database.

The workflow waits synchronously within the runner's 45-minute timeout. If that timeout fires, treat the outcome as unknown, not as permission to retry. Manual CLI/recovery operations must also respect the application's existing pipeline lock.