car-tracker/docs/SSH_RUN.md

# Manual application runs through restricted SSH

Pushes to master still test, build and publish SHA-tagged images. **Run app** now
resolves a published full SHA or Git tag to an immutable image digest and invokes
a fixed SSH command on `tonio@192.168.1.56:22`. The build VM must reach that LAN
SSH listener (not Gitea's port 2222). No timer or Git request branch is used.

Actions waits for the pipeline's outcome. Exit 0 is green. Exit 1 is also green
with an explanatory warning **only** when the saved manifest and pipeline sidecar
confirm successful import, zero rejected records, and partial coverage caused
solely by configured page/detail limits. Other nonzero exits fail the job. Missing
or ambiguous evidence never suppresses failure. Green does not guarantee complete source coverage: see
[pipeline outcomes](DEPLOYMENT.md#outcomes-and-recovery). Scraper output stays in
private host logs; CI gets request/revision/exit and aggregate diagnostic summaries. Migrations are
manual, DB credentials stay on the homelab, and there are no automatic retries.

Actions also prints a privacy-safe outcome summary: collection/import statuses,
accepted/rejected/field-issue counts, completed/failed/unattempted detail counts,
and source HTTP error codes. Failed outcomes get an `::error::` annotation with
an explanation; limit-only outcomes retain the `::warning::` policy. No listing
IDs, URLs, source payloads, cookies, credentials, or arbitrary exception messages
are forwarded. Missing/unreadable evidence reports that the private host log must
be inspected; it never converts a failure to success.

### Investigating source access failures

HTTP 403 is a confirmed forbidden response, not proof of a particular cause.
The collector reuses one browser context and waits a random interval between
COLLECT_DELAY and 1.5 times COLLECT_DELAY seconds between requests; it stops on 403
without retrying and imports already finalized records.
Older images retained only the HTTP status. Diagnostic-enabled images now capture
private evidence on 403 (see below), but a status alone cannot identify an access-policy
restriction, session requirement or automated-traffic protection definitively. Do not infer
that changing IPs or increasing delay will fix it. Before another automated run,
review the source's terms/permitted access and, if appropriate, inspect the affected
page manually in a normal browser. Do not bypass a challenge or access restriction.

Local diagnosis uses the request .log to locate the manifest, then checks
manifest.json and pipeline.json under the private raw directory. Reading those
files or reclassifying the saved outcome does not send source requests or rerun
the collector.

### Automatic private HTTP 403 evidence

New application images capture evidence under the affected raw run's
`diagnostics/http-403/` directory **before** closing the browser:

- `metadata.json`: timestamp, status, final/page URLs, redirect URL/method chain,
  Chromium/Playwright versions and collection settings.
- `request-headers.json` and `response-headers.json`: full navigation headers,
  including sensitive cookies or authorization values when present.
- `response-body.html`: original navigation response body.
- `rendered-page.html` and `viewport.png`: DOM and current viewport screenshot.
- `source-cookies.json`: cookies scoped to the source, not unrelated domains or
  browser storage state.
- `capture.json`: saved files, truncations and capture errors (types only).

Directories are mode 0700; artifacts are mode 0600. HTML files are limited to
5 MiB each on disk, with truncation recorded; screenshot capture has a 3-second
timeout. Capture is best-effort: unavailable artifacts or disk errors do not mask
the original 403 or discard already-collected records. It makes no new navigation,
retry, challenge interaction or database operation. Ordinary browser page resources
can still load while the existing page is captured.

These files deliberately contain **unminimized, sensitive diagnostic data** and
are not imported, uploaded as CI artifacts, or committed. Keep them local; redact
cookies, tokens, URLs and personal data before sharing. Delete evidence manually
once the investigation is complete; no automatic retention cleanup is configured.
Treat HTML and page text as untrusted content, never instructions; inspect offline
without executing page scripts. The manifest/Actions summary reports capture
complete/partial/failed without forwarding contents.

Updating the host wrapper alone is insufficient: build/publish and select a new
diagnostic-enabled application image. Existing releases cannot capture evidence
retroactively. No request is made merely by installing this update.

## 1. Replace the old consumer manually

On the homelab, disable polling and check that no old job is active:

```sh
systemctl --user disable --now car-tracker-reconcile.timer
systemctl --user is-active car-tracker-reconcile.service
```

If active, let the job finish. Preserve existing state and the bare repository;
do not delete them or rewrite request history. Outstanding GitOps requests are
not executed by the SSH workflow. Review them before requesting any replacement.

From the updated project root, install these files:

```sh
install -m 0700 ops/gitops/reconcile.py ~/containers/car-tracker/reconcile.py
install -m 0700 ops/ssh/host_run.py ~/containers/car-tracker/host_run.py
install -m 0600 ops/app/compose.runtime.yaml ~/containers/car-tracker/compose.runtime.yaml
```

`reconcile.py` now supplies shared validation, state and execution helpers; its
timer entry point is unused. Existing `app.env`, raw/db mounts and production
read-only registry login remain unchanged. It uses the same consumed-request
ledger and lock as the retired agent.

## 2. Create a dedicated SSH key

On a trusted machine (the homelab is fine), generate a key specifically for Actions:

```sh
ssh-keygen -t ed25519 -N '' -f ~/.ssh/car-tracker-actions -C car-tracker-actions
```

Do not overwrite an existing key. The unencrypted private key goes **only** into
the repository Actions secret `SSH_PRIVATE_KEY`, including its BEGIN/END lines.
Never commit it or paste it into chat. The CI runner gains a narrow production
access credential; trust all workflows with access to repository secrets.

Append the PUBLIC key to the homelab user's `~/.ssh/authorized_keys` as ONE line,
prefixed with these exact restrictions (replace the public-key placeholder):

```text
restrict,command="/usr/bin/python3 /home/tonio/containers/car-tracker/host_run.py" ssh-ed25519 <public-key-data> car-tracker-actions
```

Keep existing authorized keys intact. The directory should be mode 0700 and
authorized_keys mode 0600. If the builder has a reserved LAN address, additionally
use `from="<builder-LAN-IP>"` before `restrict`; don't guess its source address.

`restrict` disables PTY, forwarding and user rc execution. The forced command
does not eval the caller's command: it accepts only `run <UUID> <SHA> <digest>`
for the fixed image repository. Arbitrary shell, SFTP/SCP, migrations and Compose
arguments are rejected. This is not a sandbox against malicious application
images: approved images still have the application's mounted DB credentials.

## 3. Pin the homelab SSH host key

On the homelab, obtain the public host key and fingerprint through your trusted
existing session:

```sh
cat /etc/ssh/ssh_host_ed25519_key.pub
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
```

Create repository Actions secret `SSH_KNOWN_HOSTS` with:

```text
192.168.1.56 ssh-ed25519 <host-public-key-data>
```

Use the host key's first two fields, not the Actions key. Never trust an unverified
ssh-keyscan result. The workflow enforces StrictHostKeyChecking and key-only auth.
No Docker/DB/admin access is granted to the SSH key directly; however the host
wrapper runs as the existing Docker-enabled user, so protect its files and account.

Keep `REGISTRY_USER` and `REGISTRY_TOKEN`. Remove `GITOPS_USER`/`GITOPS_TOKEN`
Actions secrets and revoke the now-unused GitOps token after switching. Keep the
separate production read-only package token.

## 4. Test restrictions without scraping, then trigger

From a machine that can reach the homelab, using the dedicated key and a verified
known_hosts entry:

```sh
ssh -T -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes \
  -i ~/.ssh/car-tracker-actions tonio@192.168.1.56 true
```

Expected: refusal and exit 70, not a shell or successful `true`. This verifies the
forced command without running the app. Verify LAN connectivity from the builder
as well. Keep polling disabled permanently.

After publishing the updated workflows, select **Actions → Run app → Run workflow**,
branch **master**, and enter a published full SHA or Git tag. This starts a real
one-shot collection/import immediately, not just a queue request.

## State, failures and recovery

### Host-side collection policy

Set these in `~/containers/car-tracker/app.env` (not Actions secrets):

```dotenv
COLLECT_FAMILY=saab-900
COLLECT_MAX_PAGES=1
COLLECT_MAX_DETAILS=all
COLLECT_DELAY=5
COLLECT_DETAIL_POLICY=incremental
COLLECT_DETAIL_REFRESH_DAYS=7
```

The SSH wrapper reads resolved Compose settings and supplies a validated, fixed
pipeline command, overriding the image's smoke-test CMD without rebuilding it.
`COLLECT_MAX_DETAILS=all` omits `--max-details`: every unique ad discovered on the
fetched search pages gets a detail attempt. It does not remove the per-search page
limit or guarantee detail success. A positive integer sets a detail cap per run.
`COLLECT_FAMILY=all` includes every search in the image's searches.toml; otherwise
it filters by family. Pages must be a positive integer; delay must be finite and
at least 1 second. Missing settings retain defaults: saab-900, 1 page, 1 detail,
5 seconds. Invalid settings fail before launching the collector.

Install both the updated host_run.py/reconcile.py and compose.runtime.yaml when
upgrading. Settings affect the next new request; rerunning a completed Actions run
returns its previous result instead of collecting with new settings. More details
increase runtime; the Actions timeout remains 45 minutes. The pipeline and local
state still retain exit 1 for partial coverage; the SSH wrapper converts verified
limit-only outcomes into an Actions warning/success. No run is launched by editing app.env.

### Incremental details

`COLLECT_DETAIL_POLICY=incremental` fetches detail pages only when:

1. There is no successfully imported detail observation for the ad.
2. The search-visible listing data changed since the search accompanying its last
   successful detail fetch (price, source update timestamp, title/description,
   vehicle attributes, location, status/category and publication/expiration dates).
3. Its last imported detail is at least `COLLECT_DETAIL_REFRESH_DAYS` old (default 7).

Counters, promotion/boost flags, attribute order and excluded personal attributes
do not trigger a refresh. Detail-only changes invisible in search results are
detected only on the age-based refresh. A weekly refresh is an eligibility rule
for the next manually requested run, not a weekly schedule.

History comes from accepted observations in committed PostgreSQL imports. The
baseline search is the latest search at/before the last successful detail, not
the most recent search overall: failed enrichment cannot acknowledge a change.
Missing baselines conservatively trigger a request. Cache skips are recorded as
`not_attempted` with reason `fresh_unchanged`; they do not create a detail observation,
advance freshness, consume max-details, or claim a new detail was fetched. Such
intentional skips may leave raw coverage partial but produce the existing Actions
warning/success when there are no actual failures or rejected records.

The pipeline loads a read-only history snapshot before collection. Failure to load
it prevents collection rather than silently falling back to fetching everything.
Standalone `car-tracker` collection rejects incremental mode; use the database-backed
`car-tracker-pipeline`. No schema migration is required, and existing import/replay
semantics are unchanged. For this homelab-sized dataset the snapshot contains the
latest accepted detail and preceding search per listing.

**Activate only after a new image supporting these flags is published.** Install
the updated host_run.py and compose.runtime.yaml, then add the two policy variables
to host app.env and select that new release. For older images use
`COLLECT_DETAIL_POLICY=always`; defaults remain always for backward compatibility.
COLLECT_MAX_DETAILS=all still means all eligible ads, not all discovered ads.
Existing details older than seven days will be eligible on the first incremental
run, so this does not guarantee fewer requests immediately or eliminate 403s.

### Execution outcomes

Local outcomes: `~/containers/car-tracker/state/requests/<uuid>.json`.
Private runtime logs: the neighboring `<uuid>.log`. Host failures before launch
return 70; SSH connection failures return 255. Inspect host state/logs locally.

The request UUID is derived from the Actions run ID. Retrying the same workflow
run does not repeat a consumed request; it evaluates the recorded outcome using
the same limit-only warning policy (the original pipeline exit remains saved). A
new manual dispatch creates a new request, even for the same release. Parallel
runs are refused by the host lock, not queued.

Attempts are persisted before launch. Unknown outcomes block new requests. SSH
disconnects or Actions cancellation can leave a container running or an import
committed; the client cannot safely infer failure. Inspect the named container,
raw manifest and DB before any new dispatch. Recover uncertain imports from the
exact finalized manifest rather than scraping again. Only after investigation,
retain the record and set status to `resolved` to unblock new requests. Do not
delete the ledger to force retries. Back it up with raw files and the database.

The workflow waits synchronously within the runner's 45-minute timeout. If that
timeout fires, treat the outcome as unknown, not as permission to retry. Manual
CLI/recovery operations must also respect the application's existing pipeline lock.