car-tracker/README.md

# Youngtimer Market Observatory

First implementation: collect unnormalized listing observations from Leboncoin
search pages and individual ads, including descriptions and source-provided locations.
Finalized raw runs can now be cleaned and imported idempotently into PostgreSQL.
Manual Docker Compose execution is available; scheduling and automated deployment
remain deferred.
Asking prices are not sale prices; disappearing listings do not prove a sale.

## Run

Python 3.13+ and `uv`:

```sh
uv sync
uv run playwright install chromium
uv run car-tracker --family saab-900 --max-pages 1 --max-details 1 --headed
```

The smoke command deliberately caps coverage and returns exit code 1 if further
pages or details exist. Collect all configured searches and their individual ads with:

```sh
uv run car-tracker --headed
```

Options: `--config searches.toml`, `--output data/raw/leboncoin`, `--family`,
`--max-pages 10` (per search), `--delay 5` (seconds between requests), `--headed`.
Detail visits are enabled by default, once per unique listing ID per run.
`--max-details N` caps detail requests across the whole run (unlimited by default).
`--search-only` disables detail visits. Delay applies to both search and detail
requests. More detail visits mean substantially longer runs and greater exposure
to source access failures.
Linux may need Playwright's system libraries installed by an administrator.
Headed mode requires a graphical session. Omitting `--headed` runs headless, but
headless access failed in the initial live test on this machine.
Do not run overlapping jobs. No credentials, proxies, CAPTCHA solvers, stealth
plugins, or automatic access-block retries are used.

`searches.toml` defines broad nationwide car-category searches. Add queries/families
there without changing code. Saab 900 and Volvo 850 queries do not restrict
generation, body style, or engine. Mercedes 124 queries include chassis spellings,
but ads using only badges such as "300 CE" may be missed. Search coverage is not a
complete census, and false positives remain for later cleaning. Overlapping query
results are intentionally preserved with their query provenance.

## Project Structure

```text
src/car_tracker/      # Installed Python package
  __init__.py
  __main__.py        # python -m car_tracker entry point
  collect.py         # Collector and CLI
  database.py        # PostgreSQL migrations and import CLI
  pipeline.py        # Manual collect -> raw -> import job
  container_entrypoint.py # Virtual display and graceful shutdown
  transform.py       # Pure, conservative silver field cleaning
  migrations/        # Versioned SQL schema and analytical views
tests/               # Offline tests
docs/POSTGRESQL.md    # Homelab provisioning, import, queries, database tests
docs/CI.md            # Gitea 1.23 CI and repository-scoped runner registration
ops/ci-runner/        # Optional CI runner infrastructure, not app deployment
ops/app/             # Manual homelab job and runtime configuration example
docs/DEPLOYMENT.md    # Storage, credentials, network access, and recovery
searches.toml        # Search configuration
pyproject.toml       # Dependencies, build configuration, and CLI entry point
data/                # Local snapshots (ignored by Git)
```

`uv sync` installs the package in editable mode from `src/`. Run commands from the
repository root so the default configuration and data paths resolve correctly.
`uv run python -m car_tracker` also works as an alternative to `uv run car-tracker`.

## Silver Import

Run the importer **on your laptop**, connecting to PostgreSQL **on your homelab**.
No local application database is required. See
[PostgreSQL setup and silver storage](docs/POSTGRESQL.md) for the one-time homelab
role/database setup, network access, and laptop connection configuration.

Once the remote connection is configured:

```sh
uv run car-tracker-db migrate
uv run car-tracker-db import data/raw/leboncoin/2026-09-08/ed585bcc-188d-401e-bc9f-8bc1318d4ca6/manifest.json
```

Importing an unchanged run again is a no-op. Search and detail observations retain
separate timestamps; source coverage and data-quality issues remain explicit.
SQL views expose typed listing history, latest observations, price history, coverage,
and validation issues. Generation/trim inference, redaction, and scheduling are deferred.

## Manual Homelab Job

For a manual **Run app** button in Gitea Actions, see [SSH jobs](docs/SSH_RUN.md).
CI publishes tested images; the button runs a digest-pinned image through a
restricted SSH command using host-local DB credentials. No automatic
migrations or retries of failed scrapes are introduced.

`car-tracker-pipeline` checks database connectivity/schema, collects raw observations,
and imports that exact finalized run. Partial/failed collections remain marked as
such even when their import succeeds. A lock prevents overlapping pipeline jobs on
the same output directory, and `pipeline.json` records the import outcome separately
from the immutable raw manifest.

For a laptop smoke run with the existing database connection configured:

```sh
uv run car-tracker-pipeline --headed --family saab-900 --max-pages 1 --max-details 1
```

For Docker on the homelab, follow [manual deployment](docs/DEPLOYMENT.md) first to
configure private raw storage, the password file, and container network access.
Then, from the homelab checkout:

```sh
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml build app
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml run --rm app car-tracker-db check
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml run --rm app
```

The default image command is bounded to Saab 900, one search page and one detail
request. It uses headed Chromium through a virtual display, not a desktop/VNC
service. No PostgreSQL server, scheduler, or automatic migration is added.
Live Leboncoin access from the homelab still needs a manual smoke test.

## Raw Format

**Gzip-compressed JSON Lines (`.jsonl.gz`)**, with one observation per line and one
file per search page or individual ad. JSON preserves nested attributes and original types, including
missing fields and source strings. Gzip reduces storage using standard Python tools.
CSV would flatten or discard structure. Parquet is better considered later, once a
stable cleaned schema exists. These are filtered raw records, not full-page archives.

```text
data/raw/leboncoin/YYYY-MM-DD/<run-uuid>/
  manifest.json
  search-001-page-0001.jsonl.gz
  search-001-page-0002.jsonl.gz
  detail-3248166132.jsonl.gz
```

Each line contains `schema_version: 2`, `record_type` (`search` or `detail`), `source`,
`run_id`, UTC `observed_at`, and source `payload`. Search rows also include discovery
`family`, `search_text`, `search_url`, and `page`; detail rows include `listing_url`.
Join them using `(source, run_id, payload.list_id)`. A listing discovered through
multiple queries has multiple search rows but only one detail visit per run.
Family is a search label, not a validated vehicle classification.

Search observations are saved before detail requests. Detail observations have
their own timestamps and never overwrite search prices or other source values.
Prices and vehicle attributes are not coerced or normalized. Publication timestamps
remain distinct from observation timestamps. A new collection creates a new run,
never overwriting previous runs. Existing version-1 snapshots remain unchanged;
they lack the newly captured fields and cannot be enriched without recollection.

Files are finalized with an atomic rename. The manifest is updated after each page
and records counts, source-reported totals/page limits, the planned searches, and
coverage status. Use only finalized files referenced in a manifest for replay.
An interrupted process may leave `.tmp` files; ignore them. A hard kill can leave
`status: running`; treat that run as incomplete. Storage failures may also prevent
the final manifest update.

`complete` means all pages reported by the source for each configured query were
visited and every discovered unique listing had a successful detail visit (unless
explicitly disabled), not that every relevant ad was found. Results can change
during pagination; reported totals are diagnostic, not transactional guarantees.
`partial` means a page/detail cap, unavailable detail, interruption, or failure after
some pages; `failed` means no pages were saved. Exit code is 0 only for a complete run.

Search coverage and `details.status` are tracked separately. Detail coverage refers
only to discovered IDs, not ads on unvisited search pages. `details.listings` maps
each discovered ID to `complete`, `unavailable`, `failed`, or `not_attempted`, plus
its snapshot filename or failure/skip reason. Successful empty discovery needs no
detail requests; failed discovery does not imply successful detail coverage.
HTTP 404/410 marks a detail unavailable and continues, without inferring a sale.
A block (including 403/429), malformed payload, or repeated search page stops the
whole run without retries. Inspect `error_type`, `reason`, and `http_status` in the
manifest; exception bodies are not stored to avoid leaking page contents.

Read a snapshot without cleaning:

```python
import gzip
import json

with gzip.open("data/raw/leboncoin/<date>/<run>/search-001-page-0001.jsonl.gz", "rt") as f:
    for line in f:
        observation = json.loads(line)
        print(observation["observed_at"], observation["payload"]["price"])
```

## Privacy And Retention

The expanded capture was explicitly requested for later vehicle analysis and a map
dashboard. It retains listing IDs, timestamps, category/status, prices, title
(`subject`), description (`body`), URL, brand, listing options/counters, and the full
source `location` object. This includes city/postcode, latitude/longitude, and
location provenance/precision metadata when supplied. Coordinates are **not proof
of an exact vehicle or seller address**; they may identify a city centroid or an
approximate point. Missing coordinates stay missing; no geocoding is performed.

Attributes are now retained by default, including newly introduced vehicle fields,
except known seller/profile/rating/contact fields, plate/VIN identifiers, and the
external vehicle-history report link. Structured seller objects, contact buttons,
photos, and related-listing objects are excluded. Unknown top-level fields remain
excluded; this is not a promise to capture every field the website may ever expose.

Titles and descriptions are stored **verbatim, without redaction**. They can contain
names, phone numbers, addresses, or other personal information supplied by sellers.
New attribute keys and location fields can also contain unexpected information;
the field filter is not an anonymization guarantee. Do not publish raw text or
location records directly in the dashboard. Add privacy review/redaction and an
appropriate map precision policy before exposing a public dataset or dashboard.

Listing IDs are linkable. Treat the raw directory as private: restrict filesystem
access, do not publish snapshots, and exclude it from Git (already configured).
New run directories are created with owner-only permissions (`0700` on Linux).
Initial retention policy: delete whole run directories after **30 days**, including
backups. Retention enforcement is currently manual, not an automatic deletion job;
set it up before unattended collection. Full page HTML is processed in browser
memory, not archived. Do not enable browser tracing or save browser profiles by default.

## Source Access

On 2026-09-08, direct HTTP search returned 403, while the inspection browser loaded
a Saab 900 search and exposed 35 records in
`#__NEXT_DATA__.props.pageProps.searchData` (101 reported results, three pages).
The collector therefore uses Playwright Chromium and reads that embedded JSON.
Individual ads expose `#__NEXT_DATA__.props.pageProps.ad`; inspection confirmed a
description was populated there while the search record's `body` was empty.
Access can differ across browser environments,
IP addresses, and runs. This observation is not proof of reliable daily access or
permission to collect; review applicable source terms and requirements before
unattended deployment. A challenge or changed payload is a failure, not zero ads.

The original search-only Python CLI was also tested on 2026-09-08: headless collection failed
before saving records; headed Chromium saved **35 real Saab 900 observations**.
A subsequent headed pagination test saved page 1, then received HTTP 403 on page 2.
Its manifest correctly reports `partial`, `SearchAccessError`, and `http_status: 403`.
Live first-page acquisition is verified; full pagination, all-family coverage, and
reliable unattended daily collection are **not** verified. The collector stops on
that access failure rather than retrying or presenting partial data as complete.

The expanded CLI was tested with `--max-pages 1 --max-details 1 --headed` on the
same date: it saved 35 search observations and one successful individual-ad snapshot
under run `ed585bcc-188d-401e-bc9f-8bc1318d4ca6`. The run is intentionally partial
because of both limits. Full detail coverage and reliable unattended collection
remain unverified.

## Tests

```sh
uv sync --locked --dev
uv run --locked ruff check src tests
uv run python -m unittest discover -s tests -v
```

Tests use clearly synthetic in-memory responses and temporary storage. They never
contact Leboncoin, and fixtures are not mixed with real observations.

[Gitea CI](docs/CI.md) runs linting, the full suite against disposable PostgreSQL,
and package validation on pushes and pull requests. Enable Repository Actions and
register the documented `python-ci` runner before expecting remote jobs to execute.
CI never scrapes Leboncoin or connects to the homelab application database.