Youngtimer Market Observatory

First implementation: collect unnormalized listing observations from Leboncoin search pages and individual ads, including descriptions and source-provided locations. Finalized raw runs can now be cleaned and imported idempotently into PostgreSQL. Manual Docker Compose execution is available; scheduling and automated deployment remain deferred. Asking prices are not sale prices; disappearing listings do not prove a sale.

Run

Python 3.13+ and uv:

uv sync
uv run playwright install chromium
uv run car-traker --family saab-900 --max-pages 1 --max-details 1 --headed

The smoke command deliberately caps coverage and returns exit code 1 if further pages or details exist. Collect all configured searches and their individual ads with:

uv run car-traker --headed

Options: --config searches.toml, --output data/raw/leboncoin, --family, --max-pages 10 (per search), --delay 5 (seconds between requests), --headed. Detail visits are enabled by default, once per unique listing ID per run. --max-details N caps detail requests across the whole run (unlimited by default). --search-only disables detail visits. Delay applies to both search and detail requests. More detail visits mean substantially longer runs and greater exposure to source access failures. Linux may need Playwright's system libraries installed by an administrator. Headed mode requires a graphical session. Omitting --headed runs headless, but headless access failed in the initial live test on this machine. Do not run overlapping jobs. No credentials, proxies, CAPTCHA solvers, stealth plugins, or automatic access-block retries are used.

searches.toml defines broad nationwide car-category searches. Add queries/families there without changing code. Saab 900 and Volvo 850 queries do not restrict generation, body style, or engine. Mercedes 124 queries include chassis spellings, but ads using only badges such as "300 CE" may be missed. Search coverage is not a complete census, and false positives remain for later cleaning. Overlapping query results are intentionally preserved with their query provenance.

Project Structure

src/car_traker/       # Installed Python package
  __init__.py
  __main__.py        # python -m car_traker entry point
  collect.py         # Collector and CLI
  database.py        # PostgreSQL migrations and import CLI
  pipeline.py        # Manual collect -> raw -> import job
  container_entrypoint.py # Virtual display and graceful shutdown
  transform.py       # Pure, conservative silver field cleaning
  migrations/        # Versioned SQL schema and analytical views
tests/               # Offline tests
docs/POSTGRESQL.md    # Homelab provisioning, import, queries, database tests
docs/CI.md            # Gitea 1.23 CI and repository-scoped runner registration
ops/ci-runner/        # Optional CI runner infrastructure, not app deployment
ops/app/             # Manual homelab job and runtime configuration example
docs/DEPLOYMENT.md    # Storage, credentials, network access, and recovery
searches.toml        # Search configuration
pyproject.toml       # Dependencies, build configuration, and CLI entry point
data/                # Local snapshots (ignored by Git)

uv sync installs the package in editable mode from src/. Run commands from the repository root so the default configuration and data paths resolve correctly. uv run python -m car_traker also works as an alternative to uv run car-traker.

Silver Import

Run the importer on your laptop, connecting to PostgreSQL on your homelab. No local application database is required. See PostgreSQL setup and silver storage for the one-time homelab role/database setup, network access, and laptop connection configuration.

Once the remote connection is configured:

uv run car-traker-db migrate
uv run car-traker-db import data/raw/leboncoin/2026-09-08/ed585bcc-188d-401e-bc9f-8bc1318d4ca6/manifest.json

Importing an unchanged run again is a no-op. Search and detail observations retain separate timestamps; source coverage and data-quality issues remain explicit. SQL views expose typed listing history, latest observations, price history, coverage, and validation issues. Generation/trim inference, redaction, and scheduling are deferred.

Manual Homelab Job

car-traker-pipeline checks database connectivity/schema, collects raw observations, and imports that exact finalized run. Partial/failed collections remain marked as such even when their import succeeds. A lock prevents overlapping pipeline jobs on the same output directory, and pipeline.json records the import outcome separately from the immutable raw manifest.

For a laptop smoke run with the existing database connection configured:

uv run car-traker-pipeline --headed --family saab-900 --max-pages 1 --max-details 1

For Docker on the homelab, follow manual deployment first to configure private raw storage, the password file, and container network access. Then, from the homelab checkout:

docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml build app
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml run --rm app car-traker-db check
docker compose --env-file /srv/car-tracker/app.env -f ops/app/compose.yaml run --rm app

The default image command is bounded to Saab 900, one search page and one detail request. It uses headed Chromium through a virtual display, not a desktop/VNC service. No PostgreSQL server, scheduler, or automatic migration is added. Live Leboncoin access from the homelab still needs a manual smoke test.

Raw Format

Gzip-compressed JSON Lines (.jsonl.gz), with one observation per line and one file per search page or individual ad. JSON preserves nested attributes and original types, including missing fields and source strings. Gzip reduces storage using standard Python tools. CSV would flatten or discard structure. Parquet is better considered later, once a stable cleaned schema exists. These are filtered raw records, not full-page archives.

data/raw/leboncoin/YYYY-MM-DD/<run-uuid>/
  manifest.json
  search-001-page-0001.jsonl.gz
  search-001-page-0002.jsonl.gz
  detail-3248166132.jsonl.gz

Each line contains schema_version: 2, record_type (search or detail), source, run_id, UTC observed_at, and source payload. Search rows also include discovery family, search_text, search_url, and page; detail rows include listing_url. Join them using (source, run_id, payload.list_id). A listing discovered through multiple queries has multiple search rows but only one detail visit per run. Family is a search label, not a validated vehicle classification.

Search observations are saved before detail requests. Detail observations have their own timestamps and never overwrite search prices or other source values. Prices and vehicle attributes are not coerced or normalized. Publication timestamps remain distinct from observation timestamps. A new collection creates a new run, never overwriting previous runs. Existing version-1 snapshots remain unchanged; they lack the newly captured fields and cannot be enriched without recollection.

Files are finalized with an atomic rename. The manifest is updated after each page and records counts, source-reported totals/page limits, the planned searches, and coverage status. Use only finalized files referenced in a manifest for replay. An interrupted process may leave .tmp files; ignore them. A hard kill can leave status: running; treat that run as incomplete. Storage failures may also prevent the final manifest update.

complete means all pages reported by the source for each configured query were visited and every discovered unique listing had a successful detail visit (unless explicitly disabled), not that every relevant ad was found. Results can change during pagination; reported totals are diagnostic, not transactional guarantees. partial means a page/detail cap, unavailable detail, interruption, or failure after some pages; failed means no pages were saved. Exit code is 0 only for a complete run.

Search coverage and details.status are tracked separately. Detail coverage refers only to discovered IDs, not ads on unvisited search pages. details.listings maps each discovered ID to complete, unavailable, failed, or not_attempted, plus its snapshot filename or failure/skip reason. Successful empty discovery needs no detail requests; failed discovery does not imply successful detail coverage. HTTP 404/410 marks a detail unavailable and continues, without inferring a sale. A block (including 403/429), malformed payload, or repeated search page stops the whole run without retries. Inspect error_type, reason, and http_status in the manifest; exception bodies are not stored to avoid leaking page contents.

Read a snapshot without cleaning:

import gzip
import json

with gzip.open("data/raw/leboncoin/<date>/<run>/search-001-page-0001.jsonl.gz", "rt") as f:
    for line in f:
        observation = json.loads(line)
        print(observation["observed_at"], observation["payload"]["price"])

Privacy And Retention

The expanded capture was explicitly requested for later vehicle analysis and a map dashboard. It retains listing IDs, timestamps, category/status, prices, title (subject), description (body), URL, brand, listing options/counters, and the full source location object. This includes city/postcode, latitude/longitude, and location provenance/precision metadata when supplied. Coordinates are not proof of an exact vehicle or seller address; they may identify a city centroid or an approximate point. Missing coordinates stay missing; no geocoding is performed.

Attributes are now retained by default, including newly introduced vehicle fields, except known seller/profile/rating/contact fields, plate/VIN identifiers, and the external vehicle-history report link. Structured seller objects, contact buttons, photos, and related-listing objects are excluded. Unknown top-level fields remain excluded; this is not a promise to capture every field the website may ever expose.

Titles and descriptions are stored verbatim, without redaction. They can contain names, phone numbers, addresses, or other personal information supplied by sellers. New attribute keys and location fields can also contain unexpected information; the field filter is not an anonymization guarantee. Do not publish raw text or location records directly in the dashboard. Add privacy review/redaction and an appropriate map precision policy before exposing a public dataset or dashboard.

Listing IDs are linkable. Treat the raw directory as private: restrict filesystem access, do not publish snapshots, and exclude it from Git (already configured). New run directories are created with owner-only permissions (0700 on Linux). Initial retention policy: delete whole run directories after 30 days, including backups. Retention enforcement is currently manual, not an automatic deletion job; set it up before unattended collection. Full page HTML is processed in browser memory, not archived. Do not enable browser tracing or save browser profiles by default.

Source Access

On 2026-09-08, direct HTTP search returned 403, while the inspection browser loaded a Saab 900 search and exposed 35 records in #__NEXT_DATA__.props.pageProps.searchData (101 reported results, three pages). The collector therefore uses Playwright Chromium and reads that embedded JSON. Individual ads expose #__NEXT_DATA__.props.pageProps.ad; inspection confirmed a description was populated there while the search record's body was empty. Access can differ across browser environments, IP addresses, and runs. This observation is not proof of reliable daily access or permission to collect; review applicable source terms and requirements before unattended deployment. A challenge or changed payload is a failure, not zero ads.

The original search-only Python CLI was also tested on 2026-09-08: headless collection failed before saving records; headed Chromium saved 35 real Saab 900 observations. A subsequent headed pagination test saved page 1, then received HTTP 403 on page 2. Its manifest correctly reports partial, SearchAccessError, and http_status: 403. Live first-page acquisition is verified; full pagination, all-family coverage, and reliable unattended daily collection are not verified. The collector stops on that access failure rather than retrying or presenting partial data as complete.

The expanded CLI was tested with --max-pages 1 --max-details 1 --headed on the same date: it saved 35 search observations and one successful individual-ad snapshot under run ed585bcc-188d-401e-bc9f-8bc1318d4ca6. The run is intentionally partial because of both limits. Full detail coverage and reliable unattended collection remain unverified.

Tests

uv sync --locked --dev
uv run --locked ruff check src tests
uv run python -m unittest discover -s tests -v

Tests use clearly synthetic in-memory responses and temporary storage. They never contact Leboncoin, and fixtures are not mixed with real observations.

Gitea CI runs linting, the full suite against disposable PostgreSQL, and package validation on pushes and pull requests. Enable Repository Actions and register the documented python-ci runner before expecting remote jobs to execute. CI never scrapes Leboncoin or connects to the homelab application database.