car-tracker/PROJECT_PLAN.md

# Youngtimer Market Observatory: Project Handoff

## Implementation Update (2026-09-08)

The user subsequently selected live Leboncoin collection and explicitly requested
titles, descriptions, maximum vehicle attributes, and source-provided precise
locations for a future map dashboard, including visits to individual ads. This
supersedes the earlier outstanding input decision and the exclusions of free text
and precise location below. Seller/contact objects and photos remain excluded;
free text may still contain personal information and must remain private until
reviewed/redacted for downstream publication. Source coordinates may be approximate.

The collector now lives in `src/car_tracker/`, storing separate search/detail
observations in versioned compressed JSON Lines with a coverage manifest. See
`README.md` for current commands, capture policy, retention, and verified live
results. The original handoff below remains planning context, not a description
of the current implementation.

The next milestone is implemented: SQL migrations, conservative silver cleaning,
atomic/idempotent PostgreSQL import, and views for observations, price history,
coverage, and quality issues. Development/CLI execution is on the user's laptop;
the intended database is the existing remote homelab PostgreSQL instance. Local
PostgreSQL containers are disposable tests only. Homelab provisioning and remote
connection setup remain user actions documented in `docs/POSTGRESQL.md`; no homelab
database has been accessed or modified during implementation.

The manual homelab job is now packaged in `Dockerfile` and `ops/app/compose.yaml`.
`car-tracker-pipeline` performs a database preflight, bounded collection, and an
idempotent import of the exact returned manifest, retaining partial coverage and
separate operational outcomes. There is no scheduler or automatic deployment.
The optional Gitea Actions/restricted SSH workflow is documented in
`docs/SSH_RUN.md`: tests/build/publish occur on a dedicated runner VM; explicit
requests run selected images on the production VM over SSH. It is not a daily
collection schedule, and migrations remain manual.
See `docs/DEPLOYMENT.md`; container tests use isolated PostgreSQL 17 and synthetic
browser responses. Actual live access from the homelab still needs verification.

## Purpose

Build a French used-car asking-price observatory focused initially on youngtimers.
The goals are market analytics and a data engineering portfolio, not primarily a
car-buying assistant. The user is an experienced data engineer.

Track listing observations over time, normalize vehicle variants, preserve price
history, and eventually identify listings priced below suitable comparables.
Asking prices are not transaction prices. No confirmed sale-price source has been
identified.

## Confirmed Requirements

- Initial families: Saab 900, Mercedes-Benz 124-series, and Volvo 850.
- Include broad variant coverage: both Saab 900 generations; Mercedes sedans,
  estates, coupes, and cabriolets; Volvo 850 body styles and engine variants.
- Keep the vehicle catalog extensible to many more models later.
- Geographic scope: all France.
- Target collection cadence: daily, subject to source access feasibility.
- Preferred first platform: Leboncoin.
- No paid data sources.
- Python and SQL; FastAPI if an API becomes necessary, but no API needed initially.
- Polars is optional, not a requirement. Use it only where it adds value over SQL.
- No distributed infrastructure.
- Docker Compose deployment on the user's homelab.
- No deadline or fixed time budget.
- First milestone: a working end-to-end pipeline, not just an exploratory notebook.

## Environment

- Homelab: Ryzen 5 7500F, 32 GB RAM.
- PostgreSQL already runs on a Debian VM; reuse it rather than require a second
  production database.
- Repository hosted on the user's own Gitea instance.
- Current workspace: `/home/tonio/Workspace/perso/car-tracker`.
- At inspection, the repository contained `.gitignore`, `.python-version`,
  `main.py`, `pyproject.toml`, and an empty `README.md`.
- `pyproject.toml` specifies Python >=3.13, project name `car-tracker`, version
  `0.1.0`, and no dependencies.
- The worktree was clean before this handoff document was added.
- No application implementation was changed during the planning discussion.
- The user previously wrote a Leboncoin scraper, but no longer has its code or
  sample responses.

Reinspect the repository before implementation; do not assume it is unchanged.

## Source Access

The user prefers Leboncoin and has experience scraping it. They stated that
publicly available data can be collected under French law, then argued that
Leboncoin's allowance for ChatGPT bots permits this project to scrape it.


### Outstanding Decision

The previous agent offered two paths:

1. Build the source-independent foundation using clearly synthetic snapshots
   while source access is investigated.
2. Investigate alternative free sources before implementing the foundation.

The recommendation was path 1, but the user has not selected either path. Their
latest request was only to write this handoff document. Do not treat synthetic
data, an alternative platform, or live Leboncoin collection as an agreed choice.

## Proposed Architecture

These are recommendations discussed with the user, not a fully approved technical
design or an implemented system.

### Collection and Processing

- A small Python CLI with source-specific collection/parsing separated from the
  normalized import path.
- Prefer a direct HTTP client where an accessible source exposes the required
  data without browser rendering.
- Use Playwright when rendering or interaction is genuinely required. The user
  suggested it but is open to another suitable approach.
- Do not choose a browser framework before inspecting the source. Playwright
  does not itself resolve access restrictions or guarantee anti-bot compatibility.
- No need initially for Scrapy, distributed queues, or a large orchestrator.
- Configurable model families and source search definitions rather than
  hardcoded logic for three models.

### Persistence and Scheduling

- Existing PostgreSQL for collection runs, listings, observations, normalized
  attributes, and analytical SQL views.
- Local mounted storage for minimized source snapshots where retention is
  appropriate, enabling replay and parser debugging.
- Snapshot storage outside Git, with a documented retention policy.
- A Compose job triggered daily by a host systemd timer is the proposed simplest
  deployment. Compose is confirmed; the timer mechanism is not yet finalized.
- Runtime database configuration supplied outside committed code. PostgreSQL
  version, network reachability, credentials, TLS, and available privileges still
  need to be established. Never request or commit secrets in the handoff.
- No FastAPI service until there is an actual API consumer.
- SQL-first analytics; optional Polars for exploration or file processing without
  duplicating SQL transformations unnecessarily.

## Data Modeling Principles

- A source listing is not a physical vehicle. Keep source listing identity
  separate from tentative vehicle/repost matches.
- Preserve timestamped observations and asking-price changes, including source
  provenance. Repeat imports must not create duplicate observations.
- Separate observation time from any source-provided publication/update time.
- Track collection-run success, failure, partial coverage, and scope so collection
  outages or incomplete pagination cannot masquerade as market changes.
- A listing disappearing means it is no longer observed, not that it sold.
- Last observed price is not sale price.
- Observed listing duration is not confirmed total time on market.
- Youngtimers may appreciate. Changes in asking prices or cohort composition are
  not automatically vehicle depreciation.
- Missing condition, mileage, or service-history information stays unknown.
- Preserve uncertain normalization rather than inventing precise variants.
- Do not mix synthetic fixtures into real market analytics.

### Initial Vehicle Distinctions

| Family | Important distinctions |
| --- | --- |
| Saab 900 | Classic vs second generation, body style, turbo vs naturally aspirated, engine, transmission |
| Mercedes 124-series | Sedan/estate/coupe/cabriolet, chassis designation where known, engine designation, transmission |
| Volvo 850 | Sedan/estate, engine variant, T-5/T-5R/R, transmission |

Retain these distinctions even when discovery uses broad family searches. Avoid
pooling materially different variants in price comparisons.

Exclude or explicitly classify parts, donor cars, restoration projects, and
wanted ads. Condition and originality may explain substantial price differences;
below-comparable asking prices are candidates for investigation, not guaranteed
bargains.

## Privacy and Retention

The analytics do not need seller names, phone numbers, exact addresses,
registration plates, or photos. Avoid collecting them by default.

Descriptions and full page responses may contain personal information. Prefer
minimized source records, and define filtering and retention before persisting
raw content. Do not assume retaining every page indefinitely is necessary for
reproducibility. Source identifiers and URLs should also be assessed for their
linkability and retention needs.

## Proposed First Milestone

1. Establish the initial data-input path and document the source-access decision.
2. Define an extensible vehicle catalog and a validated observation contract.
3. Implement schema setup/migrations for collection runs, source listings, and
   immutable timestamped observations.
4. Implement an idempotent import CLI with validation and transaction handling.
5. Add analytical SQL views for listing price history, comparable cohorts,
   observed duration, and collection coverage.
6. Package the job in Docker Compose, connecting to the existing PostgreSQL.
7. Add tests and operational documentation, including scheduling and replay.

### Acceptance Criteria

- A small input dataset can be ingested end to end into PostgreSQL.
- Replaying the same observations creates no duplicates or false price changes.
- A later observation preserves earlier values and exposes genuine price changes.
- Vehicle families, generations, and materially different variants remain
  distinguishable in analytics.
- Invalid or ambiguous records are handled explicitly, not silently coerced.
- Failed/partial collection cannot falsely mark listings as sold or disappeared.
- Tests can run without contacting Leboncoin or using private data.
- Compose configuration and documented commands support the homelab deployment.
- Documentation clearly distinguishes a replayable foundation from a functioning
  live daily collector; synthetic imports alone do not prove live acquisition.

## Deferred Work

- Deal scoring and alerts until there is sufficient comparable data.
- Dashboard and FastAPI service until the ingestion/modeling foundation works.
- Probabilistic repost/vehicle deduplication with uncertainty tracking.
- Additional free listing sources, subject to access investigation.
- Government enrichment datasets for market context, not assumed sale prices.
- More advanced longitudinal price analysis that accounts for variant and
  condition differences.

## Guidance for the Next Agent

Read this document and inspect the current files first. Keep the implementation
small, testable, and source-independent. Do not introduce paid services,
distributed infrastructure, or GitHub-specific deployment assumptions into this
Gitea/homelab project.