car-tracker/PROJECT_PLAN.md

Youngtimer Market Observatory: Project Handoff

Implementation Update (2026-09-08)

The user subsequently selected live Leboncoin collection and explicitly requested titles, descriptions, maximum vehicle attributes, and source-provided precise locations for a future map dashboard, including visits to individual ads. This supersedes the earlier outstanding input decision and the exclusions of free text and precise location below. Seller/contact objects and photos remain excluded; free text may still contain personal information and must remain private until reviewed/redacted for downstream publication. Source coordinates may be approximate.

The collector now lives in src/car_tracker/, storing separate search/detail observations in versioned compressed JSON Lines with a coverage manifest. See README.md for current commands, capture policy, retention, and verified live results. The original handoff below remains planning context, not a description of the current implementation.

The next milestone is implemented: SQL migrations, conservative silver cleaning, atomic/idempotent PostgreSQL import, and views for observations, price history, coverage, and quality issues. Development/CLI execution is on the user's laptop; the intended database is the existing remote homelab PostgreSQL instance. Local PostgreSQL containers are disposable tests only. Homelab provisioning and remote connection setup remain user actions documented in docs/POSTGRESQL.md; no homelab database has been accessed or modified during implementation.

The manual homelab job is now packaged in Dockerfile and ops/app/compose.yaml. car-tracker-pipeline performs a database preflight, bounded collection, and an idempotent import of the exact returned manifest, retaining partial coverage and separate operational outcomes. There is no scheduler or automatic deployment. See docs/DEPLOYMENT.md; container tests use isolated PostgreSQL 17 and synthetic browser responses. Actual live access from the homelab still needs verification.

Purpose

Build a French used-car asking-price observatory focused initially on youngtimers. The goals are market analytics and a data engineering portfolio, not primarily a car-buying assistant. The user is an experienced data engineer.

Track listing observations over time, normalize vehicle variants, preserve price history, and eventually identify listings priced below suitable comparables. Asking prices are not transaction prices. No confirmed sale-price source has been identified.

Confirmed Requirements

• Initial families: Saab 900, Mercedes-Benz 124-series, and Volvo 850.
• Include broad variant coverage: both Saab 900 generations; Mercedes sedans, estates, coupes, and cabriolets; Volvo 850 body styles and engine variants.
• Keep the vehicle catalog extensible to many more models later.
• Geographic scope: all France.
• Target collection cadence: daily, subject to source access feasibility.
• Preferred first platform: Leboncoin.
• No paid data sources.
• Python and SQL; FastAPI if an API becomes necessary, but no API needed initially.
• Polars is optional, not a requirement. Use it only where it adds value over SQL.
• No distributed infrastructure.
• Docker Compose deployment on the user's homelab.
• No deadline or fixed time budget.
• First milestone: a working end-to-end pipeline, not just an exploratory notebook.

Environment

• Homelab: Ryzen 5 7500F, 32 GB RAM.
• PostgreSQL already runs on a Debian VM; reuse it rather than require a second production database.
• Repository hosted on the user's own Gitea instance.
• Current workspace: /home/tonio/Workspace/perso/car-tracker.
• At inspection, the repository contained .gitignore, .python-version, main.py, pyproject.toml, and an empty README.md.
• pyproject.toml specifies Python >=3.13, project name car-tracker, version 0.1.0, and no dependencies.
• The worktree was clean before this handoff document was added.
• No application implementation was changed during the planning discussion.
• The user previously wrote a Leboncoin scraper, but no longer has its code or sample responses.

Reinspect the repository before implementation; do not assume it is unchanged.

Source Access

The user prefers Leboncoin and has experience scraping it. They stated that publicly available data can be collected under French law, then argued that Leboncoin's allowance for ChatGPT bots permits this project to scrape it.

Outstanding Decision

The previous agent offered two paths:

1. Build the source-independent foundation using clearly synthetic snapshots while source access is investigated.
2. Investigate alternative free sources before implementing the foundation.

The recommendation was path 1, but the user has not selected either path. Their latest request was only to write this handoff document. Do not treat synthetic data, an alternative platform, or live Leboncoin collection as an agreed choice.

Proposed Architecture

These are recommendations discussed with the user, not a fully approved technical design or an implemented system.

Collection and Processing

• A small Python CLI with source-specific collection/parsing separated from the normalized import path.
• Prefer a direct HTTP client where an accessible source exposes the required data without browser rendering.
• Use Playwright when rendering or interaction is genuinely required. The user suggested it but is open to another suitable approach.
• Do not choose a browser framework before inspecting the source. Playwright does not itself resolve access restrictions or guarantee anti-bot compatibility.
• No need initially for Scrapy, distributed queues, or a large orchestrator.
• Configurable model families and source search definitions rather than hardcoded logic for three models.

Persistence and Scheduling

• Existing PostgreSQL for collection runs, listings, observations, normalized attributes, and analytical SQL views.
• Local mounted storage for minimized source snapshots where retention is appropriate, enabling replay and parser debugging.
• Snapshot storage outside Git, with a documented retention policy.
• A Compose job triggered daily by a host systemd timer is the proposed simplest deployment. Compose is confirmed; the timer mechanism is not yet finalized.
• Runtime database configuration supplied outside committed code. PostgreSQL version, network reachability, credentials, TLS, and available privileges still need to be established. Never request or commit secrets in the handoff.
• No FastAPI service until there is an actual API consumer.
• SQL-first analytics; optional Polars for exploration or file processing without duplicating SQL transformations unnecessarily.

Data Modeling Principles

• A source listing is not a physical vehicle. Keep source listing identity separate from tentative vehicle/repost matches.
• Preserve timestamped observations and asking-price changes, including source provenance. Repeat imports must not create duplicate observations.
• Separate observation time from any source-provided publication/update time.
• Track collection-run success, failure, partial coverage, and scope so collection outages or incomplete pagination cannot masquerade as market changes.
• A listing disappearing means it is no longer observed, not that it sold.
• Last observed price is not sale price.
• Observed listing duration is not confirmed total time on market.
• Youngtimers may appreciate. Changes in asking prices or cohort composition are not automatically vehicle depreciation.
• Missing condition, mileage, or service-history information stays unknown.
• Preserve uncertain normalization rather than inventing precise variants.
• Do not mix synthetic fixtures into real market analytics.

Initial Vehicle Distinctions

Family	Important distinctions
Saab 900	Classic vs second generation, body style, turbo vs naturally aspirated, engine, transmission
Mercedes 124-series	Sedan/estate/coupe/cabriolet, chassis designation where known, engine designation, transmission
Volvo 850	Sedan/estate, engine variant, T-5/T-5R/R, transmission

Retain these distinctions even when discovery uses broad family searches. Avoid pooling materially different variants in price comparisons.

Exclude or explicitly classify parts, donor cars, restoration projects, and wanted ads. Condition and originality may explain substantial price differences; below-comparable asking prices are candidates for investigation, not guaranteed bargains.

Privacy and Retention

The analytics do not need seller names, phone numbers, exact addresses, registration plates, or photos. Avoid collecting them by default.

Descriptions and full page responses may contain personal information. Prefer minimized source records, and define filtering and retention before persisting raw content. Do not assume retaining every page indefinitely is necessary for reproducibility. Source identifiers and URLs should also be assessed for their linkability and retention needs.

Proposed First Milestone

1. Establish the initial data-input path and document the source-access decision.
2. Define an extensible vehicle catalog and a validated observation contract.
3. Implement schema setup/migrations for collection runs, source listings, and immutable timestamped observations.
4. Implement an idempotent import CLI with validation and transaction handling.
5. Add analytical SQL views for listing price history, comparable cohorts, observed duration, and collection coverage.
6. Package the job in Docker Compose, connecting to the existing PostgreSQL.
7. Add tests and operational documentation, including scheduling and replay.

Acceptance Criteria

• A small input dataset can be ingested end to end into PostgreSQL.
• Replaying the same observations creates no duplicates or false price changes.
• A later observation preserves earlier values and exposes genuine price changes.
• Vehicle families, generations, and materially different variants remain distinguishable in analytics.
• Invalid or ambiguous records are handled explicitly, not silently coerced.
• Failed/partial collection cannot falsely mark listings as sold or disappeared.
• Tests can run without contacting Leboncoin or using private data.
• Compose configuration and documented commands support the homelab deployment.
• Documentation clearly distinguishes a replayable foundation from a functioning live daily collector; synthetic imports alone do not prove live acquisition.

Deferred Work

• Deal scoring and alerts until there is sufficient comparable data.
• Dashboard and FastAPI service until the ingestion/modeling foundation works.
• Probabilistic repost/vehicle deduplication with uncertainty tracking.
• Additional free listing sources, subject to access investigation.
• Government enrichment datasets for market context, not assumed sale prices.
• More advanced longitudinal price analysis that accounts for variant and condition differences.

Guidance for the Next Agent

Read this document and inspect the current files first. Keep the implementation small, testable, and source-independent. Do not introduce paid services, distributed infrastructure, or GitHub-specific deployment assumptions into this Gitea/homelab project.