job-research/docs/job-agent-project-plan.md

# Job Agent Project Plan

## Objective

Build a local AI-assisted job discovery system that:

- understands your CV and work experience from a PDF,
- uses an additional markdown profile as structured context for more precise matching,
- understands the kinds of jobs you want,
- searches and extracts job listings from major French job sites,
- ranks listings against your profile,
- explains why each listing is or is not a good fit,
- lets you discuss results through OpenCode using your connected LLM providers such as OpenAI.

The system is not a generic job recommender. It is optimized for a narrow Target Role Profile:

- junior Data Engineer roles, around 2-3 years of experience,
- strong Python,
- good SQL, Terraform, GCP, and BigQuery,
- France-based CDI listings only,
- French and English listings both allowed, with a slight preference for French,
- no ranking preference between remote, hybrid, and onsite.

## v1 Scope

The first version should optimize for job discovery and ranking, not application automation.

Included in v1:

- PDF CV ingestion and structured profile extraction
- Markdown profile ingestion as a first-class structured input
- One saved default Search Preferences profile plus temporary chat overrides
- Read-only scraping / extraction from one public job board first
- Normalized storage of logical Job Listings and Listing Snapshots
- Matching and ranking pipeline with deterministic-first scoring
- Structured explanations with fit score, strengths, blockers, and summary
- Dismissed Listing memory so rejected listings stay hidden by default
- CLI-oriented workflow that fits OpenCode usage
- OpenCode discussion over stored results, with refresh as an explicit action

Explicitly out of scope for v1:

- Automatic application submission
- Cover letter generation as a core flow
- Full browser UI product
- CRM-grade candidate tracking system
- Support for every job site from day one
- Login-required scraping flows
- Salary-based ranking or salary filtering

## Resolved Product Decisions

These decisions were resolved during planning and should be treated as the default product behavior for v1.

### Search Target

- Primary matches: Data Engineer, Analytics Engineer, BI Engineer with strong SQL/Python pipelines, Junior Data Platform Engineer
- Stretch Opportunities: ML Engineer, Data Scientist, Backend Engineer, Senior Platform Engineer
- Seniority-mismatched roles should not rank in the main list; they belong in Stretch Opportunities

### Geography and Contract Rules

- Only France-based listings are in scope
- France-based means the listing explicitly targets France or a French city or region
- Company nationality alone is not enough
- Only CDI listings belong in the main ranking
- CDD, freelance, internship, and alternance listings are excluded in v1
- Remote, hybrid, and onsite are treated equally if the listing is in France

### Language and Freshness

- French and English listings are both in scope
- French gets a small ranking preference
- Listings from the last 14 days are preferred
- Listings from 15-30 days are penalized
- Listings older than 30 days are hidden unless the source shows a recent refresh

### Crawl and Refresh Rules

- v1 starts with a single source: Apec
- Only public listings are in scope in v1
- Refresh is explicit, not automatic during chat
- Refresh means: crawl current enabled sources using saved Search Preferences, then normalize, deduplicate, and re-rank automatically
- Chat-level Search Preference overrides can re-rank stored results immediately without triggering a new crawl
- Stored results remain discussable, but the system should warn if the latest crawl is older than 3 days

### Profile Merge Rules

- PDF CV is the source of truth for raw career history unless explicitly corrected
- Markdown profile is the source of truth for curated interpretation of strengths and goals
- Search Preferences from chat override current-session ranking priorities only
- The normalized Candidate Profile must be human-inspectable and editable in YAML or JSON

### Data Retention and Explanation Rules

- Store logical Job Listings separately from Listing Snapshots
- Keep raw HTML/content snapshots plus extraction metadata by default
- Keep screenshots only for failures or debug mode
- Dismissed Listings stay hidden by default but remain recoverable on request
- Explanations must show both strengths and blockers
- Deterministic ranking remains primary; the LLM may slightly adjust and explain, but should not dominate ordering

## Primary User Flow

1. You provide a PDF CV and a markdown profile.
2. The system extracts and merges both into a structured candidate profile.
3. You save default Search Preferences and optionally add temporary chat overrides.
4. You run an explicit refresh.
5. The system crawls Apec public listings using broad source-side filters for France and CDI.
6. The system stores raw artifacts, normalizes data, deduplicates into logical Job Listings and Listing Snapshots, and ranks results.
7. The system presents a ranked main list plus a separate Stretch Opportunities section.
8. An LLM adds bounded explanation and analysis on top of deterministic ranking.
9. You can ask follow-up questions in OpenCode, for example:
   - "Show me the best Data Engineer jobs from the latest crawl."
   - "Why was this Apec listing ranked low?"
   - "For this session, bias toward Analytics Engineer roles and re-rank."
   - "Show dismissed listings from the last refresh."

## Target Job Sources

v1 target:

- Apec

Post-v1 targets:

- Welcome to the Jungle
- Indeed France

Recommended rollout order:

1. Apec
2. Welcome to the Jungle
3. Indeed France

Reasoning:

- Start with one adapter and a stable data model.
- Some sites are more hostile to scraping or have more dynamic rendering.
- A phased rollout reduces debugging complexity and anti-bot risk.

## Product Requirements

### Core Requirements

- The system must extract structured profile data from a PDF CV.
- The system must ingest a markdown profile containing curated context that may be missing or ambiguous in the CV.
- The system must let you define Search Preferences beyond what is written in the CV.
- The system must support one saved default Search Preferences profile and temporary chat overrides.
- The system must query public France-based job listings, starting with Apec only in v1.
- The system must support Playwright-based extraction for JavaScript-heavy websites.
- The system must store raw artifacts, normalized Job Listings, and Listing Snapshots locally.
- The system must deduplicate reposted or updated pages into one logical Job Listing when confidence is high.
- The system must rank job listings with transparent reasoning.
- The system must keep deterministic ranking primary and use the LLM only as a bounded secondary layer.
- The system must expose results in a format that is easy to inspect from OpenCode.
- The system must support repeated refresh runs without flooding results with duplicate listings.
- The system must support Dismissed Listings that remain hidden by default but recoverable on request.
- The system must support re-ranking stored results after chat preference overrides without a new crawl.

### Quality Requirements

- Explainability: every score should have a short rationale.
- Traceability: keep source URL, scrape timestamp, and extraction metadata.
- Resilience: one failing site adapter must not break the full refresh.
- Local-first privacy: CV and job data stay local by default.
- Configurability: role filters and ranking criteria must be editable.
- Transparency: the structured Candidate Profile must be inspectable and editable.
- Freshness awareness: the system must warn when discussion relies on crawl data older than 3 days.

## Suggested Architecture

Use a modular Python application with clear boundaries.

### Components

1. Profile ingestion
2. Preference management
3. Site adapter for Apec
4. Crawl artifact and snapshot storage
5. Normalization and deduplication pipeline
6. Matching and ranking engine
7. LLM reasoning layer
8. CLI commands for refresh, inspect, re-rank, dismiss, and explain

### High-Level Flow

1. `ingest-cv` parses the PDF and produces a raw career-history contribution.
2. `ingest-profile` reads a markdown profile and merges curated intent and clarifications into the candidate profile.
3. `set-preferences` stores the saved default Search Preferences.
4. `refresh` crawls Apec using broad source-side filters for France and CDI.
5. The crawl persists raw HTML/content and extraction metadata as Listing Snapshots.
6. The pipeline normalizes and deduplicates snapshots into logical Job Listings.
7. `rank` computes deterministic scores first, then bounded LLM explanation and small adjustments.
8. OpenCode reads stored results, applies temporary preference overrides, and re-ranks without crawling again when requested.

## Recommended Technical Stack

### Core Language

- Python 3.13

Reasoning:

- The repo already starts in Python.
- Python is a good fit for scraping orchestration, PDF parsing, structured data pipelines, and LLM integration.

### Browser Automation / Extraction

- `playwright` for Python
- Playwright MCP for development-time debugging and manual inspection

Reasoning:

- Many job boards render content dynamically.
- Python Playwright should be the runtime dependency for the product itself.
- MCP is best treated as a development and inspection tool, not the only runtime path.

### Data Modeling and Validation

- `pydantic`

Use for:

- candidate profile schema,
- search preferences schema,
- normalized job listing schema,
- scoring explanation schema.

### Storage

- Start with SQLite
- Consider Postgres only if the project grows into a multi-user system

Store:

- candidate profile snapshots,
- preferences,
- raw scraped payloads,
- normalized job listings,
- score history,
- run logs.

### CLI and Developer Experience

- `typer` for CLI commands
- `rich` for readable terminal output

### LLM Integration

- OpenAI-compatible provider access through your OpenCode-connected providers
- Keep the LLM layer behind a small internal interface so providers can be swapped

Suggested uses for the LLM:

- reconciling CV text with markdown profile context,
- CV structuring when deterministic parsing is insufficient
- extracting nuanced skills and experience signals
- explaining ranking outcomes
- answering chat questions over stored job data

Avoid using the LLM for:

- the entire ranking logic,
- basic filtering,
- source-of-truth data storage.

### PDF and Text Extraction

Candidates:

- `pypdf`
- `pymupdf`
- OCR only if needed later

Recommendation:

- Start with `pymupdf` or `pypdf` for text extraction.
- Add OCR later only if the CV is image-based.

### HTML Parsing and Helpers

- `beautifulsoup4` as a fallback parser for static fragments
- `httpx` for non-browser HTTP calls where a full browser is unnecessary

## Proposed Local Data Model

### CandidateProfile

- name
- summary
- location
- languages
- profile_notes
- target_roles
- seniority
- skills
- industries
- preferred_contract_types
- years_of_experience
- experience_entries
- education_entries
- exclusion_rules

### Profile Sources

- `cv_pdf`
- `profile_md`
- merge strategy metadata

Recommendation:

- Treat the markdown profile as the higher-trust source for intent, preferences, and clarifications.
- Treat the PDF CV as the source for factual career history unless explicitly corrected in the markdown profile.
- Persist the merged result as a human-editable normalized profile in YAML or JSON.

### JobPreferences

- desired_titles
- required_keywords
- excluded_keywords
- target_locations
- remote_policy
- company_types
- board_filters
- language_preference
- include_stretch_opportunities

Notes:

- Salary is ignored in v1.
- Saved JobPreferences and temporary session overrides should be stored separately.

### JobListing

- source
- source_job_id
- url
- title
- company
- location
- remote_type
- contract_type
- salary_text
- description_text
- tags
- published_at
- refreshed_at
- scraped_at
- raw_payload_path
- freshness_status
- dismissal_state

### ListingSnapshot

- job_listing_id
- source
- source_job_id
- url
- captured_at
- html_snapshot_path
- extraction_metadata
- screenshot_path

### DismissedListing

- listing_id
- dismissed_at
- reason
- hidden_by_default

### JobScore

- listing_id
- deterministic_score
- llm_adjustment
- final_score
- positives
- concerns
- explanation
- stretch_reason

## Matching Strategy

Use a hybrid ranking approach.

### Deterministic Layer First

Compute a base score from explicit signals such as:

- title-family match
- skill overlap
- related-signal overlap
- seniority match
- location match
- contract type match
- language requirements
- freshness
- hard exclusions

Benefits:

- predictable behavior
- easier debugging
- lower cost than pure LLM ranking

Suggested deterministic rules for v1:

- Treat CDI and France-based scope as hard filters
- Treat non-target role families as heavy penalties or Stretch Opportunities
- Treat seniority mismatch as a strong penalty that usually pushes a listing into Stretch Opportunities
- Prefer explicit Data Engineer / Analytics Engineer / BI pipeline signals over generic Python backend signals
- Use a curated related-signal map for BigQuery, GCP, Dataflow, Airflow, dbt, warehousing, ETL/ELT, and SQL pipelines
- Penalize missing fields only when they block a critical decision, such as contract type or country
- Ignore salary entirely in v1
- Apply freshness weighting before the LLM layer

### LLM Layer Second

Use the LLM to:

- interpret fuzzy fit,
- detect transferable experience,
- summarize tradeoffs,
- explain why a role is still interesting even if some keywords are missing.

Keep the LLM bounded:

- pass structured profile + structured listing,
- ask for JSON output,
- cap influence over final score.
- require positives and blockers in every explanation.

## Scraping Strategy

Start with one adapter only: `ApecAdapter`.

### Adapter Responsibilities

- navigate search results
- handle pagination
- capture job detail pages
- extract structured fields
- respect rate limits and retries
- emit raw + normalized records

### Recommended Adapter Pattern

- `BaseJobBoardAdapter`
- `ApecAdapter`
- `WelcomeToTheJungleAdapter`
- `IndeedAdapter`

Each adapter should define:

- search URL generation
- page navigation flow
- field extraction selectors
- fallback extraction logic
- deduplication key strategy

### Important Constraints

- Review site terms of service before scraping.
- Expect anti-bot measures, dynamic DOM changes, and inconsistent markup.
- Prefer low-volume, respectful scraping with caching.
- Treat login-required flows as a later phase unless strictly necessary.

### v1 Crawl Policy

- Use broad source-side filters for France and CDI when the source supports them
- Avoid over-filtering by title at crawl time; keep recall reasonably broad and rank afterward
- Crawl public listings only
- Persist raw HTML/content snapshots and extraction metadata for every Listing Snapshot
- Persist screenshots only for failures or debug mode

## OpenCode Integration Plan

The simplest useful approach is not to build a separate chat agent first.

Instead:

- keep the job system as a local Python app,
- expose CLI commands that produce structured JSON and readable markdown,
- use OpenCode as the conversational layer on top.

OpenCode should behave as a tool-using assistant over local data and CLI commands, not as a separate long-running orchestration agent.

Example commands:

- `python -m job_research ingest-cv data/cv.pdf`
- `python -m job_research ingest-profile data/profile.md`
- `python -m job_research set-preferences config/preferences.yaml`
- `python -m job_research refresh --source apec`
- `python -m job_research rank --top 50`
- `python -m job_research rerank --session-overrides overrides.yaml`
- `python -m job_research explain --job-id <id>`
- `python -m job_research dismiss --job-id <id>`
- `python -m job_research show-dismissed`
- `python -m job_research export --format markdown`

This gives OpenCode something concrete to call and inspect while still letting you discuss opportunities in natural language.

## Suggested Folder Structure

```text
docs/
  job-agent-project-plan.md
src/
  job_research/
    cli.py
    config.py
    models.py
    storage.py
    freshness.py
    profile/
      ingest.py
      extract.py
    boards/
      base.py
      apec.py
      welcome_to_the_jungle.py
      indeed.py
    listings/
      normalize.py
      dedupe.py
    ranking/
      rules.py
      llm.py
    reporting/
      explain.py
      export.py
data/
  raw/
  snapshots/
  normalized/
  runs/
  profiles/
  exports/
  preferences/
tests/
```

## Executable Roadmap

### Phase 1: Foundations and Local State

Goal:

- establish the package shape, persistence model, and command surface for a single-user local workflow

- Create Python package structure
- Add configuration and CLI skeleton
- Define Pydantic models
- Set up SQLite storage
- Add logging and run tracking

Deliverables:

- package scaffold under `src/job_research/`
- initial SQLite schema for profiles, preferences, listings, snapshots, scores, and dismissals
- CLI entrypoints for `ingest-cv`, `ingest-profile`, `set-preferences`, `refresh`, `rank`, `explain`, and `dismiss`

Exit criteria:

- CLI runs locally
- models and storage are stable enough for feature work
- a refresh run can create a tracked run record even before scraping is implemented

### Phase 2: Profile Ingestion

Goal:

- create a trustworthy merged Candidate Profile from PDF, markdown, and manual correction

- Parse PDF CV
- Parse markdown profile
- Extract structured profile fields
- Merge CV facts with markdown clarifications
- Add manual correction path via JSON or markdown
- Save profile snapshot locally

Deliverables:

- PDF text extraction module
- markdown profile parser
- normalized Candidate Profile serializer in YAML or JSON
- merge policy implementation reflecting the resolved authority rules

Exit criteria:

- You can ingest a real CV plus markdown profile and inspect a usable merged profile
- you can manually correct the normalized profile and re-use it for ranking

### Phase 3: First Job Source Adapter

Goal:

- build one reliable end-to-end public crawler on Apec

- Implement one source adapter, preferably Apec
- Support search, listing extraction, and normalization
- Store raw and normalized results

Deliverables:

- Playwright-based Apec adapter
- broad source-side filters for France and CDI when possible
- raw HTML/content snapshot persistence
- extraction metadata and failure logging

Exit criteria:

- One command fetches real listings and stores them locally
- repeated refresh runs do not create uncontrolled duplicates

### Phase 4: Normalization, Deduplication, and Freshness

Goal:

- turn raw crawl output into stable logical Job Listings that remain useful across repeated refreshes

- normalize source payloads into the shared JobListing schema
- deduplicate reposted or updated pages into one logical listing when confidence is high
- classify freshness from published and refreshed dates
- preserve Listing Snapshots under the logical listing

Deliverables:

- normalization pipeline
- deduplication heuristics for company/title/location/time-window matching
- freshness classification module

Exit criteria:

- the same opportunity can appear across multiple refreshes without flooding the result set
- each logical listing can be inspected alongside its snapshots

### Phase 5: Ranking Engine

Goal:

- produce trustworthy main results and Stretch Opportunities from deterministic rules first

- Implement deterministic scoring
- Add top-match reporting
- Add explainability output

Deliverables:

- title-family and seniority scoring rules
- curated related-signal map for data stack technologies
- hard filters for CDI and France-based scope
- separate Stretch Opportunities classification

Exit criteria:

- The system can rank jobs against your profile with understandable reasons
- a generic backend Python role is outranked by a real data-platform role with weaker Python emphasis

### Phase 6: LLM-Assisted Explanations and Re-ranking

Goal:

- add bounded LLM reasoning without giving up deterministic control

- Add provider abstraction
- Add structured LLM prompts for fit analysis
- Generate concise explanations and follow-up insights

Deliverables:

- provider interface compatible with OpenCode-connected providers
- JSON-shaped prompt/response contract
- `rerank` flow for temporary Search Preferences overrides over stored listings

Exit criteria:

- Explanations are better than raw score tables and remain grounded in structured data
- chat overrides can re-rank stored results without a new crawl

### Phase 7: Dismissal, Export, and OpenCode Workflow Hardening

Goal:

- make the tool comfortable for repeated day-to-day use inside OpenCode

- Add markdown / JSON exports optimized for chat workflows
- Add commands for summaries, comparisons, and filters
- Add Dismissed Listing behavior
- Add stale-data warning behavior for discussion over old crawls

Deliverables:

- dismissed listing persistence and recovery flow
- markdown export shaped for OpenCode review
- stale crawl warning logic for data older than 3 days

Exit criteria:

- OpenCode becomes the comfortable day-to-day interface for analyzing opportunities
- dismissed listings no longer repeatedly waste attention

### Phase 8: Multi-Source Expansion

Goal:

- expand source coverage after the full loop is proven on Apec

- Add Welcome to the Jungle adapter
- Add Indeed adapter
- Improve deduplication across boards

Deliverables:

- source-specific adapters for additional boards
- cross-source deduplication heuristics

Exit criteria:

- A full refresh can aggregate from multiple supported sources

## Key Risks

### Legal / Platform Risk

- Some sites may restrict scraping or aggressively block automation.
- Mitigation: start small, inspect ToS, use throttling, and favor stable flows.

### Extraction Fragility

- DOM selectors may break often.
- Mitigation: isolate selectors per adapter and keep raw payloads for debugging.

### LLM Hallucination

- The model may overstate job fit.
- Mitigation: deterministic scoring stays primary; LLM output is constrained and explainable.

### CV Parsing Quality

- PDF extraction may be messy.
- Mitigation: keep a manual review/edit path for the structured profile.

## Immediate Build Order

Build the first thin vertical slice in this order:

1. create the package, storage, and CLI surface,
2. ingest the PDF CV and markdown profile,
3. persist a corrected normalized Candidate Profile,
4. implement Apec crawl and artifact storage,
5. normalize, deduplicate, and classify freshness,
6. rank into main results and Stretch Opportunities,
7. add structured explanations,
8. expose rerank, dismiss, and export flows for OpenCode.

## Recommended Next Step

Start with Phase 1 and Phase 2 only:

1. scaffold the Python package and local SQLite state,
2. implement PDF CV + markdown profile ingestion,
3. persist a normalized editable Candidate Profile,
4. stop there and validate the profile before touching scraping.

That is the safest first checkpoint because a bad Candidate Profile will poison every later ranking decision.