Antoine e171228626 docs: simplify candidate profile ingestion plan

2026-05-28 12:23:23 +02:00

25 KiB

Raw Blame History

Candidate Profile Ingestion Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build one CLI command that reads a PDF CV and a light-template markdown profile, then writes a single editable candidate-profile.yaml with explicit warnings when facts conflict or extraction is uncertain.

Architecture: The implementation is a simple normalizer, not a workflow engine. Deterministic CV extraction and markdown parsing feed a profile merger that writes one canonical YAML file; warnings are embedded in that file instead of blocking output or requiring separate override files.

Tech Stack: Python 3.13, Typer, Pydantic v2, PyYAML, pypdf, pytest

File Map

Create: src/job_research/__init__.py — package marker
Create: src/job_research/cli.py — one-command Typer CLI
Create: src/job_research/models.py — Pydantic models for canonical profile output and warnings
Create: src/job_research/storage.py — YAML persistence helper
Create: src/job_research/profile/__init__.py — profile package marker
Create: src/job_research/profile/cv_extractor.py — deterministic PDF extraction and basic CV parsing
Create: src/job_research/profile/profile_parser.py — light markdown template parser
Create: src/job_research/profile/merge.py — canonical profile assembly and warning generation
Create: tests/test_cli.py — CLI integration tests
Create: tests/profile/test_cv_extractor.py — CV parsing tests
Create: tests/profile/test_profile_parser.py — markdown parser tests
Create: tests/profile/test_merge.py — profile merge and warning tests
Create: tests/test_storage.py — YAML output tests
Create: docs/profile-template.md — light required markdown template for the user
Modify: pyproject.toml — package metadata and dependencies

Task 1: Package Skeleton and Single Command Surface

Files:

Modify: pyproject.toml
Create: src/job_research/__init__.py
Create: src/job_research/cli.py
Create: tests/test_cli.py
Step 1: Write the failing CLI help test

# tests/test_cli.py
from typer.testing import CliRunner

from job_research.cli import app


def test_cli_help_exposes_single_build_command() -> None:
    result = CliRunner().invoke(app, ["--help"])

    assert result.exit_code == 0
    assert "build-profile" in result.stdout

Step 2: Run the help test to verify it fails

Run: uv run pytest tests/test_cli.py::test_cli_help_exposes_single_build_command -v Expected: FAIL with ModuleNotFoundError: No module named 'job_research'

Step 3: Add dependencies and the minimal CLI shell

# pyproject.toml
[project]
name = "job-research"
version = "0.1.0"
description = "Local job discovery tooling"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
  "pydantic>=2.7,<3",
  "pypdf>=5.0,<6",
  "pyyaml>=6.0,<7",
  "typer>=0.12,<1",
]

[dependency-groups]
dev = [
  "pytest>=8.2,<9",
]

[project.scripts]
job-research = "job_research.cli:main"

# src/job_research/__init__.py
__all__ = ["__version__"]

__version__ = "0.1.0"

# src/job_research/cli.py
import typer

app = typer.Typer(help="Build one canonical candidate profile YAML")


@app.command("build-profile")
def build_profile() -> None:
    """Build candidate-profile.yaml from CV and markdown profile."""


def main() -> None:
    app()


if __name__ == "__main__":
    main()

Step 4: Run the help test to verify it passes

Run: uv run pytest tests/test_cli.py::test_cli_help_exposes_single_build_command -v Expected: PASS

Step 5: Commit the skeleton

git add pyproject.toml src/job_research/__init__.py src/job_research/cli.py tests/test_cli.py
git commit -m "feat: add candidate profile CLI skeleton"

Task 2: Canonical Output Model and YAML Persistence

Files:

Create: src/job_research/models.py
Create: src/job_research/storage.py
Create: tests/test_storage.py
Step 1: Write the failing canonical YAML test

# tests/test_storage.py
from job_research.models import CandidateProfileOutput, ExperienceEntry, WarningItem
from job_research.storage import save_candidate_profile_yaml, load_yaml


def test_save_candidate_profile_yaml_round_trips_readable_output(tmp_path) -> None:
    profile = CandidateProfileOutput(
        name="Tonio",
        summary="Junior data engineer focused on Python and GCP.",
        target_roles=["Data Engineer"],
        strengths=["Python", "SQL"],
        skills_to_emphasize=["GCP", "BigQuery"],
        constraints=["CDI only", "France only"],
        notes=["Slight preference for French listings."],
        location="France",
        languages=["French", "English"],
        skills=["Python", "SQL", "Terraform", "GCP", "BigQuery"],
        experience_entries=[ExperienceEntry(company="A", title="Data Engineer")],
        education_entries=[],
        warnings=[WarningItem(field="years_of_experience", message="CV and profile disagree.")],
    )

    out = tmp_path / "candidate-profile.yaml"
    save_candidate_profile_yaml(out, profile)

    payload = load_yaml(out)

    assert payload["name"] == "Tonio"
    assert payload["constraints"] == ["CDI only", "France only"]
    assert payload["warnings"][0]["field"] == "years_of_experience"

Step 2: Run the storage test to verify it fails

Run: uv run pytest tests/test_storage.py::test_save_candidate_profile_yaml_round_trips_readable_output -v Expected: FAIL with ImportError for missing models or storage helpers

Step 3: Implement the canonical output model and YAML helper

# src/job_research/models.py
from pydantic import BaseModel, Field


class ExperienceEntry(BaseModel):
    company: str
    title: str
    start: str | None = None
    end: str | None = None
    highlights: list[str] = Field(default_factory=list)


class EducationEntry(BaseModel):
    institution: str
    credential: str
    start: str | None = None
    end: str | None = None


class WarningItem(BaseModel):
    field: str
    message: str


class CandidateProfileOutput(BaseModel):
    name: str | None = None
    summary: str | None = None
    target_roles: list[str] = Field(default_factory=list)
    strengths: list[str] = Field(default_factory=list)
    skills_to_emphasize: list[str] = Field(default_factory=list)
    constraints: list[str] = Field(default_factory=list)
    notes: list[str] = Field(default_factory=list)
    location: str | None = None
    languages: list[str] = Field(default_factory=list)
    skills: list[str] = Field(default_factory=list)
    experience_entries: list[ExperienceEntry] = Field(default_factory=list)
    education_entries: list[EducationEntry] = Field(default_factory=list)
    warnings: list[WarningItem] = Field(default_factory=list)

# src/job_research/storage.py
from __future__ import annotations

from pathlib import Path
from typing import Any

import yaml

from job_research.models import CandidateProfileOutput


def save_candidate_profile_yaml(path: Path, profile: CandidateProfileOutput) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    payload = profile.model_dump(mode="json")
    path.write_text(yaml.safe_dump(payload, sort_keys=False, allow_unicode=True), encoding="utf-8")


def load_yaml(path: Path) -> dict[str, Any]:
    return yaml.safe_load(path.read_text(encoding="utf-8")) or {}

Step 4: Run the storage test to verify it passes

Run: uv run pytest tests/test_storage.py::test_save_candidate_profile_yaml_round_trips_readable_output -v Expected: PASS

Step 5: Commit the models and storage helper

git add src/job_research/models.py src/job_research/storage.py tests/test_storage.py
git commit -m "feat: add canonical candidate profile output model"

Task 3: Light Markdown Profile Template and Parser

Files:

Create: src/job_research/profile/__init__.py
Create: src/job_research/profile/profile_parser.py
Create: tests/profile/test_profile_parser.py
Create: docs/profile-template.md
Step 1: Write the failing markdown parser test

# tests/profile/test_profile_parser.py
from textwrap import dedent

from job_research.profile.profile_parser import parse_profile_markdown


def test_parse_profile_markdown_reads_light_required_template() -> None:
    markdown = dedent(
        """
        # Candidate Profile

        ## Summary
        Junior data engineer focused on Python and GCP.

        ## Target Roles
        - Data Engineer
        - Analytics Engineer

        ## Strengths
        - Python
        - SQL

        ## Skills To Emphasize
        - BigQuery
        - Terraform

        ## Constraints
        - CDI only
        - France only

        ## Notes
        - Slight preference for French listings.
        """
    ).strip()

    profile = parse_profile_markdown(markdown)

    assert profile.summary == "Junior data engineer focused on Python and GCP."
    assert profile.target_roles == ["Data Engineer", "Analytics Engineer"]
    assert profile.strengths == ["Python", "SQL"]
    assert profile.skills_to_emphasize == ["BigQuery", "Terraform"]
    assert profile.constraints == ["CDI only", "France only"]
    assert profile.notes == ["Slight preference for French listings."]

Step 2: Run the parser test to verify it fails

Run: uv run pytest tests/profile/test_profile_parser.py::test_parse_profile_markdown_reads_light_required_template -v Expected: FAIL with ImportError for missing parser

Step 3: Implement the authored profile parser and template

# src/job_research/profile/__init__.py
__all__ = ["profile_parser"]

# src/job_research/profile/profile_parser.py
from __future__ import annotations

from collections import defaultdict
from dataclasses import dataclass, field


@dataclass
class AuthoredProfile:
    summary: str | None = None
    target_roles: list[str] = field(default_factory=list)
    strengths: list[str] = field(default_factory=list)
    skills_to_emphasize: list[str] = field(default_factory=list)
    constraints: list[str] = field(default_factory=list)
    notes: list[str] = field(default_factory=list)


REQUIRED_SECTIONS = {
    "summary",
    "target roles",
    "strengths",
    "skills to emphasize",
    "constraints",
    "notes",
}


def parse_profile_markdown(markdown: str) -> AuthoredProfile:
    sections: dict[str, list[str]] = defaultdict(list)
    current_section: str | None = None

    for raw_line in markdown.splitlines():
        line = raw_line.strip()
        if line.startswith("## "):
            current_section = line[3:].strip().lower()
            continue
        if not line or current_section is None:
            continue
        sections[current_section].append(line)

    missing = REQUIRED_SECTIONS - set(sections)
    if missing:
        missing_text = ", ".join(sorted(missing))
        raise ValueError(f"Missing required markdown sections: {missing_text}")

    return AuthoredProfile(
        summary=" ".join(sections["summary"]),
        target_roles=[line[2:].strip() for line in sections["target roles"] if line.startswith("- ")],
        strengths=[line[2:].strip() for line in sections["strengths"] if line.startswith("- ")],
        skills_to_emphasize=[line[2:].strip() for line in sections["skills to emphasize"] if line.startswith("- ")],
        constraints=[line[2:].strip() for line in sections["constraints"] if line.startswith("- ")],
        notes=[line[2:].strip() if line.startswith("- ") else line for line in sections["notes"]],
    )

# docs/profile-template.md
# Candidate Profile

## Summary
One short paragraph describing your target profile.

## Target Roles
- Data Engineer

## Strengths
- Python
- SQL

## Skills To Emphasize
- GCP
- BigQuery

## Constraints
- CDI only
- France only

## Notes
- Anything the CV parser might miss but future ranking should understand.

Step 4: Run the parser test to verify it passes

Run: uv run pytest tests/profile/test_profile_parser.py::test_parse_profile_markdown_reads_light_required_template -v Expected: PASS

Step 5: Commit the markdown parser

git add src/job_research/profile/__init__.py src/job_research/profile/profile_parser.py tests/profile/test_profile_parser.py docs/profile-template.md
git commit -m "feat: add light markdown candidate profile parser"

Task 4: Deterministic CV Extraction

Files:

Create: src/job_research/profile/cv_extractor.py
Create: tests/profile/test_cv_extractor.py
Step 1: Write the failing CV extraction test

# tests/profile/test_cv_extractor.py
from job_research.profile.cv_extractor import extract_cv_signals


def test_extract_cv_signals_reads_basic_fields_from_text() -> None:
    text = """
    Tonio
    Location: France
    Languages: French, English
    Skills: Python, SQL, Terraform, GCP, BigQuery
    Data Engineer at Company A
    Analytics Engineer at Company B
    """

    extracted = extract_cv_signals(text)

    assert extracted["name"] == "Tonio"
    assert extracted["location"] == "France"
    assert extracted["languages"] == ["French", "English"]
    assert extracted["skills"] == ["Python", "SQL", "Terraform", "GCP", "BigQuery"]
    assert extracted["experience_entries"][0]["title"] == "Data Engineer"
    assert len(extracted["experience_entries"]) == 2

Step 2: Run the CV extraction test to verify it fails

Run: uv run pytest tests/profile/test_cv_extractor.py::test_extract_cv_signals_reads_basic_fields_from_text -v Expected: FAIL with ImportError for missing CV extractor

Step 3: Implement PDF text extraction and basic signal parsing

# src/job_research/profile/cv_extractor.py
from __future__ import annotations

from pathlib import Path

from pypdf import PdfReader


def extract_pdf_text(path: Path) -> str:
    reader = PdfReader(str(path))
    return "\n".join(page.extract_text() or "" for page in reader.pages)


def extract_cv_signals(text: str) -> dict:
    lines = [line.strip() for line in text.splitlines() if line.strip()]
    profile = {
        "name": lines[0] if lines else None,
        "location": None,
        "languages": [],
        "skills": [],
        "experience_entries": [],
        "education_entries": [],
    }

    for line in lines:
        lower = line.lower()
        if lower.startswith("location:"):
            profile["location"] = line.split(":", 1)[1].strip()
        elif lower.startswith("languages:"):
            profile["languages"] = [part.strip() for part in line.split(":", 1)[1].split(",") if part.strip()]
        elif lower.startswith("skills:"):
            profile["skills"] = [part.strip() for part in line.split(":", 1)[1].split(",") if part.strip()]
        elif " at " in line:
            title, company = line.split(" at ", 1)
            profile["experience_entries"].append({"title": title.strip(), "company": company.strip()})

    return profile

Step 4: Run the CV extraction test to verify it passes

Run: uv run pytest tests/profile/test_cv_extractor.py::test_extract_cv_signals_reads_basic_fields_from_text -v Expected: PASS

Step 5: Commit the CV extractor

git add src/job_research/profile/cv_extractor.py tests/profile/test_cv_extractor.py
git commit -m "feat: add deterministic CV extraction"

Task 5: Canonical Profile Merge and Warning Generation

Files:

Create: src/job_research/profile/merge.py
Create: tests/profile/test_merge.py
Step 1: Write the failing merge test for warnings

# tests/profile/test_merge.py
from job_research.profile.merge import build_candidate_profile_output
from job_research.profile.profile_parser import AuthoredProfile


def test_build_candidate_profile_output_writes_warning_when_facts_conflict() -> None:
    cv_signals = {
        "name": "Tonio",
        "location": "France",
        "languages": ["French", "English"],
        "skills": ["Python", "SQL"],
        "experience_entries": [{"title": "Data Engineer", "company": "A"}],
        "education_entries": [],
        "years_of_experience": 2,
    }
    authored = AuthoredProfile(
        summary="Junior data engineer focused on GCP.",
        target_roles=["Data Engineer"],
        strengths=["Python"],
        skills_to_emphasize=["BigQuery", "GCP"],
        constraints=["CDI only"],
        notes=["Years of experience feels closer to 3."],
    )

    output = build_candidate_profile_output(cv_signals, authored)

    assert output.summary == "Junior data engineer focused on GCP."
    assert output.constraints == ["CDI only"]
    assert output.warnings[0].field == "years_of_experience"

Step 2: Run the merge test to verify it fails

Run: uv run pytest tests/profile/test_merge.py::test_build_candidate_profile_output_writes_warning_when_facts_conflict -v Expected: FAIL with ImportError for missing merge module

Step 3: Implement the merger and warnings

# src/job_research/profile/merge.py
from __future__ import annotations

from job_research.models import CandidateProfileOutput, ExperienceEntry, EducationEntry, WarningItem
from job_research.profile.profile_parser import AuthoredProfile


def build_candidate_profile_output(cv_signals: dict, authored: AuthoredProfile) -> CandidateProfileOutput:
    warnings: list[WarningItem] = []

    if "years_of_experience" in cv_signals and any("experience" in note.lower() for note in authored.notes):
        warnings.append(
            WarningItem(
                field="years_of_experience",
                message="CV-derived experience may differ from markdown interpretation. Review manually.",
            )
        )

    merged_skills: list[str] = []
    for skill in [*cv_signals.get("skills", []), *authored.strengths, *authored.skills_to_emphasize]:
        if skill not in merged_skills:
            merged_skills.append(skill)

    return CandidateProfileOutput(
        name=cv_signals.get("name"),
        summary=authored.summary,
        target_roles=authored.target_roles,
        strengths=authored.strengths,
        skills_to_emphasize=authored.skills_to_emphasize,
        constraints=authored.constraints,
        notes=authored.notes,
        location=cv_signals.get("location"),
        languages=cv_signals.get("languages", []),
        skills=merged_skills,
        experience_entries=[ExperienceEntry.model_validate(item) for item in cv_signals.get("experience_entries", [])],
        education_entries=[EducationEntry.model_validate(item) for item in cv_signals.get("education_entries", [])],
        warnings=warnings,
    )

Step 4: Run the merge test to verify it passes

Run: uv run pytest tests/profile/test_merge.py::test_build_candidate_profile_output_writes_warning_when_facts_conflict -v Expected: PASS

Step 5: Commit the merger

git add src/job_research/profile/merge.py tests/profile/test_merge.py
git commit -m "feat: add canonical profile merger with warnings"

Task 6: End-to-End build-profile Command

Files:

Modify: src/job_research/cli.py
Modify: tests/test_cli.py
Step 1: Write the failing end-to-end CLI tests

# tests/test_cli.py
from pathlib import Path

from typer.testing import CliRunner

from job_research.cli import app


def test_build_profile_writes_canonical_yaml(monkeypatch, tmp_path: Path) -> None:
    cv_path = tmp_path / "cv.txt"
    profile_path = tmp_path / "profile.md"
    out_path = tmp_path / "candidate-profile.yaml"
    cv_path.write_text(
        "Tonio\nLocation: France\nLanguages: French, English\nSkills: Python, SQL\nData Engineer at Company A\n",
        encoding="utf-8",
    )
    profile_path.write_text(
        "# Candidate Profile\n\n## Summary\nJunior data engineer.\n\n## Target Roles\n- Data Engineer\n\n## Strengths\n- Python\n\n## Skills To Emphasize\n- BigQuery\n\n## Constraints\n- CDI only\n\n## Notes\n- Slight preference for French listings.\n",
        encoding="utf-8",
    )

    result = CliRunner().invoke(
        app,
        ["build-profile", "--cv", str(cv_path), "--profile", str(profile_path), "--out", str(out_path)],
    )

    assert result.exit_code == 0
    assert out_path.exists()
    assert "candidate-profile.yaml written" in result.stdout


def test_build_profile_mentions_warnings_when_present(monkeypatch, tmp_path: Path) -> None:
    cv_path = tmp_path / "cv.txt"
    profile_path = tmp_path / "profile.md"
    out_path = tmp_path / "candidate-profile.yaml"
    cv_path.write_text(
        "Tonio\nLocation: France\nLanguages: French, English\nSkills: Python, SQL\n",
        encoding="utf-8",
    )
    profile_path.write_text(
        "# Candidate Profile\n\n## Summary\nJunior data engineer.\n\n## Target Roles\n- Data Engineer\n\n## Strengths\n- Python\n\n## Skills To Emphasize\n- BigQuery\n\n## Constraints\n- CDI only\n\n## Notes\n- Years of experience feels closer to 3.\n",
        encoding="utf-8",
    )
    monkeypatch.setattr(
        "job_research.cli.extract_cv_signals",
        lambda text: {
            "name": "Tonio",
            "location": "France",
            "languages": ["French", "English"],
            "skills": ["Python", "SQL"],
            "experience_entries": [],
            "education_entries": [],
            "years_of_experience": 2,
        },
    )

    result = CliRunner().invoke(
        app,
        ["build-profile", "--cv", str(cv_path), "--profile", str(profile_path), "--out", str(out_path)],
    )

    assert result.exit_code == 0
    assert "warnings included" in result.stdout.lower()

Step 2: Run the CLI tests to verify they fail

Run: uv run pytest tests/test_cli.py -v Expected: FAIL because build-profile does not accept options or write output yet

Step 3: Implement the one-command build flow

# src/job_research/cli.py
from pathlib import Path

import typer

from job_research.profile.cv_extractor import extract_cv_signals, extract_pdf_text
from job_research.profile.merge import build_candidate_profile_output
from job_research.profile.profile_parser import parse_profile_markdown
from job_research.storage import save_candidate_profile_yaml

app = typer.Typer(help="Build one canonical candidate profile YAML")


@app.command("build-profile")
def build_profile(
    cv: Path = typer.Option(..., exists=True, readable=True),
    profile: Path = typer.Option(..., exists=True, readable=True),
    out: Path = typer.Option(Path("data/candidate-profile.yaml")),
) -> None:
    cv_text = extract_pdf_text(cv) if cv.suffix.lower() == ".pdf" else cv.read_text(encoding="utf-8")
    cv_signals = extract_cv_signals(cv_text)
    authored = parse_profile_markdown(profile.read_text(encoding="utf-8"))
    candidate_profile = build_candidate_profile_output(cv_signals, authored)
    save_candidate_profile_yaml(out, candidate_profile)

    typer.echo(f"candidate-profile.yaml written to {out}")
    if candidate_profile.warnings:
        typer.echo(f"Warnings included: {len(candidate_profile.warnings)}")
    else:
        typer.echo("No warnings included.")


def main() -> None:
    app()


if __name__ == "__main__":
    main()

Step 4: Run the CLI tests to verify they pass

Run: uv run pytest tests/test_cli.py -v Expected: PASS

Step 5: Commit the one-command workflow

git add src/job_research/cli.py tests/test_cli.py
git commit -m "feat: add one-command candidate profile build flow"

Task 7: Full Regression and Manual Validation

Files:

Modify: none
Test: tests/test_cli.py
Test: tests/test_storage.py
Test: tests/profile/test_cv_extractor.py
Test: tests/profile/test_profile_parser.py
Test: tests/profile/test_merge.py
Step 1: Run the full test suite

Run: uv run pytest tests -v Expected: PASS with all simplified first-slice tests green

Step 2: Manually verify one-command output generation

mkdir -p /tmp/job-research-simple && \
printf 'Tonio\nLocation: France\nLanguages: French, English\nSkills: Python, SQL, GCP, BigQuery\nData Engineer at Company A\n' > /tmp/job-research-simple/cv.txt && \
printf '# Candidate Profile\n\n## Summary\nJunior data engineer focused on Python and GCP.\n\n## Target Roles\n- Data Engineer\n\n## Strengths\n- Python\n- SQL\n\n## Skills To Emphasize\n- BigQuery\n- Terraform\n\n## Constraints\n- CDI only\n- France only\n\n## Notes\n- Slight preference for French listings.\n' > /tmp/job-research-simple/profile.md && \
uv run python -m job_research.cli build-profile --cv /tmp/job-research-simple/cv.txt --profile /tmp/job-research-simple/profile.md --out /tmp/job-research-simple/candidate-profile.yaml && \
uv run python - <<'PY'
from pathlib import Path
from job_research.storage import load_yaml

path = Path('/tmp/job-research-simple/candidate-profile.yaml')
print(path.exists())
print(sorted(load_yaml(path).keys()))
PY

Expected: candidate-profile.yaml exists and includes top-level keys such as summary, target_roles, skills, and warnings

Step 3: Commit the validated slice

git add pyproject.toml src/job_research tests docs/profile-template.md
git commit -m "feat: complete simplified candidate profile ingestion slice"

Spec Coverage Check

One-command workflow: covered by Tasks 1 and 6
Single canonical candidate-profile.yaml: covered by Tasks 2 and 6
Light markdown template: covered by Task 3
Deterministic-first CV extraction: covered by Task 4
Warnings instead of blocked review state: covered by Task 5
Editable human-readable YAML output: covered by Tasks 2 and 7
Repeatable CLI usage: covered by Tasks 6 and 7

25 KiB Raw Blame History

Candidate Profile Ingestion Implementation Plan

File Map

Task 1: Package Skeleton and Single Command Surface

Task 2: Canonical Output Model and YAML Persistence

Task 3: Light Markdown Profile Template and Parser

Task 4: Deterministic CV Extraction

Task 5: Canonical Profile Merge and Warning Generation

Task 6: End-to-End build-profile Command

Task 7: Full Regression and Manual Validation

Spec Coverage Check

25 KiB

Raw Blame History