# Candidate Profile Ingestion Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Build one CLI command that reads a PDF CV and a light-template markdown profile, then writes a single editable `candidate-profile.yaml` with explicit warnings when facts conflict or extraction is uncertain. **Architecture:** The implementation is a simple normalizer, not a workflow engine. Deterministic CV extraction and markdown parsing feed a profile merger that writes one canonical YAML file; warnings are embedded in that file instead of blocking output or requiring separate override files. **Tech Stack:** Python 3.13, Typer, Pydantic v2, PyYAML, pypdf, pytest --- ## File Map - Create: `src/job_research/__init__.py` — package marker - Create: `src/job_research/cli.py` — one-command Typer CLI - Create: `src/job_research/models.py` — Pydantic models for canonical profile output and warnings - Create: `src/job_research/storage.py` — YAML persistence helper - Create: `src/job_research/profile/__init__.py` — profile package marker - Create: `src/job_research/profile/cv_extractor.py` — deterministic PDF extraction and basic CV parsing - Create: `src/job_research/profile/profile_parser.py` — light markdown template parser - Create: `src/job_research/profile/merge.py` — canonical profile assembly and warning generation - Create: `tests/test_cli.py` — CLI integration tests - Create: `tests/profile/test_cv_extractor.py` — CV parsing tests - Create: `tests/profile/test_profile_parser.py` — markdown parser tests - Create: `tests/profile/test_merge.py` — profile merge and warning tests - Create: `tests/test_storage.py` — YAML output tests - Create: `docs/profile-template.md` — light required markdown template for the user - Modify: `pyproject.toml` — package metadata and dependencies ## Task 1: Package Skeleton and Single Command Surface **Files:** - Modify: `pyproject.toml` - Create: `src/job_research/__init__.py` - Create: `src/job_research/cli.py` - Create: `tests/test_cli.py` - [ ] **Step 1: Write the failing CLI help test** ```python # tests/test_cli.py from typer.testing import CliRunner from job_research.cli import app def test_cli_help_exposes_single_build_command() -> None: result = CliRunner().invoke(app, ["--help"]) assert result.exit_code == 0 assert "build-profile" in result.stdout ``` - [ ] **Step 2: Run the help test to verify it fails** Run: `uv run pytest tests/test_cli.py::test_cli_help_exposes_single_build_command -v` Expected: FAIL with `ModuleNotFoundError: No module named 'job_research'` - [ ] **Step 3: Add dependencies and the minimal CLI shell** ```toml # pyproject.toml [project] name = "job-research" version = "0.1.0" description = "Local job discovery tooling" readme = "README.md" requires-python = ">=3.13" dependencies = [ "pydantic>=2.7,<3", "pypdf>=5.0,<6", "pyyaml>=6.0,<7", "typer>=0.12,<1", ] [dependency-groups] dev = [ "pytest>=8.2,<9", ] [project.scripts] job-research = "job_research.cli:main" ``` ```python # src/job_research/__init__.py __all__ = ["__version__"] __version__ = "0.1.0" ``` ```python # src/job_research/cli.py import typer app = typer.Typer(help="Build one canonical candidate profile YAML") @app.command("build-profile") def build_profile() -> None: """Build candidate-profile.yaml from CV and markdown profile.""" def main() -> None: app() if __name__ == "__main__": main() ``` - [ ] **Step 4: Run the help test to verify it passes** Run: `uv run pytest tests/test_cli.py::test_cli_help_exposes_single_build_command -v` Expected: PASS - [ ] **Step 5: Commit the skeleton** ```bash git add pyproject.toml src/job_research/__init__.py src/job_research/cli.py tests/test_cli.py git commit -m "feat: add candidate profile CLI skeleton" ``` ## Task 2: Canonical Output Model and YAML Persistence **Files:** - Create: `src/job_research/models.py` - Create: `src/job_research/storage.py` - Create: `tests/test_storage.py` - [ ] **Step 1: Write the failing canonical YAML test** ```python # tests/test_storage.py from job_research.models import CandidateProfileOutput, ExperienceEntry, WarningItem from job_research.storage import save_candidate_profile_yaml, load_yaml def test_save_candidate_profile_yaml_round_trips_readable_output(tmp_path) -> None: profile = CandidateProfileOutput( name="Tonio", summary="Junior data engineer focused on Python and GCP.", target_roles=["Data Engineer"], strengths=["Python", "SQL"], skills_to_emphasize=["GCP", "BigQuery"], constraints=["CDI only", "France only"], notes=["Slight preference for French listings."], location="France", languages=["French", "English"], skills=["Python", "SQL", "Terraform", "GCP", "BigQuery"], experience_entries=[ExperienceEntry(company="A", title="Data Engineer")], education_entries=[], warnings=[WarningItem(field="years_of_experience", message="CV and profile disagree.")], ) out = tmp_path / "candidate-profile.yaml" save_candidate_profile_yaml(out, profile) payload = load_yaml(out) assert payload["name"] == "Tonio" assert payload["constraints"] == ["CDI only", "France only"] assert payload["warnings"][0]["field"] == "years_of_experience" ``` - [ ] **Step 2: Run the storage test to verify it fails** Run: `uv run pytest tests/test_storage.py::test_save_candidate_profile_yaml_round_trips_readable_output -v` Expected: FAIL with `ImportError` for missing models or storage helpers - [ ] **Step 3: Implement the canonical output model and YAML helper** ```python # src/job_research/models.py from pydantic import BaseModel, Field class ExperienceEntry(BaseModel): company: str title: str start: str | None = None end: str | None = None highlights: list[str] = Field(default_factory=list) class EducationEntry(BaseModel): institution: str credential: str start: str | None = None end: str | None = None class WarningItem(BaseModel): field: str message: str class CandidateProfileOutput(BaseModel): name: str | None = None summary: str | None = None target_roles: list[str] = Field(default_factory=list) strengths: list[str] = Field(default_factory=list) skills_to_emphasize: list[str] = Field(default_factory=list) constraints: list[str] = Field(default_factory=list) notes: list[str] = Field(default_factory=list) location: str | None = None languages: list[str] = Field(default_factory=list) skills: list[str] = Field(default_factory=list) experience_entries: list[ExperienceEntry] = Field(default_factory=list) education_entries: list[EducationEntry] = Field(default_factory=list) warnings: list[WarningItem] = Field(default_factory=list) ``` ```python # src/job_research/storage.py from __future__ import annotations from pathlib import Path from typing import Any import yaml from job_research.models import CandidateProfileOutput def save_candidate_profile_yaml(path: Path, profile: CandidateProfileOutput) -> None: path.parent.mkdir(parents=True, exist_ok=True) payload = profile.model_dump(mode="json") path.write_text(yaml.safe_dump(payload, sort_keys=False, allow_unicode=True), encoding="utf-8") def load_yaml(path: Path) -> dict[str, Any]: return yaml.safe_load(path.read_text(encoding="utf-8")) or {} ``` - [ ] **Step 4: Run the storage test to verify it passes** Run: `uv run pytest tests/test_storage.py::test_save_candidate_profile_yaml_round_trips_readable_output -v` Expected: PASS - [ ] **Step 5: Commit the models and storage helper** ```bash git add src/job_research/models.py src/job_research/storage.py tests/test_storage.py git commit -m "feat: add canonical candidate profile output model" ``` ## Task 3: Light Markdown Profile Template and Parser **Files:** - Create: `src/job_research/profile/__init__.py` - Create: `src/job_research/profile/profile_parser.py` - Create: `tests/profile/test_profile_parser.py` - Create: `docs/profile-template.md` - [ ] **Step 1: Write the failing markdown parser test** ```python # tests/profile/test_profile_parser.py from textwrap import dedent from job_research.profile.profile_parser import parse_profile_markdown def test_parse_profile_markdown_reads_light_required_template() -> None: markdown = dedent( """ # Candidate Profile ## Summary Junior data engineer focused on Python and GCP. ## Target Roles - Data Engineer - Analytics Engineer ## Strengths - Python - SQL ## Skills To Emphasize - BigQuery - Terraform ## Constraints - CDI only - France only ## Notes - Slight preference for French listings. """ ).strip() profile = parse_profile_markdown(markdown) assert profile.summary == "Junior data engineer focused on Python and GCP." assert profile.target_roles == ["Data Engineer", "Analytics Engineer"] assert profile.strengths == ["Python", "SQL"] assert profile.skills_to_emphasize == ["BigQuery", "Terraform"] assert profile.constraints == ["CDI only", "France only"] assert profile.notes == ["Slight preference for French listings."] ``` - [ ] **Step 2: Run the parser test to verify it fails** Run: `uv run pytest tests/profile/test_profile_parser.py::test_parse_profile_markdown_reads_light_required_template -v` Expected: FAIL with `ImportError` for missing parser - [ ] **Step 3: Implement the authored profile parser and template** ```python # src/job_research/profile/__init__.py __all__ = ["profile_parser"] ``` ```python # src/job_research/profile/profile_parser.py from __future__ import annotations from collections import defaultdict from dataclasses import dataclass, field @dataclass class AuthoredProfile: summary: str | None = None target_roles: list[str] = field(default_factory=list) strengths: list[str] = field(default_factory=list) skills_to_emphasize: list[str] = field(default_factory=list) constraints: list[str] = field(default_factory=list) notes: list[str] = field(default_factory=list) REQUIRED_SECTIONS = { "summary", "target roles", "strengths", "skills to emphasize", "constraints", "notes", } def parse_profile_markdown(markdown: str) -> AuthoredProfile: sections: dict[str, list[str]] = defaultdict(list) current_section: str | None = None for raw_line in markdown.splitlines(): line = raw_line.strip() if line.startswith("## "): current_section = line[3:].strip().lower() continue if not line or current_section is None: continue sections[current_section].append(line) missing = REQUIRED_SECTIONS - set(sections) if missing: missing_text = ", ".join(sorted(missing)) raise ValueError(f"Missing required markdown sections: {missing_text}") return AuthoredProfile( summary=" ".join(sections["summary"]), target_roles=[line[2:].strip() for line in sections["target roles"] if line.startswith("- ")], strengths=[line[2:].strip() for line in sections["strengths"] if line.startswith("- ")], skills_to_emphasize=[line[2:].strip() for line in sections["skills to emphasize"] if line.startswith("- ")], constraints=[line[2:].strip() for line in sections["constraints"] if line.startswith("- ")], notes=[line[2:].strip() if line.startswith("- ") else line for line in sections["notes"]], ) ``` ```markdown # docs/profile-template.md # Candidate Profile ## Summary One short paragraph describing your target profile. ## Target Roles - Data Engineer ## Strengths - Python - SQL ## Skills To Emphasize - GCP - BigQuery ## Constraints - CDI only - France only ## Notes - Anything the CV parser might miss but future ranking should understand. ``` - [ ] **Step 4: Run the parser test to verify it passes** Run: `uv run pytest tests/profile/test_profile_parser.py::test_parse_profile_markdown_reads_light_required_template -v` Expected: PASS - [ ] **Step 5: Commit the markdown parser** ```bash git add src/job_research/profile/__init__.py src/job_research/profile/profile_parser.py tests/profile/test_profile_parser.py docs/profile-template.md git commit -m "feat: add light markdown candidate profile parser" ``` ## Task 4: Deterministic CV Extraction **Files:** - Create: `src/job_research/profile/cv_extractor.py` - Create: `tests/profile/test_cv_extractor.py` - [ ] **Step 1: Write the failing CV extraction test** ```python # tests/profile/test_cv_extractor.py from job_research.profile.cv_extractor import extract_cv_signals def test_extract_cv_signals_reads_basic_fields_from_text() -> None: text = """ Tonio Location: France Languages: French, English Skills: Python, SQL, Terraform, GCP, BigQuery Data Engineer at Company A Analytics Engineer at Company B """ extracted = extract_cv_signals(text) assert extracted["name"] == "Tonio" assert extracted["location"] == "France" assert extracted["languages"] == ["French", "English"] assert extracted["skills"] == ["Python", "SQL", "Terraform", "GCP", "BigQuery"] assert extracted["experience_entries"][0]["title"] == "Data Engineer" assert len(extracted["experience_entries"]) == 2 ``` - [ ] **Step 2: Run the CV extraction test to verify it fails** Run: `uv run pytest tests/profile/test_cv_extractor.py::test_extract_cv_signals_reads_basic_fields_from_text -v` Expected: FAIL with `ImportError` for missing CV extractor - [ ] **Step 3: Implement PDF text extraction and basic signal parsing** ```python # src/job_research/profile/cv_extractor.py from __future__ import annotations from pathlib import Path from pypdf import PdfReader def extract_pdf_text(path: Path) -> str: reader = PdfReader(str(path)) return "\n".join(page.extract_text() or "" for page in reader.pages) def extract_cv_signals(text: str) -> dict: lines = [line.strip() for line in text.splitlines() if line.strip()] profile = { "name": lines[0] if lines else None, "location": None, "languages": [], "skills": [], "experience_entries": [], "education_entries": [], } for line in lines: lower = line.lower() if lower.startswith("location:"): profile["location"] = line.split(":", 1)[1].strip() elif lower.startswith("languages:"): profile["languages"] = [part.strip() for part in line.split(":", 1)[1].split(",") if part.strip()] elif lower.startswith("skills:"): profile["skills"] = [part.strip() for part in line.split(":", 1)[1].split(",") if part.strip()] elif " at " in line: title, company = line.split(" at ", 1) profile["experience_entries"].append({"title": title.strip(), "company": company.strip()}) return profile ``` - [ ] **Step 4: Run the CV extraction test to verify it passes** Run: `uv run pytest tests/profile/test_cv_extractor.py::test_extract_cv_signals_reads_basic_fields_from_text -v` Expected: PASS - [ ] **Step 5: Commit the CV extractor** ```bash git add src/job_research/profile/cv_extractor.py tests/profile/test_cv_extractor.py git commit -m "feat: add deterministic CV extraction" ``` ## Task 5: Canonical Profile Merge and Warning Generation **Files:** - Create: `src/job_research/profile/merge.py` - Create: `tests/profile/test_merge.py` - [ ] **Step 1: Write the failing merge test for warnings** ```python # tests/profile/test_merge.py from job_research.profile.merge import build_candidate_profile_output from job_research.profile.profile_parser import AuthoredProfile def test_build_candidate_profile_output_writes_warning_when_facts_conflict() -> None: cv_signals = { "name": "Tonio", "location": "France", "languages": ["French", "English"], "skills": ["Python", "SQL"], "experience_entries": [{"title": "Data Engineer", "company": "A"}], "education_entries": [], "years_of_experience": 2, } authored = AuthoredProfile( summary="Junior data engineer focused on GCP.", target_roles=["Data Engineer"], strengths=["Python"], skills_to_emphasize=["BigQuery", "GCP"], constraints=["CDI only"], notes=["Years of experience feels closer to 3."], ) output = build_candidate_profile_output(cv_signals, authored) assert output.summary == "Junior data engineer focused on GCP." assert output.constraints == ["CDI only"] assert output.warnings[0].field == "years_of_experience" ``` - [ ] **Step 2: Run the merge test to verify it fails** Run: `uv run pytest tests/profile/test_merge.py::test_build_candidate_profile_output_writes_warning_when_facts_conflict -v` Expected: FAIL with `ImportError` for missing merge module - [ ] **Step 3: Implement the merger and warnings** ```python # src/job_research/profile/merge.py from __future__ import annotations from job_research.models import CandidateProfileOutput, ExperienceEntry, EducationEntry, WarningItem from job_research.profile.profile_parser import AuthoredProfile def build_candidate_profile_output(cv_signals: dict, authored: AuthoredProfile) -> CandidateProfileOutput: warnings: list[WarningItem] = [] if "years_of_experience" in cv_signals and any("experience" in note.lower() for note in authored.notes): warnings.append( WarningItem( field="years_of_experience", message="CV-derived experience may differ from markdown interpretation. Review manually.", ) ) merged_skills: list[str] = [] for skill in [*cv_signals.get("skills", []), *authored.strengths, *authored.skills_to_emphasize]: if skill not in merged_skills: merged_skills.append(skill) return CandidateProfileOutput( name=cv_signals.get("name"), summary=authored.summary, target_roles=authored.target_roles, strengths=authored.strengths, skills_to_emphasize=authored.skills_to_emphasize, constraints=authored.constraints, notes=authored.notes, location=cv_signals.get("location"), languages=cv_signals.get("languages", []), skills=merged_skills, experience_entries=[ExperienceEntry.model_validate(item) for item in cv_signals.get("experience_entries", [])], education_entries=[EducationEntry.model_validate(item) for item in cv_signals.get("education_entries", [])], warnings=warnings, ) ``` - [ ] **Step 4: Run the merge test to verify it passes** Run: `uv run pytest tests/profile/test_merge.py::test_build_candidate_profile_output_writes_warning_when_facts_conflict -v` Expected: PASS - [ ] **Step 5: Commit the merger** ```bash git add src/job_research/profile/merge.py tests/profile/test_merge.py git commit -m "feat: add canonical profile merger with warnings" ``` ## Task 6: End-to-End build-profile Command **Files:** - Modify: `src/job_research/cli.py` - Modify: `tests/test_cli.py` - [ ] **Step 1: Write the failing end-to-end CLI tests** ```python # tests/test_cli.py from pathlib import Path from typer.testing import CliRunner from job_research.cli import app def test_build_profile_writes_canonical_yaml(monkeypatch, tmp_path: Path) -> None: cv_path = tmp_path / "cv.txt" profile_path = tmp_path / "profile.md" out_path = tmp_path / "candidate-profile.yaml" cv_path.write_text( "Tonio\nLocation: France\nLanguages: French, English\nSkills: Python, SQL\nData Engineer at Company A\n", encoding="utf-8", ) profile_path.write_text( "# Candidate Profile\n\n## Summary\nJunior data engineer.\n\n## Target Roles\n- Data Engineer\n\n## Strengths\n- Python\n\n## Skills To Emphasize\n- BigQuery\n\n## Constraints\n- CDI only\n\n## Notes\n- Slight preference for French listings.\n", encoding="utf-8", ) result = CliRunner().invoke( app, ["build-profile", "--cv", str(cv_path), "--profile", str(profile_path), "--out", str(out_path)], ) assert result.exit_code == 0 assert out_path.exists() assert "candidate-profile.yaml written" in result.stdout def test_build_profile_mentions_warnings_when_present(monkeypatch, tmp_path: Path) -> None: cv_path = tmp_path / "cv.txt" profile_path = tmp_path / "profile.md" out_path = tmp_path / "candidate-profile.yaml" cv_path.write_text( "Tonio\nLocation: France\nLanguages: French, English\nSkills: Python, SQL\n", encoding="utf-8", ) profile_path.write_text( "# Candidate Profile\n\n## Summary\nJunior data engineer.\n\n## Target Roles\n- Data Engineer\n\n## Strengths\n- Python\n\n## Skills To Emphasize\n- BigQuery\n\n## Constraints\n- CDI only\n\n## Notes\n- Years of experience feels closer to 3.\n", encoding="utf-8", ) monkeypatch.setattr( "job_research.cli.extract_cv_signals", lambda text: { "name": "Tonio", "location": "France", "languages": ["French", "English"], "skills": ["Python", "SQL"], "experience_entries": [], "education_entries": [], "years_of_experience": 2, }, ) result = CliRunner().invoke( app, ["build-profile", "--cv", str(cv_path), "--profile", str(profile_path), "--out", str(out_path)], ) assert result.exit_code == 0 assert "warnings included" in result.stdout.lower() ``` - [ ] **Step 2: Run the CLI tests to verify they fail** Run: `uv run pytest tests/test_cli.py -v` Expected: FAIL because `build-profile` does not accept options or write output yet - [ ] **Step 3: Implement the one-command build flow** ```python # src/job_research/cli.py from pathlib import Path import typer from job_research.profile.cv_extractor import extract_cv_signals, extract_pdf_text from job_research.profile.merge import build_candidate_profile_output from job_research.profile.profile_parser import parse_profile_markdown from job_research.storage import save_candidate_profile_yaml app = typer.Typer(help="Build one canonical candidate profile YAML") @app.command("build-profile") def build_profile( cv: Path = typer.Option(..., exists=True, readable=True), profile: Path = typer.Option(..., exists=True, readable=True), out: Path = typer.Option(Path("data/candidate-profile.yaml")), ) -> None: cv_text = extract_pdf_text(cv) if cv.suffix.lower() == ".pdf" else cv.read_text(encoding="utf-8") cv_signals = extract_cv_signals(cv_text) authored = parse_profile_markdown(profile.read_text(encoding="utf-8")) candidate_profile = build_candidate_profile_output(cv_signals, authored) save_candidate_profile_yaml(out, candidate_profile) typer.echo(f"candidate-profile.yaml written to {out}") if candidate_profile.warnings: typer.echo(f"Warnings included: {len(candidate_profile.warnings)}") else: typer.echo("No warnings included.") def main() -> None: app() if __name__ == "__main__": main() ``` - [ ] **Step 4: Run the CLI tests to verify they pass** Run: `uv run pytest tests/test_cli.py -v` Expected: PASS - [ ] **Step 5: Commit the one-command workflow** ```bash git add src/job_research/cli.py tests/test_cli.py git commit -m "feat: add one-command candidate profile build flow" ``` ## Task 7: Full Regression and Manual Validation **Files:** - Modify: none - Test: `tests/test_cli.py` - Test: `tests/test_storage.py` - Test: `tests/profile/test_cv_extractor.py` - Test: `tests/profile/test_profile_parser.py` - Test: `tests/profile/test_merge.py` - [ ] **Step 1: Run the full test suite** Run: `uv run pytest tests -v` Expected: PASS with all simplified first-slice tests green - [ ] **Step 2: Manually verify one-command output generation** ```bash mkdir -p /tmp/job-research-simple && \ printf 'Tonio\nLocation: France\nLanguages: French, English\nSkills: Python, SQL, GCP, BigQuery\nData Engineer at Company A\n' > /tmp/job-research-simple/cv.txt && \ printf '# Candidate Profile\n\n## Summary\nJunior data engineer focused on Python and GCP.\n\n## Target Roles\n- Data Engineer\n\n## Strengths\n- Python\n- SQL\n\n## Skills To Emphasize\n- BigQuery\n- Terraform\n\n## Constraints\n- CDI only\n- France only\n\n## Notes\n- Slight preference for French listings.\n' > /tmp/job-research-simple/profile.md && \ uv run python -m job_research.cli build-profile --cv /tmp/job-research-simple/cv.txt --profile /tmp/job-research-simple/profile.md --out /tmp/job-research-simple/candidate-profile.yaml && \ uv run python - <<'PY' from pathlib import Path from job_research.storage import load_yaml path = Path('/tmp/job-research-simple/candidate-profile.yaml') print(path.exists()) print(sorted(load_yaml(path).keys())) PY ``` Expected: `candidate-profile.yaml` exists and includes top-level keys such as `summary`, `target_roles`, `skills`, and `warnings` - [ ] **Step 3: Commit the validated slice** ```bash git add pyproject.toml src/job_research tests docs/profile-template.md git commit -m "feat: complete simplified candidate profile ingestion slice" ``` ## Spec Coverage Check - One-command workflow: covered by Tasks 1 and 6 - Single canonical `candidate-profile.yaml`: covered by Tasks 2 and 6 - Light markdown template: covered by Task 3 - Deterministic-first CV extraction: covered by Task 4 - Warnings instead of blocked review state: covered by Task 5 - Editable human-readable YAML output: covered by Tasks 2 and 7 - Repeatable CLI usage: covered by Tasks 6 and 7