docs: add Apec ingestion design
This commit is contained in:
Antoine
parent
fb150dbe69
commit
ad36de0a3f
236
docs/superpowers/specs/2026-06-01-apec-ingestion-design.md
Normal file
236
docs/superpowers/specs/2026-06-01-apec-ingestion-design.md
Normal file
@ -0,0 +1,236 @@
|
|||||||
|
# Apec Ingestion Design
|
||||||
|
|
||||||
|
This spec covers the next executable slice of the job discovery system: use the canonical `Candidate Profile` to fetch public Apec listings, store raw listing snapshots, and produce a first normalized YAML listing file for inspection. It stops before ranking, explanations, and multi-source support.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This slice must let the user:
|
||||||
|
|
||||||
|
- run one explicit command,
|
||||||
|
- read `data/candidate-profile.yaml`,
|
||||||
|
- derive deterministic Apec search queries from that profile,
|
||||||
|
- fetch up to 50 public Apec listings,
|
||||||
|
- store raw listing detail-page HTML and crawl metadata,
|
||||||
|
- produce one normalized YAML file of listings,
|
||||||
|
- tolerate partial failure by keeping successful listings and recording per-listing errors.
|
||||||
|
|
||||||
|
Out of scope:
|
||||||
|
|
||||||
|
- ranking,
|
||||||
|
- LLM explanation,
|
||||||
|
- non-Apec sources,
|
||||||
|
- cross-run or cross-source deduplication,
|
||||||
|
- database storage,
|
||||||
|
- authenticated sessions.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
The design is a fetch-and-normalize pipeline centered on one explicit command:
|
||||||
|
|
||||||
|
- `fetch-apec`
|
||||||
|
|
||||||
|
Inputs:
|
||||||
|
|
||||||
|
- `data/candidate-profile.yaml`
|
||||||
|
|
||||||
|
Outputs:
|
||||||
|
|
||||||
|
- raw Apec listing snapshots,
|
||||||
|
- per-run metadata,
|
||||||
|
- one normalized `listings.yaml` file.
|
||||||
|
|
||||||
|
The key boundary is that this slice acquires and structures listing data but does not score it. It should preserve enough raw evidence to debug extraction and enough normalized structure to support the future ranking slice.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
### Profile Query Derivation
|
||||||
|
|
||||||
|
Responsibility:
|
||||||
|
|
||||||
|
- read the canonical `Candidate Profile`,
|
||||||
|
- derive a small deterministic set of Apec search queries from profile content.
|
||||||
|
|
||||||
|
This should use explicit profile fields such as:
|
||||||
|
|
||||||
|
- target roles,
|
||||||
|
- strengths,
|
||||||
|
- skills to emphasize,
|
||||||
|
- constraints.
|
||||||
|
|
||||||
|
It must be deterministic and reproducible, not LLM-generated.
|
||||||
|
|
||||||
|
### Apec Search Adapter
|
||||||
|
|
||||||
|
Responsibility:
|
||||||
|
|
||||||
|
- execute public Apec searches,
|
||||||
|
- gather listing URLs and lightweight result metadata,
|
||||||
|
- paginate as needed,
|
||||||
|
- respect the 50-listing cap across the full run.
|
||||||
|
|
||||||
|
### Listing Snapshot Fetcher
|
||||||
|
|
||||||
|
Responsibility:
|
||||||
|
|
||||||
|
- visit each public Apec listing detail page,
|
||||||
|
- store:
|
||||||
|
- listing URL,
|
||||||
|
- raw HTML,
|
||||||
|
- crawl timestamp,
|
||||||
|
- basic fetch metadata,
|
||||||
|
- per-listing error information on failure.
|
||||||
|
|
||||||
|
### Normalizer
|
||||||
|
|
||||||
|
Responsibility:
|
||||||
|
|
||||||
|
- transform successful Apec detail pages into a first normalized listing shape,
|
||||||
|
- extract obvious fields without overreaching.
|
||||||
|
|
||||||
|
### Within-Run Deduper
|
||||||
|
|
||||||
|
Responsibility:
|
||||||
|
|
||||||
|
- remove obvious duplicates within the current run only,
|
||||||
|
- use Apec URL or source job ID when available.
|
||||||
|
|
||||||
|
This slice does not attempt cross-run or cross-source deduplication.
|
||||||
|
|
||||||
|
## Data Flow
|
||||||
|
|
||||||
|
1. The user runs `fetch-apec`.
|
||||||
|
2. The system reads `data/candidate-profile.yaml`.
|
||||||
|
3. It derives a small deterministic set of Apec search queries.
|
||||||
|
4. It fetches search results from Apec.
|
||||||
|
5. It caps the full run at 50 listings.
|
||||||
|
6. For each listing URL:
|
||||||
|
- fetch the public detail page,
|
||||||
|
- store raw HTML and crawl metadata,
|
||||||
|
- normalize successful pages into listing records,
|
||||||
|
- record per-listing errors when fetch or normalization fails.
|
||||||
|
7. It performs minimal within-run deduplication.
|
||||||
|
8. It writes:
|
||||||
|
- `listings.yaml`
|
||||||
|
- `run-meta.yaml`
|
||||||
|
- snapshot HTML files
|
||||||
|
9. It prints a concise run summary.
|
||||||
|
|
||||||
|
## File Layout
|
||||||
|
|
||||||
|
```text
|
||||||
|
data/
|
||||||
|
candidate-profile.yaml
|
||||||
|
apec/
|
||||||
|
runs/
|
||||||
|
<timestamp>/
|
||||||
|
listings.yaml
|
||||||
|
run-meta.yaml
|
||||||
|
snapshots/
|
||||||
|
<listing-id-or-slug>.html
|
||||||
|
```
|
||||||
|
|
||||||
|
File roles:
|
||||||
|
|
||||||
|
- `listings.yaml` contains normalized deduplicated listings for that run
|
||||||
|
- `run-meta.yaml` contains derived queries, counts, crawl metadata, and per-listing errors
|
||||||
|
- `snapshots/*.html` contains raw listing detail pages for debugging and later extraction improvements
|
||||||
|
|
||||||
|
## Command Surface
|
||||||
|
|
||||||
|
This slice should expose one explicit command:
|
||||||
|
|
||||||
|
- `uv run job-research fetch-apec`
|
||||||
|
|
||||||
|
Expected behavior:
|
||||||
|
|
||||||
|
- load the default candidate profile,
|
||||||
|
- derive queries automatically,
|
||||||
|
- fetch public Apec listings,
|
||||||
|
- write run artifacts,
|
||||||
|
- print a summary including:
|
||||||
|
- query count,
|
||||||
|
- fetched listing count,
|
||||||
|
- normalized listing count,
|
||||||
|
- deduplicated count,
|
||||||
|
- failed listing count.
|
||||||
|
|
||||||
|
## Normalized Listing Shape
|
||||||
|
|
||||||
|
The first normalized listing shape should include:
|
||||||
|
|
||||||
|
- `source`
|
||||||
|
- `source_job_id` when available
|
||||||
|
- `url`
|
||||||
|
- `title`
|
||||||
|
- `company`
|
||||||
|
- `location`
|
||||||
|
- `contract_type`
|
||||||
|
- `description_text`
|
||||||
|
- `published_at` when available
|
||||||
|
- `fetched_at`
|
||||||
|
- `warnings`
|
||||||
|
|
||||||
|
This is intentionally minimal. Salary, remote mode, seniority inference, and richer tagging belong to later slices unless extraction proves stable enough to add them cheaply.
|
||||||
|
|
||||||
|
## Validation and Error Handling
|
||||||
|
|
||||||
|
### Hard Failures
|
||||||
|
|
||||||
|
These should fail the run before or during crawling:
|
||||||
|
|
||||||
|
- missing or unreadable `data/candidate-profile.yaml`,
|
||||||
|
- no usable derived search queries,
|
||||||
|
- Apec unreachable for the whole run,
|
||||||
|
- every listing fetch failing.
|
||||||
|
|
||||||
|
### Partial-Success Behavior
|
||||||
|
|
||||||
|
These should still produce output artifacts:
|
||||||
|
|
||||||
|
- some listing detail pages fail to load,
|
||||||
|
- some listing pages load but normalize incompletely,
|
||||||
|
- some duplicates are removed during within-run deduplication.
|
||||||
|
|
||||||
|
When partial success occurs, the run should:
|
||||||
|
|
||||||
|
- keep successful normalized listings,
|
||||||
|
- record per-listing failures in `run-meta.yaml`,
|
||||||
|
- preserve available snapshots.
|
||||||
|
|
||||||
|
## Testing Strategy
|
||||||
|
|
||||||
|
The slice should be tested around correctness of ingestion, resilience, and artifact shape.
|
||||||
|
|
||||||
|
Core test cases:
|
||||||
|
|
||||||
|
- deterministic query derivation from `candidate-profile.yaml`,
|
||||||
|
- 50-listing cap behavior,
|
||||||
|
- snapshot persistence for successful detail pages,
|
||||||
|
- partial-success handling with per-listing errors,
|
||||||
|
- minimal within-run deduplication,
|
||||||
|
- normalized YAML output shape,
|
||||||
|
- run summary metadata.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
This slice is done when:
|
||||||
|
|
||||||
|
- one command can fetch public Apec listings from the canonical candidate profile,
|
||||||
|
- raw detail-page HTML snapshots are preserved,
|
||||||
|
- one normalized `listings.yaml` file is produced,
|
||||||
|
- partial failures are visible rather than silent,
|
||||||
|
- the resulting listing artifact is good enough to feed the later ranking slice.
|
||||||
|
|
||||||
|
## Non-Goals for This Slice
|
||||||
|
|
||||||
|
This design intentionally does not solve:
|
||||||
|
|
||||||
|
- listing ranking,
|
||||||
|
- fit explanations,
|
||||||
|
- source-agnostic ingestion,
|
||||||
|
- multi-board aggregation,
|
||||||
|
- database-backed run history,
|
||||||
|
- login-required crawling,
|
||||||
|
- long-term deduplication across runs.
|
||||||
|
|
||||||
|
Those belong to later slices after raw listing acquisition is stable.
|
||||||
Loading…
x
Reference in New Issue
Block a user