job-research/docs/superpowers/specs/2026-06-01-apec-ingestion-design.md

Apec Ingestion Design

This spec covers the next executable slice of the job discovery system: use the canonical Candidate Profile to fetch public Apec listings, store raw listing snapshots, and produce a first normalized YAML listing file for inspection. It stops before ranking, explanations, and multi-source support.

Scope

This slice must let the user:

• run one explicit command,
• read data/candidate-profile.yaml,
• derive deterministic Apec search queries from that profile,
• fetch up to 50 public Apec listings,
• store raw listing detail-page HTML and crawl metadata,
• produce one normalized YAML file of listings,
• tolerate partial failure by keeping successful listings and recording per-listing errors.

Out of scope:

• ranking,
• LLM explanation,
• non-Apec sources,
• cross-run or cross-source deduplication,
• database storage,
• authenticated sessions.

Architecture

The design is a fetch-and-normalize pipeline centered on one explicit command:

• fetch-apec

Inputs:

• data/candidate-profile.yaml

Outputs:

• raw Apec listing snapshots,
• per-run metadata,
• one normalized listings.yaml file.

The key boundary is that this slice acquires and structures listing data but does not score it. It should preserve enough raw evidence to debug extraction and enough normalized structure to support the future ranking slice.

Components

Profile Query Derivation

Responsibility:

• read the canonical Candidate Profile,
• derive a small deterministic set of Apec search queries from profile content.

This should use explicit profile fields such as:

• target roles,
• strengths,
• skills to emphasize,
• constraints.

It must be deterministic and reproducible, not LLM-generated.

Apec Search Adapter

Responsibility:

• execute public Apec searches,
• gather listing URLs and lightweight result metadata,
• paginate as needed,
• respect the 50-listing cap across the full run.

Listing Snapshot Fetcher

Responsibility:

• visit each public Apec listing detail page,
• store:
  • listing URL,
  • raw HTML,
  • crawl timestamp,
  • basic fetch metadata,
  • per-listing error information on failure.

Normalizer

Responsibility:

• transform successful Apec detail pages into a first normalized listing shape,
• extract obvious fields without overreaching.

Within-Run Deduper

Responsibility:

• remove obvious duplicates within the current run only,
• use Apec URL or source job ID when available.

This slice does not attempt cross-run or cross-source deduplication.

Data Flow

1. The user runs fetch-apec.
2. The system reads data/candidate-profile.yaml.
3. It derives a small deterministic set of Apec search queries.
4. It fetches search results from Apec.
5. It caps the full run at 50 listings.
6. For each listing URL:
   • fetch the public detail page,
   • store raw HTML and crawl metadata,
   • normalize successful pages into listing records,
   • record per-listing errors when fetch or normalization fails.
7. It performs minimal within-run deduplication.
8. It writes:
   • listings.yaml
   • run-meta.yaml
   • snapshot HTML files
9. It prints a concise run summary.

File Layout

data/
  candidate-profile.yaml
  apec/
    runs/
      <timestamp>/
        listings.yaml
        run-meta.yaml
        snapshots/
          <listing-id-or-slug>.html

File roles:

• listings.yaml contains normalized deduplicated listings for that run
• run-meta.yaml contains derived queries, counts, crawl metadata, and per-listing errors
• snapshots/*.html contains raw listing detail pages for debugging and later extraction improvements

Command Surface

This slice should expose one explicit command:

• uv run job-research fetch-apec

Expected behavior:

• load the default candidate profile,
• derive queries automatically,
• fetch public Apec listings,
• write run artifacts,
• print a summary including:
  • query count,
  • fetched listing count,
  • normalized listing count,
  • deduplicated count,
  • failed listing count.

Normalized Listing Shape

The first normalized listing shape should include:

• source
• source_job_id when available
• url
• title
• company
• location
• contract_type
• description_text
• published_at when available
• fetched_at
• warnings

This is intentionally minimal. Salary, remote mode, seniority inference, and richer tagging belong to later slices unless extraction proves stable enough to add them cheaply.

Validation and Error Handling

Hard Failures

These should fail the run before or during crawling:

• missing or unreadable data/candidate-profile.yaml,
• no usable derived search queries,
• Apec unreachable for the whole run,
• every listing fetch failing.

Partial-Success Behavior

These should still produce output artifacts:

• some listing detail pages fail to load,
• some listing pages load but normalize incompletely,
• some duplicates are removed during within-run deduplication.

When partial success occurs, the run should:

• keep successful normalized listings,
• record per-listing failures in run-meta.yaml,
• preserve available snapshots.

Testing Strategy

The slice should be tested around correctness of ingestion, resilience, and artifact shape.

Core test cases:

• deterministic query derivation from candidate-profile.yaml,
• 50-listing cap behavior,
• snapshot persistence for successful detail pages,
• partial-success handling with per-listing errors,
• minimal within-run deduplication,
• normalized YAML output shape,
• run summary metadata.

Acceptance Criteria

This slice is done when:

• one command can fetch public Apec listings from the canonical candidate profile,
• raw detail-page HTML snapshots are preserved,
• one normalized listings.yaml file is produced,
• partial failures are visible rather than silent,
• the resulting listing artifact is good enough to feed the later ranking slice.

Non-Goals for This Slice

This design intentionally does not solve:

• listing ranking,
• fit explanations,
• source-agnostic ingestion,
• multi-board aggregation,
• database-backed run history,
• login-required crawling,
• long-term deduplication across runs.

Those belong to later slices after raw listing acquisition is stable.