37 lines
3.6 KiB
Markdown
37 lines
3.6 KiB
Markdown
# SOURCES — v1 source decision
|
||
|
||
**Decision (2026-11-10): Kijiji — NS + NB windows (`kijiji-ns` / `kijiji-nb`). One site, one parser, two regional feeds.**
|
||
|
||
Chosen after the Torres engineering consultation; the evidence below is from live probes + snapshot analysis run 2026-11-10 (memo + probe scripts kept with the project's working notes).
|
||
|
||
## Candidates at a glance
|
||
|
||
| candidate | live volume | churn | field fill | photos | parse | verdict |
|
||
|---|---|---|---|---|---|---|
|
||
| **Kijiji** (ns+nb) | ~640 cards in the current window; depth beyond it (page 15 still full in NS; ≥600 NS alone) | **~18 new + 18 gone / day (NS)** | price ~100%, km 30–36%, title/loc/desc ~100% | image on **every card** at parse time | SSR HTML; transient-partial quirk **already mitigated** (retries + last-known-good, re-verified stable) | **v1 source** |
|
||
| AutoTrader.ca (ns+nb) | ~117 cruiser listings; full motorcycle view padded with ATV/SxS (34/53 and 49/64 rows from ONE dealer) | slow | 100% (structured JSON) | inline | structured `__NEXT_DATA__` JSON — the easiest parse | source #2 later (field-rich enrichment) |
|
||
| Dealer Spike (procycle / tbb) | 160 units each | none (static stock) | price only; no km/desc/year | present | HTML w/ quirks | later source (new stock) |
|
||
| Facebook Marketplace | — | — | — | — | — | **never scraped — Glenn's manual column** |
|
||
|
||
## Why Kijiji over "the easiest scrape"
|
||
|
||
Glenn's words were "the easiest to scrape." Literally easiest = AutoTrader (structured JSON, `numberOfResults` handed to you). But the deck is the product: Kijiji carries ~4–5× the live volume, real daily churn to feed a nightly deck, a photo on every card, and a private + dealer mix — and its one known flakiness is already a solved problem in the production crawler that has run daily for weeks. Engineering ease that ships a thin, slow-turning deck is not actually easy. AutoTrader returns as source #2 through the same import path (its parser already exists in the hunt pipeline).
|
||
|
||
## Evidence highlights (2026-11-10)
|
||
|
||
- Snapshot: `kijiji-ns` 320 items, `kijiji-nb` 320 items; **74 cross-province twin groups (normalized title + price) → 80 dup rows under the v1 collapse-to-one rule** (verified against the live import 2026-11-10; the earlier "≈58" was a rough estimate). Two multi-unit dealer groups (BMW G 310 R) over-merged under that rule; **P2 fixed it (2026-11-10)** — `min(ns,nb)` completeness pairing: 640 rows → **563 active + 77 dup**, surplus siblings stay active.
|
||
- Live fetch: page 1 fetched twice → byte-identical 40 cards (no flake that day); **40/40 cards carry image URLs**; pages 9 and 15 still full (depth available beyond the current window).
|
||
- Images: `media.kijiji.ca` URLs are **token-free and size-rewritable** (`?rule=kijijica-640-webp` → 200; `960-webp` rewrite → 200; URL without rule → 404 — the rule is the size selector, not a token).
|
||
- Anti-bot: none observed across 15 pages of probes + production traffic.
|
||
|
||
## Scope & growth path
|
||
|
||
- **v1 mirrors the proven 8-page/province window** (~640 cards). Depth exists past it; grow only after two clean weeks, throttled — anti-bot behavior on deep sweeps is untested long-term.
|
||
- **v2** adds AutoTrader (same importer path) — dense fields complement Kijiji's private-sale mix.
|
||
- **Later** dealer inventories (new stock); the import schema is source-agnostic from day one.
|
||
|
||
## Revisit criteria
|
||
|
||
- Kijiji block / anti-bot escalation persisting ≥2 weeks → flip the order, AutoTrader first.
|
||
- Kijiji markup drift breaking the parser → importer gates fail loud (deck stays last-known-good); the crawler's `fail_streak` monitoring already covers detection.
|