3.3 KiB
SOURCES — v1 source decision
Decision (2026-11-10): Kijiji — NS + NB windows (kijiji-ns / kijiji-nb). One site, one parser, two regional feeds.
Chosen after the Torres engineering consultation; the evidence below is from live probes + snapshot analysis run 2026-11-10 (memo + probe scripts kept with the project's working notes).
Candidates at a glance
| candidate | live volume | churn | field fill | photos | parse | verdict |
|---|---|---|---|---|---|---|
| Kijiji (ns+nb) | ~640 cards in the current window; depth beyond it (page 15 still full in NS; ≥600 NS alone) | ~18 new + 18 gone / day (NS) | price ~100%, km 30–36%, title/loc/desc ~100% | image on every card at parse time | SSR HTML; transient-partial quirk already mitigated (retries + last-known-good, re-verified stable) | v1 source |
| AutoTrader.ca (ns+nb) | ~117 cruiser listings; full motorcycle view padded with ATV/SxS (34/53 and 49/64 rows from ONE dealer) | slow | 100% (structured JSON) | inline | structured __NEXT_DATA__ JSON — the easiest parse |
source #2 later (field-rich enrichment) |
| Dealer Spike (procycle / tbb) | 160 units each | none (static stock) | price only; no km/desc/year | present | HTML w/ quirks | later source (new stock) |
| Facebook Marketplace | — | — | — | — | — | never scraped — Glenn's manual column |
Why Kijiji over "the easiest scrape"
Glenn's words were "the easiest to scrape." Literally easiest = AutoTrader (structured JSON, numberOfResults handed to you). But the deck is the product: Kijiji carries ~4–5× the live volume, real daily churn to feed a nightly deck, a photo on every card, and a private + dealer mix — and its one known flakiness is already a solved problem in the production crawler that has run daily for weeks. Engineering ease that ships a thin, slow-turning deck is not actually easy. AutoTrader returns as source #2 through the same import path (its parser already exists in the hunt pipeline).
Evidence highlights (2026-11-10)
- Snapshot:
kijiji-ns320 items,kijiji-nb320 items; 58/320 NS rows are NB cross-post twins → importer dedupes. - Live fetch: page 1 fetched twice → byte-identical 40 cards (no flake that day); 40/40 cards carry image URLs; pages 9 and 15 still full (depth available beyond the current window).
- Images:
media.kijiji.caURLs are token-free and size-rewritable (?rule=kijijica-640-webp→ 200;960-webprewrite → 200; URL without rule → 404 — the rule is the size selector, not a token). - Anti-bot: none observed across 15 pages of probes + production traffic.
Scope & growth path
- v1 mirrors the proven 8-page/province window (~640 cards). Depth exists past it; grow only after two clean weeks, throttled — anti-bot behavior on deep sweeps is untested long-term.
- v2 adds AutoTrader (same importer path) — dense fields complement Kijiji's private-sale mix.
- Later dealer inventories (new stock); the import schema is source-agnostic from day one.
Revisit criteria
- Kijiji block / anti-bot escalation persisting ≥2 weeks → flip the order, AutoTrader first.
- Kijiji markup drift breaking the parser → importer gates fail loud (deck stays last-known-good); the crawler's
fail_streakmonitoring already covers detection.