Bike-Tinder/docs/DESIGN.md

DESIGN — Bike-Tinder

Architecture for v1 (single source, single user, tailnet-only PWA). Written 2026-11-10 by Alex; engineering input from the Torres consultation (same date). Items marked 【decided】 are locked for v1.

Components

piece	what	where
Crawler	existing motorcycle-hunt-scan.py (unchanged) — sweeps Kijiji NS+NB nightly at 08:00	Alex's pipeline (~/.hermes/scripts/)
Importer	reads the crawler's state JSONs, gates them, upserts into the app DB, scores, dedupes, queues images	app/importer.py
API	FastAPI + uvicorn — deck, swipes, list, stats, images	app/api.py
PWA	no-build single-page frontend (deck, list, matches, trends)	app/static/
DB	SQLite (WAL mode)	data/bike-tinder.sqlite3 (stack volume)

Data flow

08:00  motorcycle-hunt crawl (existing, unchanged) → state JSONs: kijiji-ns.json, kijiji-nb.json
08:30  importer — script-only, zero LLM (Hermes cron `bike-tinder-import` → POST /api/import):
         gates → upsert → dedupe → score → gone-marking → image queue
       → SQLite (WAL)
Deck   phone ⇄ API ⇄ SQLite
             ↘ images: local mirror first, hotlink (302) fallback

Import design 【decided】

• Source of truth = the hunt's state files, not a re-crawl. Rationale: zero duplicate load on Kijiji; inherits the crawler's retry / partial / last-known-good / absence logic instead of re-implementing it; the format is stable and self-describing (ok, partial, fetched_at, per-page counts).
• Escape hatch: the app vendors the Kijiji parser from reference/motorcycle-hunt-scan.py behind a --direct flag — if the hunt pipeline ever moves or dies, the importer can sweep Kijiji itself.
• Gates (fail loud, keep last good): each state file must have ok == true, partial == false, fetched_at within 26 h, and non-empty items. Any failure → abort that region, log to import_runs, leave the DB untouched.
• Gone marking: a listing absent from ≥2 consecutive good imports flips to gone (never deleted; reappearance flips it back). House discipline: a single miss — or any blocked page — never counts as "sold."
• Dedupe: cross-province twins (same normalized title + price) collapse to one canonical row (dup status), keeping the more complete row (img > km > older). Merges are logged and reversible.
• Scoring: rubric tables ported from reference/motorcycle-hunt-rank.py (family tiers + extras + veto) — the score is Glenn's accumulated taste, computed at import, stored on each row. See RANKING.md.
• Idempotent: safe to re-run any time; also exposed as POST /api/import for manual triggers.

Scheduling 【updated 2026-11-10 — Glenn's call】

The nightly import is triggered by its own Hermes cron job — bike-tinder-import, 08:30 America/Halifax (after the 08:00 hunt), run script-only (zero LLM tokens) and silent on success. The hunt's own cron is untouched. The container additionally keeps an on-boot catch-up (runs once if the last import is >26 h stale). Every path calls the same idempotent importer and writes an import_runs row; GET /api/stats surfaces the last run. (The cron job is created now, parked; it wakes at first deploy — Phase 4.)

API surface (v1)

route	purpose
GET /api/health	liveness + last-import summary
GET /api/stats	counts: active / gone / dup / swiped; last import time
GET /api/deck?limit=	next cards (deck query below)
POST /api/swipe {id, action}	record or edit like / nope / maybe (upsert)
DELETE /api/swipe/{id}	clear a call (returns the card to the deck)
GET /api/list?filter=&q=&offset=	full list, filters + search
POST /api/import	manual import trigger (also the cron hook)
GET /img/{id}	cached image file, else 302 to source CDN

Deck query: status='active', not noped, and either no swipe or a maybe older than 7 days — ordered score DESC, first_seen DESC. Unscored rows sort below scored ones but still appear — nothing is hidden.

Images 【decided: hybrid mirror + hotlink】

• The importer pulls each new listing's card image once into data/img/ (~100 KB avg; ≈64 MB the first day, then ~20–30 MB/month). Cards reference the local mirror; a missing file falls back to a 302 hotlink of the source CDN (verified token-free and size-rewritable).
• Why: ad removal eventually kills CDN URLs (unverifiable until it happens — the mirror makes it a non-issue), phone-side speed, and "gone" cards stay viewable.
• Bootstrap (frozen 2026-11-10, P2): image URLs are enriched into the hunt state by the hunt toolchain's motorcycle-hunt-img.py — one detail-page fetch per listing, polite pacing; rescans keep curated imgs — then the importer carries them into the DB. The app mirrors files via app/imgfetch.py → data/img/{id}.webp; /img/{id} serves the file, else 302 to the source CDN, else the UI placeholder. Backfill scope at freeze: all 620 Kijiji rows missing an image. A placeholder renders for any listing still without a photo; it never holds up the deck. Mirrored files for rows gone >30 days are pruned.

Frontend

No-build SPA — vanilla JS + CSS, pointer-events drag, keyboard arrows on desktop (~300 lines). A small service worker caches the shell; the API is always live. Screens, gestures, and visual anatomy: see UI-SPEC.md.

Deployment (house constraints)

• Stack dir: /var/mnt/DATA/podman-stacks/bike-tinder/ — docker-compose.yml + Dockerfile (python:3.14-slim + fastapi/uvicorn/requests), volume ./data:/data, restart: unless-stopped.
• App networking: the app binds plain HTTP on the compose network and publishes localhost-only (127.0.0.1:9453:8000) — that's how the nightly trigger and local debugging reach it. No LAN exposure.
• Tailscale sidecar — baked into the stack (the repo carries no host-tailnet coupling; the app itself knows nothing about Tailscale):
  • a tailscale/tailscale container joins the tailnet as its own node and serves the app at https://biketinder.tailf35a34.ts.net (TCP 443 via tailscale serve → http://app:8000 over the compose network);
  • auth: a deploy-time auth key minted by Glenn, stored only in the stack's local .env — never committed, never pasted anywhere else;
  • deploy-time checks: the tailnet ACL must allow Glenn's devices to reach the biketinder node; first verification happens from his phone (the tailnet name can't be curled from this host).
• After adding files: chcon -Rt container_file_t /var/mnt/DATA/podman-stacks/bike-tinder.
• Systemd user unit podman-stack-bike-tinder.service, cloned from the machine's podman-stack-searxng.service pattern: Type=oneshot, RemainAfterExit=yes, Delegate=yes (keeps the stack out of the hermes-gateway cgroup so gateway restarts can't SIGKILL it); ExecStart/ExecStop = the venv podman-compose up -d / down.
• Backups: nightly sqlite3 .backup into data/backup/ right after import (the DB is the only mutable state; house rule). Restore = stop stack, swap the file, start.

Security

Single user; no auth in v1 — the app is reached only through the stack's own Tailscale node (plus a localhost-only port for triggers and debugging). An optional AUTH_TOKEN env var is noted as a one-line later hardening. One deploy-time secret exists in the stack: the Tailscale auth key (local .env only).

Risks & mitigations (from consultation)

risk	mitigation
Kijiji markup drift	parser keys on data-testid; failures → gates abort, deck stays last-known-good; crawler's fail/partial tracking monitors
Anti-bot on deep sweeps	ship at the proven 8-page window; grow only after two clean weeks
Image persistence after ad removal	local mirror (built-in)
Cross-province dedupe false merges	logged, reversible, low blast radius; canonical keeps the more complete row
Hunt pipeline dependency	fetched_at gate fails loud; vendored --direct parser escape hatch
Dealer blast dominance in score ties	tie-break first_seen DESC; "hide this seller" planned as a v1.1 prefs row
Rubric blindness (no family match)	rows still appear, sorted below scored ones — nothing hidden

Repo layout (target)

app/            importer.py · api.py · imgfetch.py · static/{index.html (single-file app), manifest.json, sw.js, icons/}
deploy/         docker-compose.yml (app + Tailscale sidecar) · Dockerfile · podman-stack-bike-tinder.service
reference/      copies of the hunt modules the app vendors from
data/           runtime state (gitignored)
docs/           this design set

Runbook (completed at Phase 4)

Deploy / update / rollback steps, log locations, import troubleshooting (gates), backup-restore drill, phone verification checklist.