Bike-Tinder/docs/DESIGN.md

DESIGN — Bike-Tinder

Architecture for v1 (single source, single user, tailnet-only PWA). Written 2026-11-10 by Alex; engineering input from the Torres consultation (same date). Items marked 【decided】 are locked for v1.

Components

piece	what	where
Crawler	existing motorcycle-hunt-scan.py (unchanged) — sweeps Kijiji NS+NB nightly at 08:00	Alex's pipeline (~/.hermes/scripts/)
Importer	reads the crawler's state JSONs, gates them, upserts into the app DB, scores, dedupes, queues images	app/importer.py
API	FastAPI + uvicorn — deck, swipes, list, stats, images	app/api.py
PWA	no-build single-page frontend (deck, list, matches, trends)	app/static/
DB	SQLite (WAL mode)	data/bike-tinder.sqlite3 (stack volume)

Data flow

08:00  motorcycle-hunt crawl (existing, unchanged) → state JSONs: kijiji-ns.json, kijiji-nb.json
08:30  importer — script-only, zero LLM (Hermes cron `bike-tinder-import` → POST /api/import):
         gates → upsert → dedupe → score → gone-marking → image queue
       → SQLite (WAL)
Deck   phone ⇄ API ⇄ SQLite
             ↘ images: local mirror first, hotlink (302) fallback

Import design 【decided】

• Source of truth = the hunt's state files, not a re-crawl. Rationale: zero duplicate load on Kijiji; inherits the crawler's retry / partial / last-known-good / absence logic instead of re-implementing it; the format is stable and self-describing (ok, partial, fetched_at, per-page counts).
• Escape hatch: the app vendors the Kijiji parser from reference/motorcycle-hunt-scan.py behind a --direct flag — if the hunt pipeline ever moves or dies, the importer can sweep Kijiji itself.
• Gates (fail loud, keep last good): each state file must have ok == true, partial == false, fetched_at within 26 h, and non-empty items. Any failure → abort that region, log to import_runs, leave the DB untouched.
• Gone marking: a listing absent from ≥2 consecutive good imports flips to gone (never deleted; reappearance flips it back). House discipline: a single miss — or any blocked page — never counts as "sold."
• Dedupe: cross-province twins (same normalized title + price) collapse to one canonical row (dup status), keeping the more complete row (img > km > older). Merges are logged and reversible.
• Scoring: rubric tables ported from reference/motorcycle-hunt-rank.py (family tiers + extras + veto) — the score is Glenn's accumulated taste, computed at import, stored on each row. See RANKING.md.
• Idempotent: safe to re-run any time; also exposed as POST /api/import for manual triggers.

Scheduling 【updated 2026-11-10 — Glenn's call】

The nightly import is triggered by its own Hermes cron job — bike-tinder-import, 08:30 America/Halifax (after the 08:00 hunt), run script-only (zero LLM tokens) and silent on success. The hunt's own cron is untouched. The container additionally keeps an on-boot catch-up (runs once if the last import is >26 h stale). Every path calls the same idempotent importer and writes an import_runs row; GET /api/stats surfaces the last run. (The cron job is created now, parked; it wakes at first deploy — Phase 4.)

API surface (v1)

route	purpose
GET /api/health	liveness + last-import summary
GET /api/stats	counts: active / gone / dup / swiped; last import time
GET /api/deck?limit=	next cards (deck query below)
POST /api/swipe {id, action}	record or edit like / nope / maybe (upsert)
DELETE /api/swipe/{id}	clear a call (returns the card to the deck)
GET /api/list?filter=&q=&offset=	full list, filters + search
POST /api/import	manual import trigger (also the cron hook)
GET /img/{id}	cached image file, else 302 to source CDN

Deck query: status='active', not noped, and either no swipe or a maybe older than 7 days — ordered score DESC, first_seen DESC. Unscored rows sort below scored ones but still appear — nothing is hidden.

Images 【decided: hybrid mirror + hotlink】

• The importer pulls each new listing's card image once into data/img/ (~100 KB avg; ≈64 MB the first day, then ~20–30 MB/month). Cards reference the local mirror; a missing file falls back to a 302 hotlink of the source CDN (verified token-free and size-rewritable).
• Why: ad removal eventually kills CDN URLs (unverifiable until it happens — the mirror makes it a non-issue), phone-side speed, and "gone" cards stay viewable.
• Bootstrap (frozen 2026-11-10, P2): image URLs are enriched into the hunt state by the hunt toolchain's motorcycle-hunt-img.py — one detail-page fetch per listing, polite pacing; rescans keep curated imgs — then the importer carries them into the DB. The app mirrors files via app/imgfetch.py → data/img/{id}.webp; /img/{id} serves the file, else 302 to the source CDN, else the UI placeholder. Backfill scope at freeze: all 620 Kijiji rows missing an image. A placeholder renders for any listing still without a photo; it never holds up the deck. Mirrored files for rows gone >30 days are pruned.

Frontend

No-build SPA — vanilla JS + CSS, pointer-events drag, keyboard arrows on desktop (~300 lines). A small service worker caches the shell; the API is always live. Screens, gestures, and visual anatomy: see UI-SPEC.md.

Personalization architecture (Phase 3) — frozen 2026-11-10

piece	what	where
Taste engine	deterministic points + breakdown from swipe_events + rubric	app/taste.py
Section engine	compose/forge/read sections, returns, wildcards, modes	app/sections.py
Agent pass	the ONLY LLM step — reads strikes, writes the profile; never in a request path	app/agentpass.py

The law: no LLM in the swipe path, ever. The deck renders and the engine scores with zero network; the agent pass runs between sections (forge) or on schedule (P4 nightly), and can fail without breaking anything — compose falls back to the last profile.

Forge flow (on-demand): section exhausted → POST /api/section/forge → agent pass (mock/fallback when unconfigured; live when the key exists) writes taste_profile → section engine composes the next 25. Strict mode: forge refuses (409) until a fresh pass exists post-exhaustion — the nightly cron provides it at P4.

API additions: GET /api/section · POST /api/section/forge · GET /api/trends · POST /api/agent/run. Listing payloads gain points, points_breakdown, tank, slot, round_next (see UI-SPEC §10 and DATA-MODEL P3).

Cost & keys: each pass ≈ pennies (one compact prompt; house model default via OPENROUTER_API_KEY). The key lives in the host/stack env only — never the repo, never logs. "Pause" stops passes with all other spend.

Deployment (house constraints)

• Stack dir: /var/mnt/DATA/podman-stacks/bike-tinder/ — docker-compose.yml + Dockerfile (python:3.14-slim + fastapi/uvicorn/requests), volume ./data:/data, restart: unless-stopped.
• App networking: the app binds plain HTTP on the compose network and publishes localhost-only (127.0.0.1:9453:8000) — that's how the nightly trigger and local debugging reach it. No LAN exposure.
• Tailscale sidecar — baked into the stack (the repo carries no host-tailnet coupling; the app itself knows nothing about Tailscale):
  • a tailscale/tailscale container joins the tailnet as its own node and serves the app at https://biketinder.tailf35a34.ts.net (TCP 443 via tailscale serve → http://app:8000 over the compose network);
  • auth: a deploy-time auth key minted by Glenn, stored only in the stack's local .env — never committed, never pasted anywhere else;
  • deploy-time checks: the tailnet ACL must allow Glenn's devices to reach the biketinder node; first verification happens from his phone (the tailnet name can't be curled from this host).
• After adding files: chcon -Rt container_file_t /var/mnt/DATA/podman-stacks/bike-tinder.
• Systemd user unit podman-stack-bike-tinder.service, cloned from the machine's podman-stack-searxng.service pattern: Type=oneshot, RemainAfterExit=yes, Delegate=yes (keeps the stack out of the hermes-gateway cgroup so gateway restarts can't SIGKILL it); ExecStart/ExecStop = the venv podman-compose up -d / down.
• Backups: nightly sqlite3 .backup into data/backup/ right after import (the DB is the only mutable state; house rule). Restore = stop stack, swap the file, start.

Security

Single user; no auth in v1 — the app is reached only through the stack's own Tailscale node (plus a localhost-only port for triggers and debugging). An optional AUTH_TOKEN env var is noted as a one-line later hardening. One deploy-time secret exists in the stack: the Tailscale auth key (local .env only).

Risks & mitigations (from consultation)

risk	mitigation
Kijiji markup drift	parser keys on data-testid; failures → gates abort, deck stays last-known-good; crawler's fail/partial tracking monitors
Anti-bot on deep sweeps	ship at the proven 8-page window; grow only after two clean weeks
Image persistence after ad removal	local mirror (built-in)
Cross-province dedupe false merges	logged, reversible, low blast radius; canonical keeps the more complete row
Hunt pipeline dependency	fetched_at gate fails loud; vendored --direct parser escape hatch
Dealer blast dominance in score ties	tie-break first_seen DESC; "hide this seller" planned as a v1.1 prefs row
Rubric blindness (no family match)	rows still appear, sorted below scored ones — nothing hidden

Repo layout (target)

app/            importer.py · api.py · imgfetch.py · static/{index.html (single-file app), manifest.json, sw.js, icons/}
deploy/         docker-compose.yml (app + Tailscale sidecar) · Dockerfile · podman-stack-bike-tinder.service
reference/      copies of the hunt modules the app vendors from
data/           runtime state (gitignored)
docs/           this design set

Runbook (Phase 4)

Stack home: /var/mnt/DATA/podman-stacks/bike-tinder/ — a checkout of the release branch, plus stack-local data/ (DB + image mirror + backups), .env (secrets, never committed) and tailscale-state/ (node identity). Unit: podman-stack-bike-tinder.service (installed from deploy/).

Deploy / update a cut:

# from the repo: merge dev -> main, then fast-forward release
git -C /var/mnt/DATA/podman-stacks/bike-tinder pull --ff-only
podman-compose -f deploy/docker-compose.yml --env-file .env build
systemctl --user restart podman-stack-bike-tinder.service

Docs-only cuts stop after the pull. The unit is oneshot + RemainAfterExit with Delegate=yes, so the stack survives hermes gateway restart and reboots. The historical dev service (bike-tinder-dev.service) is retired at deploy; rollback happens through the stack, not that unit.

Rollback: check out the previous release SHA in the stack dir, rebuild, restart. The DB is never touched by code rollbacks (migrations are guarded and idempotent); if a migration ever needs unwinding, restore the last backup (below).

Logs: podman logs bike-tinder-app, podman logs bike-tinder-ts, journalctl --user -u podman-stack-bike-tinder.service.

Import troubleshooting: POST /api/import refuses when the hunt state is older than 26 h or missing — check ~/.hermes/scripts/motorcycle-hunt.d/*.json. The nightly chain is the Hermes cron job bike-tinder-import (08:30 Atlantic: import → agent read → DB backup); it is silent on success by design — inspect its last run in the cron list if a morning looks stale. The container also runs an on-boot catch-up import when the last run is >26 h old.

Backups: nightly data/backup/bike-tinder-YYYYMMDD-HHMM.sqlite3, last 14 kept. Restore drill: stop the unit → remove data/bike-tinder.sqlite3-wal/-shm → copy the chosen backup over data/bike-tinder.sqlite3 → start the unit → check GET /api/stats.

Phone checklist: open https://biketinder.tailf35a34.ts.net from a tailnet device → expect the deck; Add to Home Screen (PWA installs via the manifest); swipe once and confirm the decision shows in List. If the URL fails: check the sidecar (podman logs bike-tinder-ts), confirm TS_AUTHKEY in .env, and re-run podman-compose -f deploy/docker-compose.yml --env-file .env up -d.

First-run sidecar (one-time): mint a Tailscale auth key (admin console → Settings → Keys), put TS_AUTHKEY=<key> into the stack .env, then podman-compose -f deploy/docker-compose.yml --env-file .env up -d tailscale.