Files
Bike-Tinder/docs/DESIGN.md

102 lines
7.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# DESIGN — Bike-Tinder
Architecture for v1 (single source, single user, tailnet-only PWA). Written 2026-11-10 by Alex; engineering input from the Torres consultation (same date). Items marked 【decided】 are locked for v1.
## Components
| piece | what | where |
|---|---|---|
| Crawler | existing `motorcycle-hunt-scan.py` (**unchanged**) — sweeps Kijiji NS+NB nightly at 08:00 | Alex's pipeline (`~/.hermes/scripts/`) |
| Importer | reads the crawler's state JSONs, gates them, upserts into the app DB, scores, dedupes, queues images | `app/importer.py` |
| API | FastAPI + uvicorn — deck, swipes, list, stats, images | `app/api.py` |
| PWA | no-build single-page frontend (deck, list, matches, trends) | `app/static/` |
| DB | SQLite (WAL mode) | `data/bike-tinder.sqlite3` (stack volume) |
## Data flow
```
08:00 motorcycle-hunt crawl (existing, unchanged) → state JSONs: kijiji-ns.json, kijiji-nb.json
08:15 importer (script-only, zero LLM):
gates → upsert → dedupe → score → gone-marking → image queue
→ SQLite (WAL)
Deck phone ⇄ API ⇄ SQLite
↘ images: local mirror first, hotlink (302) fallback
```
## Import design 【decided】
- **Source of truth = the hunt's state files, not a re-crawl.** Rationale: zero duplicate load on Kijiji; inherits the crawler's retry / partial / last-known-good / absence logic instead of re-implementing it; the format is stable and self-describing (`ok`, `partial`, `fetched_at`, per-page counts).
- **Escape hatch:** the app vendors the Kijiji parser from `reference/motorcycle-hunt-scan.py` behind a `--direct` flag — if the hunt pipeline ever moves or dies, the importer can sweep Kijiji itself.
- **Gates (fail loud, keep last good):** each state file must have `ok == true`, `partial == false`, `fetched_at` within 26 h, and non-empty `items`. Any failure → abort that region, log to `import_runs`, leave the DB untouched.
- **Gone marking:** a listing absent from ≥2 consecutive *good* imports flips to `gone` (never deleted; reappearance flips it back). House discipline: a single miss — or any blocked page — never counts as "sold."
- **Dedupe:** cross-province twins (same normalized title + price) collapse to one canonical row (`dup` status), keeping the more complete row (img > km > older). Merges are logged and reversible.
- **Scoring:** rubric tables ported from `reference/motorcycle-hunt-rank.py` (family tiers + extras + veto) — the score is Glenn's accumulated taste, computed at import, stored on each row. See `RANKING.md`.
- **Idempotent:** safe to re-run any time; also exposed as `POST /api/import` for manual triggers.
## Scheduling 【decided】
An in-container daily task (asyncio, 08:15 America/Halifax) inside the API process, with on-boot catch-up for missed runs. No host crontab is involved (the host user crontab is empty; the 08:00 crawl lives in Hermes cron — the app depends on neither). Every run writes an `import_runs` row; `GET /api/stats` surfaces the last run.
## API surface (v1)
| route | purpose |
|---|---|
| `GET /api/health` | liveness + last-import summary |
| `GET /api/stats` | counts: active / gone / dup / swiped; last import time |
| `GET /api/deck?limit=` | next cards (deck query below) |
| `POST /api/swipe` `{id, action}` | record or edit like / nope / maybe (upsert) |
| `DELETE /api/swipe/{id}` | clear a call (returns the card to the deck) |
| `GET /api/list?filter=&q=&offset=` | full list, filters + search |
| `POST /api/import` | manual import trigger |
| `GET /img/{id}` | cached image file, else 302 to source CDN |
**Deck query:** `status='active'`, not noped, and either no swipe or a `maybe` older than 7 days — ordered `score DESC, first_seen DESC`. Unscored rows sort below scored ones but still appear — nothing is hidden.
## Images 【decided: hybrid mirror + hotlink】
- The importer pulls each new listing's card image once into `data/img/` (~100 KB avg; ≈64 MB the first day, then ~20–30 MB/month). Cards reference the local mirror; a missing file falls back to a 302 hotlink of the source CDN (verified token-free and size-rewritable).
- Why: ad removal eventually kills CDN URLs (unverifiable until it happens — the mirror makes it a non-issue), phone-side speed, and "gone" cards stay viewable.
- Bootstrap: today's state carries `img` on only ~3% of rows — the first import does a one-time headless re-sweep (8 pages/province via the vendored parser — 16 fetches, not 640 detail pages) to harvest card images. A placeholder renders for any listing still without a photo; it never holds up the deck. Mirrored files for rows gone >30 days are pruned.
## Frontend
No-build SPA — vanilla JS + CSS, pointer-events drag, keyboard arrows on desktop (~300 lines). A small service worker caches the shell; the API is always live. Screens, gestures, and visual anatomy: see `UI-SPEC.md`.
## Deployment (house constraints)
- **Stack dir:** `/var/mnt/DATA/podman-stacks/bike-tinder/` — `docker-compose.yml` + `Dockerfile` (`python:3.14-slim` + fastapi/uvicorn/requests), volume `./data:/data`, port `127.0.0.1:9453:8000`, `restart: unless-stopped`.
- After adding files: `chcon -Rt container_file_t /var/mnt/DATA/podman-stacks/bike-tinder`.
- **Systemd user unit** `podman-stack-bike-tinder.service`, cloned from the machine's `podman-stack-searxng.service` pattern: `Type=oneshot`, `RemainAfterExit=yes`, **`Delegate=yes`** (keeps the stack out of the hermes-gateway cgroup so gateway restarts can't SIGKILL it); ExecStart/ExecStop = the venv `podman-compose up -d` / `down`.
- **Tailnet:** `tailscale serve --bg 9453` → reachable at `http://rotarran.tailf35a34.ts.net:9453`, tailnet-only. The tailnet name can't be curled from this host — first deploy verification happens from Glenn's phone.
- **Backups:** nightly `sqlite3 .backup` into `data/backup/` right after import (the DB is the only mutable state; house rule). Restore = stop stack, swap the file, start.
## Security
Single user; **no auth in v1** — localhost bind + tailnet-only reachability. An optional `AUTH_TOKEN` env var is noted as a one-line later hardening. No secrets are required in the stack.
## Risks & mitigations (from consultation)
| risk | mitigation |
|---|---|
| Kijiji markup drift | parser keys on `data-testid`; failures → gates abort, deck stays last-known-good; crawler's fail/partial tracking monitors |
| Anti-bot on deep sweeps | ship at the proven 8-page window; grow only after two clean weeks |
| Image persistence after ad removal | local mirror (built-in) |
| Cross-province dedupe false merges | logged, reversible, low blast radius; canonical keeps the more complete row |
| Hunt pipeline dependency | `fetched_at` gate fails loud; vendored `--direct` parser escape hatch |
| Dealer blast dominance in score ties | tie-break `first_seen DESC`; "hide this seller" planned as a v1.1 prefs row |
| Rubric blindness (no family match) | rows still appear, sorted below scored ones — nothing hidden |
## Repo layout (target)
```
app/ importer.py · api.py · static/{index.html,app.js,app.css,manifest.json,sw.js,icons}
deploy/ docker-compose.yml · Dockerfile · podman-stack-bike-tinder.service
reference/ copies of the hunt modules the app vendors from
data/ runtime state (gitignored)
docs/ this design set
```
## Runbook (completed at Phase 4)
Deploy / update / rollback steps, log locations, import troubleshooting (gates), backup-restore drill, phone verification checklist.