106 lines
8.7 KiB
Markdown
106 lines
8.7 KiB
Markdown
# DESIGN — Bike-Tinder
|
||
|
||
Architecture for v1 (single source, single user, tailnet-only PWA). Written 2026-11-10 by Alex; engineering input from the Torres consultation (same date). Items marked 【decided】 are locked for v1.
|
||
|
||
## Components
|
||
|
||
| piece | what | where |
|
||
|---|---|---|
|
||
| Crawler | existing `motorcycle-hunt-scan.py` (**unchanged**) — sweeps Kijiji NS+NB nightly at 08:00 | Alex's pipeline (`~/.hermes/scripts/`) |
|
||
| Importer | reads the crawler's state JSONs, gates them, upserts into the app DB, scores, dedupes, queues images | `app/importer.py` |
|
||
| API | FastAPI + uvicorn — deck, swipes, list, stats, images | `app/api.py` |
|
||
| PWA | no-build single-page frontend (deck, list, matches, trends) | `app/static/` |
|
||
| DB | SQLite (WAL mode) | `data/bike-tinder.sqlite3` (stack volume) |
|
||
|
||
## Data flow
|
||
|
||
```
|
||
08:00 motorcycle-hunt crawl (existing, unchanged) → state JSONs: kijiji-ns.json, kijiji-nb.json
|
||
08:30 importer — script-only, zero LLM (Hermes cron `bike-tinder-import` → POST /api/import):
|
||
gates → upsert → dedupe → score → gone-marking → image queue
|
||
→ SQLite (WAL)
|
||
Deck phone ⇄ API ⇄ SQLite
|
||
↘ images: local mirror first, hotlink (302) fallback
|
||
```
|
||
|
||
## Import design 【decided】
|
||
|
||
- **Source of truth = the hunt's state files, not a re-crawl.** Rationale: zero duplicate load on Kijiji; inherits the crawler's retry / partial / last-known-good / absence logic instead of re-implementing it; the format is stable and self-describing (`ok`, `partial`, `fetched_at`, per-page counts).
|
||
- **Escape hatch:** the app vendors the Kijiji parser from `reference/motorcycle-hunt-scan.py` behind a `--direct` flag — if the hunt pipeline ever moves or dies, the importer can sweep Kijiji itself.
|
||
- **Gates (fail loud, keep last good):** each state file must have `ok == true`, `partial == false`, `fetched_at` within 26 h, and non-empty `items`. Any failure → abort that region, log to `import_runs`, leave the DB untouched.
|
||
- **Gone marking:** a listing absent from ≥2 consecutive *good* imports flips to `gone` (never deleted; reappearance flips it back). House discipline: a single miss — or any blocked page — never counts as "sold."
|
||
- **Dedupe:** cross-province twins (same normalized title + price) collapse to one canonical row (`dup` status), keeping the more complete row (img > km > older). Merges are logged and reversible.
|
||
- **Scoring:** rubric tables ported from `reference/motorcycle-hunt-rank.py` (family tiers + extras + veto) — the score is Glenn's accumulated taste, computed at import, stored on each row. See `RANKING.md`.
|
||
- **Idempotent:** safe to re-run any time; also exposed as `POST /api/import` for manual triggers.
|
||
|
||
## Scheduling 【updated 2026-11-10 — Glenn's call】
|
||
|
||
The nightly import is triggered by its own Hermes cron job — `bike-tinder-import`, **08:30 America/Halifax** (after the 08:00 hunt), run **script-only (zero LLM tokens)** and silent on success. The hunt's own cron is untouched. The container additionally keeps an **on-boot catch-up** (runs once if the last import is >26 h stale). Every path calls the same idempotent importer and writes an `import_runs` row; `GET /api/stats` surfaces the last run. (The cron job is created now, parked; it wakes at first deploy — Phase 4.)
|
||
|
||
## API surface (v1)
|
||
|
||
| route | purpose |
|
||
|---|---|
|
||
| `GET /api/health` | liveness + last-import summary |
|
||
| `GET /api/stats` | counts: active / gone / dup / swiped; last import time |
|
||
| `GET /api/deck?limit=` | next cards (deck query below) |
|
||
| `POST /api/swipe` `{id, action}` | record or edit like / nope / maybe (upsert) |
|
||
| `DELETE /api/swipe/{id}` | clear a call (returns the card to the deck) |
|
||
| `GET /api/list?filter=&q=&offset=` | full list, filters + search |
|
||
| `POST /api/import` | manual import trigger (also the cron hook) |
|
||
| `GET /img/{id}` | cached image file, else 302 to source CDN |
|
||
|
||
**Deck query:** `status='active'`, not noped, and either no swipe or a `maybe` older than 7 days — ordered `score DESC, first_seen DESC`. Unscored rows sort below scored ones but still appear — nothing is hidden.
|
||
|
||
## Images 【decided: hybrid mirror + hotlink】
|
||
|
||
- The importer pulls each new listing's card image once into `data/img/` (~100 KB avg; ≈64 MB the first day, then ~20–30 MB/month). Cards reference the local mirror; a missing file falls back to a 302 hotlink of the source CDN (verified token-free and size-rewritable).
|
||
- Why: ad removal eventually kills CDN URLs (unverifiable until it happens — the mirror makes it a non-issue), phone-side speed, and "gone" cards stay viewable.
|
||
- Bootstrap (frozen 2026-11-10, P2): image URLs are enriched into the hunt state by the hunt toolchain's `motorcycle-hunt-img.py` — one detail-page fetch per listing, polite pacing; rescans keep curated imgs — then the importer carries them into the DB. The app mirrors files via `app/imgfetch.py` → `data/img/{id}.webp`; `/img/{id}` serves the file, else 302 to the source CDN, else the UI placeholder. Backfill scope at freeze: all 620 Kijiji rows missing an image. A placeholder renders for any listing still without a photo; it never holds up the deck. Mirrored files for rows gone >30 days are pruned.
|
||
|
||
## Frontend
|
||
|
||
No-build SPA — vanilla JS + CSS, pointer-events drag, keyboard arrows on desktop (~300 lines). A small service worker caches the shell; the API is always live. Screens, gestures, and visual anatomy: see `UI-SPEC.md`.
|
||
|
||
## Deployment (house constraints)
|
||
|
||
- **Stack dir:** `/var/mnt/DATA/podman-stacks/bike-tinder/` — `docker-compose.yml` + `Dockerfile` (`python:3.14-slim` + fastapi/uvicorn/requests), volume `./data:/data`, `restart: unless-stopped`.
|
||
- **App networking:** the app binds plain HTTP on the compose network and publishes **localhost-only** (`127.0.0.1:9453:8000`) — that's how the nightly trigger and local debugging reach it. No LAN exposure.
|
||
- **Tailscale sidecar — baked into the stack** (the repo carries **no host-tailnet coupling**; the app itself knows nothing about Tailscale):
|
||
- a `tailscale/tailscale` container joins the tailnet as its own node and serves the app at **`https://biketinder.tailf35a34.ts.net`** (TCP 443 via `tailscale serve` → `http://app:8000` over the compose network);
|
||
- auth: a deploy-time auth key minted by Glenn, stored only in the stack's local `.env` — never committed, never pasted anywhere else;
|
||
- deploy-time checks: the tailnet ACL must allow Glenn's devices to reach the `biketinder` node; first verification happens from his phone (the tailnet name can't be curled from this host).
|
||
- After adding files: `chcon -Rt container_file_t /var/mnt/DATA/podman-stacks/bike-tinder`.
|
||
- **Systemd user unit** `podman-stack-bike-tinder.service`, cloned from the machine's `podman-stack-searxng.service` pattern: `Type=oneshot`, `RemainAfterExit=yes`, **`Delegate=yes`** (keeps the stack out of the hermes-gateway cgroup so gateway restarts can't SIGKILL it); ExecStart/ExecStop = the venv `podman-compose up -d` / `down`.
|
||
- **Backups:** nightly `sqlite3 .backup` into `data/backup/` right after import (the DB is the only mutable state; house rule). Restore = stop stack, swap the file, start.
|
||
|
||
## Security
|
||
|
||
Single user; **no auth in v1** — the app is reached only through the stack's own Tailscale node (plus a localhost-only port for triggers and debugging). An optional `AUTH_TOKEN` env var is noted as a one-line later hardening. One deploy-time secret exists in the stack: the Tailscale auth key (local `.env` only).
|
||
|
||
## Risks & mitigations (from consultation)
|
||
|
||
| risk | mitigation |
|
||
|---|---|
|
||
| Kijiji markup drift | parser keys on `data-testid`; failures → gates abort, deck stays last-known-good; crawler's fail/partial tracking monitors |
|
||
| Anti-bot on deep sweeps | ship at the proven 8-page window; grow only after two clean weeks |
|
||
| Image persistence after ad removal | local mirror (built-in) |
|
||
| Cross-province dedupe false merges | logged, reversible, low blast radius; canonical keeps the more complete row |
|
||
| Hunt pipeline dependency | `fetched_at` gate fails loud; vendored `--direct` parser escape hatch |
|
||
| Dealer blast dominance in score ties | tie-break `first_seen DESC`; "hide this seller" planned as a v1.1 prefs row |
|
||
| Rubric blindness (no family match) | rows still appear, sorted below scored ones — nothing hidden |
|
||
|
||
## Repo layout (target)
|
||
|
||
```
|
||
app/ importer.py · api.py · imgfetch.py · static/{index.html (single-file app), manifest.json, sw.js, icons/}
|
||
deploy/ docker-compose.yml (app + Tailscale sidecar) · Dockerfile · podman-stack-bike-tinder.service
|
||
reference/ copies of the hunt modules the app vendors from
|
||
data/ runtime state (gitignored)
|
||
docs/ this design set
|
||
```
|
||
|
||
## Runbook (completed at Phase 4)
|
||
|
||
Deploy / update / rollback steps, log locations, import troubleshooting (gates), backup-restore drill, phone verification checklist.
|