145 lines
13 KiB
Markdown
145 lines
13 KiB
Markdown
# DESIGN — Bike-Tinder
|
||
|
||
Architecture for v1 (single source, single user, tailnet-only PWA). Written 2026-11-10 by Alex; engineering input from the Torres consultation (same date). Items marked 【decided】 are locked for v1.
|
||
|
||
## Components
|
||
|
||
| piece | what | where |
|
||
|---|---|---|
|
||
| Crawler | existing `motorcycle-hunt-scan.py` (**unchanged**) — sweeps Kijiji NS+NB nightly at 08:00 | Alex's pipeline (`~/.hermes/scripts/`) |
|
||
| Importer | reads the crawler's state JSONs, gates them, upserts into the app DB, scores, dedupes, queues images | `app/importer.py` |
|
||
| API | FastAPI + uvicorn — deck, swipes, list, stats, images | `app/api.py` |
|
||
| PWA | no-build single-page frontend (deck, list, matches, trends) | `app/static/` |
|
||
| DB | SQLite (WAL mode) | `data/bike-tinder.sqlite3` (stack volume) |
|
||
|
||
## Data flow
|
||
|
||
```
|
||
08:00 motorcycle-hunt crawl (existing, unchanged) → state JSONs: kijiji-ns.json, kijiji-nb.json
|
||
08:30 importer — script-only, zero LLM (Hermes cron `bike-tinder-import` → POST /api/import):
|
||
gates → upsert → dedupe → score → gone-marking → image queue
|
||
→ SQLite (WAL)
|
||
Deck phone ⇄ API ⇄ SQLite
|
||
↘ images: local mirror first, hotlink (302) fallback
|
||
```
|
||
|
||
## Import design 【decided】
|
||
|
||
- **Source of truth = the hunt's state files, not a re-crawl.** Rationale: zero duplicate load on Kijiji; inherits the crawler's retry / partial / last-known-good / absence logic instead of re-implementing it; the format is stable and self-describing (`ok`, `partial`, `fetched_at`, per-page counts).
|
||
- **Escape hatch:** the app vendors the Kijiji parser from `reference/motorcycle-hunt-scan.py` behind a `--direct` flag — if the hunt pipeline ever moves or dies, the importer can sweep Kijiji itself.
|
||
- **Gates (fail loud, keep last good):** each state file must have `ok == true`, `partial == false`, `fetched_at` within 26 h, and non-empty `items`. Any failure → abort that region, log to `import_runs`, leave the DB untouched.
|
||
- **Gone marking:** a listing absent from ≥2 consecutive *good* imports flips to `gone` (never deleted; reappearance flips it back). House discipline: a single miss — or any blocked page — never counts as "sold."
|
||
- **Dedupe:** cross-province twins (same normalized title + price) collapse to one canonical row (`dup` status), keeping the more complete row (img > km > older). Merges are logged and reversible.
|
||
- **Scoring:** rubric tables ported from `reference/motorcycle-hunt-rank.py` (family tiers + extras + veto) — the score is Glenn's accumulated taste, computed at import, stored on each row. See `RANKING.md`.
|
||
- **Idempotent:** safe to re-run any time; also exposed as `POST /api/import` for manual triggers.
|
||
|
||
## Scheduling 【updated 2026-11-10 — Glenn's call】
|
||
|
||
The nightly import is triggered by its own Hermes cron job — `bike-tinder-import`, **08:30 America/Halifax** (after the 08:00 hunt), run **script-only (zero LLM tokens)** and silent on success. The hunt's own cron is untouched. The container additionally keeps an **on-boot catch-up** (runs once if the last import is >26 h stale). Every path calls the same idempotent importer and writes an `import_runs` row; `GET /api/stats` surfaces the last run. (The cron job is created now, parked; it wakes at first deploy — Phase 4.)
|
||
|
||
## API surface (v1)
|
||
|
||
| route | purpose |
|
||
|---|---|
|
||
| `GET /api/health` | liveness + last-import summary |
|
||
| `GET /api/stats` | counts: active / gone / dup / swiped; last import time |
|
||
| `GET /api/deck?limit=` | next cards (deck query below) |
|
||
| `POST /api/swipe` `{id, action}` | record or edit like / nope / maybe (upsert) |
|
||
| `DELETE /api/swipe/{id}` | clear a call (returns the card to the deck) |
|
||
| `GET /api/list?filter=&q=&offset=` | full list, filters + search |
|
||
| `POST /api/import` | manual import trigger (also the cron hook) |
|
||
| `GET /img/{id}` | cached image file, else 302 to source CDN |
|
||
|
||
**Deck query:** `status='active'`, not noped, and either no swipe or a `maybe` older than 7 days — ordered `score DESC, first_seen DESC`. Unscored rows sort below scored ones but still appear — nothing is hidden.
|
||
|
||
## Images 【decided: hybrid mirror + hotlink】
|
||
|
||
- The importer pulls each new listing's card image once into `data/img/` (~100 KB avg; ≈64 MB the first day, then ~20–30 MB/month). Cards reference the local mirror; a missing file falls back to a 302 hotlink of the source CDN (verified token-free and size-rewritable).
|
||
- Why: ad removal eventually kills CDN URLs (unverifiable until it happens — the mirror makes it a non-issue), phone-side speed, and "gone" cards stay viewable.
|
||
- Bootstrap (frozen 2026-11-10, P2): image URLs are enriched into the hunt state by the hunt toolchain's `motorcycle-hunt-img.py` — one detail-page fetch per listing, polite pacing; rescans keep curated imgs — then the importer carries them into the DB. The app mirrors files via `app/imgfetch.py` → `data/img/{id}.webp`; `/img/{id}` serves the file, else 302 to the source CDN, else the UI placeholder. Backfill scope at freeze: all 620 Kijiji rows missing an image. A placeholder renders for any listing still without a photo; it never holds up the deck. Mirrored files for rows gone >30 days are pruned.
|
||
|
||
## Frontend
|
||
|
||
No-build SPA — vanilla JS + CSS, pointer-events drag, keyboard arrows on desktop (~300 lines). A small service worker caches the shell; the API is always live. Screens, gestures, and visual anatomy: see `UI-SPEC.md`.
|
||
|
||
## Personalization architecture (Phase 3) — frozen 2026-11-10
|
||
|
||
| piece | what | where |
|
||
|---|---|---|
|
||
| Taste engine | deterministic points + breakdown from `swipe_events` + rubric | `app/taste.py` |
|
||
| Section engine | compose/forge/read sections, returns, wildcards, modes | `app/sections.py` |
|
||
| Agent pass | the ONLY LLM step — reads strikes, writes the profile; never in a request path | `app/agentpass.py` |
|
||
|
||
**The law:** no LLM in the swipe path, ever. The deck renders and the engine scores with zero network; the agent pass runs between sections (forge) or on schedule (P4 nightly), and can fail without breaking anything — compose falls back to the last profile.
|
||
|
||
**Forge flow (on-demand):** section exhausted → `POST /api/section/forge` → agent pass (mock/fallback when unconfigured; live when the key exists) writes `taste_profile` → section engine composes the next 25. **Strict mode:** forge refuses (409) until a fresh pass exists post-exhaustion — the nightly cron provides it at P4.
|
||
|
||
**API additions:** `GET /api/section` · `POST /api/section/forge` · `GET /api/trends` · `POST /api/agent/run`. Listing payloads gain `points`, `points_breakdown`, `tank`, `slot`, `round_next` (see UI-SPEC §10 and DATA-MODEL P3).
|
||
|
||
**Cost & keys:** each pass ≈ pennies (one compact prompt; house model default via `OPENROUTER_API_KEY`). The key lives in the host/stack env only — never the repo, never logs. "Pause" stops passes with all other spend.
|
||
|
||
## Deployment (house constraints)
|
||
|
||
- **Stack dir:** `/var/mnt/DATA/podman-stacks/bike-tinder/` — `docker-compose.yml` + `Dockerfile` (`python:3.14-slim` + fastapi/uvicorn/requests), volume `./data:/data`, `restart: unless-stopped`.
|
||
- **App networking:** the app binds plain HTTP on the compose network and publishes **localhost-only** (`127.0.0.1:9453:8000`) — that's how the nightly trigger and local debugging reach it. No LAN exposure.
|
||
- **Tailscale sidecar — baked into the stack** (the repo carries **no host-tailnet coupling**; the app itself knows nothing about Tailscale):
|
||
- a `tailscale/tailscale` container joins the tailnet as its own node and serves the app at **`https://biketinder.tailf35a34.ts.net`** (TCP 443 via `tailscale serve` → `http://app:8000` over the compose network);
|
||
- auth: a deploy-time auth key minted by Glenn, stored only in the stack's local `.env` — never committed, never pasted anywhere else;
|
||
- deploy-time checks: the tailnet ACL must allow Glenn's devices to reach the `biketinder` node; first verification happens from his phone (the tailnet name can't be curled from this host).
|
||
- After adding files: `chcon -Rt container_file_t /var/mnt/DATA/podman-stacks/bike-tinder`.
|
||
- **Systemd user unit** `podman-stack-bike-tinder.service`, cloned from the machine's `podman-stack-searxng.service` pattern: `Type=oneshot`, `RemainAfterExit=yes`, **`Delegate=yes`** (keeps the stack out of the hermes-gateway cgroup so gateway restarts can't SIGKILL it); ExecStart/ExecStop = the venv `podman-compose up -d` / `down`.
|
||
- **Backups:** nightly `sqlite3 .backup` into `data/backup/` right after import (the DB is the only mutable state; house rule). Restore = stop stack, swap the file, start.
|
||
|
||
## Security
|
||
|
||
Single user; **no auth in v1** — the app is reached only through the stack's own Tailscale node (plus a localhost-only port for triggers and debugging). An optional `AUTH_TOKEN` env var is noted as a one-line later hardening. One deploy-time secret exists in the stack: the Tailscale auth key (local `.env` only).
|
||
|
||
## Risks & mitigations (from consultation)
|
||
|
||
| risk | mitigation |
|
||
|---|---|
|
||
| Kijiji markup drift | parser keys on `data-testid`; failures → gates abort, deck stays last-known-good; crawler's fail/partial tracking monitors |
|
||
| Anti-bot on deep sweeps | ship at the proven 8-page window; grow only after two clean weeks |
|
||
| Image persistence after ad removal | local mirror (built-in) |
|
||
| Cross-province dedupe false merges | logged, reversible, low blast radius; canonical keeps the more complete row |
|
||
| Hunt pipeline dependency | `fetched_at` gate fails loud; vendored `--direct` parser escape hatch |
|
||
| Dealer blast dominance in score ties | tie-break `first_seen DESC`; "hide this seller" planned as a v1.1 prefs row |
|
||
| Rubric blindness (no family match) | rows still appear, sorted below scored ones — nothing hidden |
|
||
|
||
## Repo layout (target)
|
||
|
||
```
|
||
app/ importer.py · api.py · imgfetch.py · static/{index.html (single-file app), manifest.json, sw.js, icons/}
|
||
deploy/ docker-compose.yml (app + Tailscale sidecar) · Dockerfile · podman-stack-bike-tinder.service
|
||
reference/ copies of the hunt modules the app vendors from
|
||
data/ runtime state (gitignored)
|
||
docs/ this design set
|
||
```
|
||
|
||
## Runbook (Phase 4)
|
||
|
||
**Stack home:** `/var/mnt/DATA/podman-stacks/bike-tinder/` — a checkout of the `release` branch, plus stack-local `data/` (DB + image mirror + backups), `.env` (secrets, never committed) and `tailscale-state/` (node identity). Unit: `podman-stack-bike-tinder.service` (installed from `deploy/`).
|
||
|
||
**Deploy / update a cut:**
|
||
|
||
```sh
|
||
# from the repo: merge dev -> main, then fast-forward release
|
||
git -C /var/mnt/DATA/podman-stacks/bike-tinder pull --ff-only
|
||
podman-compose -f deploy/docker-compose.yml --env-file .env build
|
||
systemctl --user restart podman-stack-bike-tinder.service
|
||
```
|
||
|
||
Docs-only cuts stop after the `pull`. The unit is `oneshot` + `RemainAfterExit` with `Delegate=yes`, so the stack survives `hermes gateway restart` and reboots. The historical dev service (`bike-tinder-dev.service`) is retired at deploy; rollback happens through the stack, not that unit.
|
||
|
||
**Rollback:** check out the previous release SHA in the stack dir, rebuild, restart. The DB is never touched by code rollbacks (migrations are guarded and idempotent); if a migration ever needs unwinding, restore the last backup (below).
|
||
|
||
**Logs:** `podman logs bike-tinder-app`, `podman logs bike-tinder-ts`, `journalctl --user -u podman-stack-bike-tinder.service`.
|
||
|
||
**Import troubleshooting:** `POST /api/import` refuses when the hunt state is older than 26 h or missing — check `~/.hermes/scripts/motorcycle-hunt.d/*.json`. The nightly chain is the Hermes cron job `bike-tinder-import` (08:30 Atlantic: import → agent read → DB backup); it is silent on success by design — inspect its last run in the cron list if a morning looks stale. The container also runs an on-boot catch-up import when the last run is >26 h old.
|
||
|
||
**Backups:** nightly `data/backup/bike-tinder-YYYYMMDD-HHMM.sqlite3`, last 14 kept. Restore drill: stop the unit → remove `data/bike-tinder.sqlite3-wal`/`-shm` → copy the chosen backup over `data/bike-tinder.sqlite3` → start the unit → check `GET /api/stats`.
|
||
|
||
**Phone checklist:** open `https://biketinder.tailf35a34.ts.net` from a tailnet device → expect the deck; Add to Home Screen (PWA installs via the manifest); swipe once and confirm the decision shows in List. If the URL fails: check the sidecar (`podman logs bike-tinder-ts`), confirm `TS_AUTHKEY` in `.env`, and re-run `podman-compose -f deploy/docker-compose.yml --env-file .env up -d`.
|
||
|
||
**First-run sidecar (one-time):** mint a Tailscale auth key (admin console → Settings → Keys), put `TS_AUTHKEY=<key>` into the stack `.env`, then `podman-compose -f deploy/docker-compose.yml --env-file .env up -d tailscale`.
|