docs: add SOURCES, DESIGN, DATA-MODEL; record v1 decisions (Kijiji source, rank-all deck, badge-only feed); vendor hunt reference modules

This commit is contained in:
AlexanderRozhenko
2026-11-10 17:48:32 -04:00
parent fc3aad1e93
commit 18bf31f361
10 changed files with 932 additions and 8 deletions

95
docs/DATA-MODEL.md Normal file
View File

@@ -0,0 +1,95 @@
# DATA-MODEL — Bike-Tinder
SQLite, WAL mode, `busy_timeout`. One file: `data/bike-tinder.sqlite3`. Schema as consulted (Torres, 2026-11-10), locked for v1.
## Tables
```sql
listings (
id INTEGER PRIMARY KEY AUTOINCREMENT,
region TEXT NOT NULL CHECK (region IN ('ns','nb')),
src_id TEXT NOT NULL, -- source ad id (Kijiji id in v1)
title TEXT NOT NULL,
price_display TEXT, -- "$5,000"; "Please Contact"
price_num INTEGER, -- NULL when unpackable
km INTEGER, -- NULL when not stated (~2/3 of cards)
location TEXT,
url TEXT NOT NULL, -- source ad page
img_url TEXT, -- source CDN card image
desc TEXT, -- card description snippet
family TEXT, -- top rubric family label (taste signal)
score INTEGER NOT NULL DEFAULT 0, -- rubric score, recomputed each import
status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active','gone','dup')),
dup_of INTEGER, -- canonical row id when status='dup'
first_seen TEXT, last_seen TEXT,
created_at TEXT DEFAULT (datetime('now')), updated_at TEXT,
UNIQUE (region, src_id)
);
CREATE INDEX idx_listings_deck ON listings (status, score DESC);
CREATE INDEX idx_listings_dup ON listings (dup_of) WHERE status = 'dup';
swipes (
listing_id INTEGER PRIMARY KEY REFERENCES listings(id) ON DELETE CASCADE,
action TEXT NOT NULL CHECK (action IN ('like','nope','maybe')),
created_at TEXT DEFAULT (datetime('now')),
updated_at TEXT
);
import_runs (
id, started_at, ok, partial,
src_fetched_ns, src_fetched_nb,
total, new, gone, dup, error
); -- one observability row per nightly run
prefs (
key TEXT PRIMARY KEY, value TEXT
); -- deck_size, depth, etc. — tiny
```
Notes:
- One row per swipe; a new swipe **upserts** (a re-swiped card overwrites its action). The latest action is the truth; edits from the list view are upserts. Append-history is deliberately not kept in v1.
- `import_runs` is the dashboard's evidence trail — every run writes one row, success or failure.
## State semantics
- **new** — `active` with no `swipes` row → the deck.
- **seen** — has any swipe row.
- **gone** — was `active`, then absent from ≥2 consecutive *good* imports → `status='gone'` (kept forever, swipes preserved; reappearance flips it back to `active`). Single misses and failed/partial imports never mark gone.
- **dup** — the second province's twin of the same ad (dedupe below) → hidden from the deck, visible under its canonical row in the list.
- **nope** — never re-enters the deck unless cleared from the list (`DELETE` swipe → back to unswiped).
- **maybe** — soft skip: the deck query re-admits `maybe` rows after **7 days** (computed at query time — no cron needed). Editing a maybe to like/nope is just an upsert.
- **like** — pins to the Liked ("matches") view.
## Deck query (one statement)
```sql
SELECT * FROM listings
WHERE status = 'active'
AND id NOT IN (SELECT listing_id FROM swipes WHERE action = 'nope')
AND id NOT IN (SELECT listing_id FROM swipes WHERE action IN ('like','maybe')
AND (action = 'like' OR updated_at > datetime('now','-7 days')))
ORDER BY score DESC, first_seen DESC
LIMIT :deck_size;
```
(Implementers: a `LEFT JOIN swipes` variant is equivalent and cheaper — the intent above is the contract. Unscored rows sort below scored ones; nothing is hidden.)
## Dedupe (Kijiji cross-province)
- Group imports by `(normalized_title, price_num)`; when a group spans both provinces, keep the more-complete row canonical (img present → km present → older row) and mark the twin(s) `status='dup', dup_of=canonical`. The twin's URL is preserved on the canonical row (extra note field / desc suffix).
- Today's snapshot: 58/320 NS rows are twins of NB rows (dealers cross-post one ad to both provinces).
- False merges are possible (same title + price, different bike) but rare; every merge is logged in `import_runs` and reversible in the DB. Honest v1 tradeoff.
## Import gates (must pass or the run aborts)
1. State file `ok == true` and `partial == false`.
2. `fetched_at` within 26 hours.
3. Non-empty `items`.
4. On failure: log, leave the DB untouched (deck stays last-known-good).
## Maintenance
- **Backup:** nightly `sqlite3 .backup` to `data/backup/` immediately after import (house rule; the DB is the app's only mutable state).
- **Prune:** mirrored image files for rows `gone` >30 days; rows themselves are never auto-deleted in v1.
- **VACUUM:** optional; the DB is KV-sized (a few thousand rows).

101
docs/DESIGN.md Normal file
View File

@@ -0,0 +1,101 @@
# DESIGN — Bike-Tinder
Architecture for v1 (single source, single user, tailnet-only PWA). Written 2026-11-10 by Alex; engineering input from the Torres consultation (same date). Items marked 【decided】 are locked for v1.
## Components
| piece | what | where |
|---|---|---|
| Crawler | existing `motorcycle-hunt-scan.py` (**unchanged**) — sweeps Kijiji NS+NB nightly at 08:00 | Alex's pipeline (`~/.hermes/scripts/`) |
| Importer | reads the crawler's state JSONs, gates them, upserts into the app DB, scores, dedupes, queues images | `app/importer.py` |
| API | FastAPI + uvicorn — deck, swipes, list, stats, images | `app/api.py` |
| PWA | no-build single-page frontend (deck, list, matches, trends) | `app/static/` |
| DB | SQLite (WAL mode) | `data/bike-tinder.sqlite3` (stack volume) |
## Data flow
```
08:00 motorcycle-hunt crawl (existing, unchanged) → state JSONs: kijiji-ns.json, kijiji-nb.json
08:15 importer (script-only, zero LLM):
gates → upsert → dedupe → score → gone-marking → image queue
→ SQLite (WAL)
Deck phone ⇄ API ⇄ SQLite
↘ images: local mirror first, hotlink (302) fallback
```
## Import design 【decided】
- **Source of truth = the hunt's state files, not a re-crawl.** Rationale: zero duplicate load on Kijiji; inherits the crawler's retry / partial / last-known-good / absence logic instead of re-implementing it; the format is stable and self-describing (`ok`, `partial`, `fetched_at`, per-page counts).
- **Escape hatch:** the app vendors the Kijiji parser from `reference/motorcycle-hunt-scan.py` behind a `--direct` flag — if the hunt pipeline ever moves or dies, the importer can sweep Kijiji itself.
- **Gates (fail loud, keep last good):** each state file must have `ok == true`, `partial == false`, `fetched_at` within 26 h, and non-empty `items`. Any failure → abort that region, log to `import_runs`, leave the DB untouched.
- **Gone marking:** a listing absent from ≥2 consecutive *good* imports flips to `gone` (never deleted; reappearance flips it back). House discipline: a single miss — or any blocked page — never counts as "sold."
- **Dedupe:** cross-province twins (same normalized title + price) collapse to one canonical row (`dup` status), keeping the more complete row (img > km > older). Merges are logged and reversible.
- **Scoring:** rubric tables ported from `reference/motorcycle-hunt-rank.py` (family tiers + extras + veto) — the score is Glenn's accumulated taste, computed at import, stored on each row. See `RANKING.md`.
- **Idempotent:** safe to re-run any time; also exposed as `POST /api/import` for manual triggers.
## Scheduling 【decided】
An in-container daily task (asyncio, 08:15 America/Halifax) inside the API process, with on-boot catch-up for missed runs. No host crontab is involved (the host user crontab is empty; the 08:00 crawl lives in Hermes cron — the app depends on neither). Every run writes an `import_runs` row; `GET /api/stats` surfaces the last run.
## API surface (v1)
| route | purpose |
|---|---|
| `GET /api/health` | liveness + last-import summary |
| `GET /api/stats` | counts: active / gone / dup / swiped; last import time |
| `GET /api/deck?limit=` | next cards (deck query below) |
| `POST /api/swipe` `{id, action}` | record or edit like / nope / maybe (upsert) |
| `DELETE /api/swipe/{id}` | clear a call (returns the card to the deck) |
| `GET /api/list?filter=&q=&offset=` | full list, filters + search |
| `POST /api/import` | manual import trigger |
| `GET /img/{id}` | cached image file, else 302 to source CDN |
**Deck query:** `status='active'`, not noped, and either no swipe or a `maybe` older than 7 days — ordered `score DESC, first_seen DESC`. Unscored rows sort below scored ones but still appear — nothing is hidden.
## Images 【decided: hybrid mirror + hotlink】
- The importer pulls each new listing's card image once into `data/img/` (~100 KB avg; ≈64 MB the first day, then ~20–30 MB/month). Cards reference the local mirror; a missing file falls back to a 302 hotlink of the source CDN (verified token-free and size-rewritable).
- Why: ad removal eventually kills CDN URLs (unverifiable until it happens — the mirror makes it a non-issue), phone-side speed, and "gone" cards stay viewable.
- Bootstrap: today's state carries `img` on only ~3% of rows — the first import does a one-time headless re-sweep (8 pages/province via the vendored parser — 16 fetches, not 640 detail pages) to harvest card images. A placeholder renders for any listing still without a photo; it never holds up the deck. Mirrored files for rows gone >30 days are pruned.
## Frontend
No-build SPA — vanilla JS + CSS, pointer-events drag, keyboard arrows on desktop (~300 lines). A small service worker caches the shell; the API is always live. Screens, gestures, and visual anatomy: see `UI-SPEC.md`.
## Deployment (house constraints)
- **Stack dir:** `/var/mnt/DATA/podman-stacks/bike-tinder/` — `docker-compose.yml` + `Dockerfile` (`python:3.14-slim` + fastapi/uvicorn/requests), volume `./data:/data`, port `127.0.0.1:9453:8000`, `restart: unless-stopped`.
- After adding files: `chcon -Rt container_file_t /var/mnt/DATA/podman-stacks/bike-tinder`.
- **Systemd user unit** `podman-stack-bike-tinder.service`, cloned from the machine's `podman-stack-searxng.service` pattern: `Type=oneshot`, `RemainAfterExit=yes`, **`Delegate=yes`** (keeps the stack out of the hermes-gateway cgroup so gateway restarts can't SIGKILL it); ExecStart/ExecStop = the venv `podman-compose up -d` / `down`.
- **Tailnet:** `tailscale serve --bg 9453` → reachable at `http://rotarran.tailf35a34.ts.net:9453`, tailnet-only. The tailnet name can't be curled from this host — first deploy verification happens from Glenn's phone.
- **Backups:** nightly `sqlite3 .backup` into `data/backup/` right after import (the DB is the only mutable state; house rule). Restore = stop stack, swap the file, start.
## Security
Single user; **no auth in v1** — localhost bind + tailnet-only reachability. An optional `AUTH_TOKEN` env var is noted as a one-line later hardening. No secrets are required in the stack.
## Risks & mitigations (from consultation)
| risk | mitigation |
|---|---|
| Kijiji markup drift | parser keys on `data-testid`; failures → gates abort, deck stays last-known-good; crawler's fail/partial tracking monitors |
| Anti-bot on deep sweeps | ship at the proven 8-page window; grow only after two clean weeks |
| Image persistence after ad removal | local mirror (built-in) |
| Cross-province dedupe false merges | logged, reversible, low blast radius; canonical keeps the more complete row |
| Hunt pipeline dependency | `fetched_at` gate fails loud; vendored `--direct` parser escape hatch |
| Dealer blast dominance in score ties | tie-break `first_seen DESC`; "hide this seller" planned as a v1.1 prefs row |
| Rubric blindness (no family match) | rows still appear, sorted below scored ones — nothing hidden |
## Repo layout (target)
```
app/ importer.py · api.py · static/{index.html,app.js,app.css,manifest.json,sw.js,icons}
deploy/ docker-compose.yml · Dockerfile · podman-stack-bike-tinder.service
reference/ copies of the hunt modules the app vendors from
data/ runtime state (gitignored)
docs/ this design set
```
## Runbook (completed at Phase 4)
Deploy / update / rollback steps, log locations, import troubleshooting (gates), backup-restore drill, phone verification checklist.

View File

@@ -10,8 +10,9 @@ A single-source, single-user, tailnet-only Tinder-style PWA on the Rotarran: nig
- [x] Engineering consultation (Torres) → memo
- [x] Design docs in this repo: SOURCES, DESIGN, DATA-MODEL, RANKING, UI-SPEC
- [ ] Glenn's read on source choice + docs
- **Exit criteria:** docs merged to `main`; source decision recorded.
- [x] Glenn's calls (2026-11-10): v1 source = **Kijiji** · deck = show all, taste-ranked · new-bike feed = in-app badge only
- [ ] Glenn's skim of the docs — build gate opens after
- **Exit criteria:** docs merged to `main` (done 2026-11-10); source decision recorded in `SOURCES.md`.
## Phase 1 — Data spine
@@ -53,5 +54,5 @@ A single-source, single-user, tailnet-only Tinder-style PWA on the Rotarran: nig
## Open decisions
- v1 source — pending consultation evidence (see `docs/SOURCES.md`)
- v1 source — **resolved 2026-11-10: Kijiji** (see `docs/SOURCES.md`)
- Mockup direction — Glenn picks at Phase 2 start

View File

@@ -1,6 +1,6 @@
# RANKING — Bike-Tinder
The deck is ordered by predicted taste. Two stages: a cold start (the hunt's existing rubric), then a small learned model that never becomes a black box.
The deck is ordered by predicted taste. Three rungs, in order: the hunt's existing rubric (v1), a cheap explainable boost from swipe clusters (v1.x), then a small learned model (Phase 3). Nothing ever becomes a black box.
## Signals
- Like = strong positive (+1). Maybe = weak positive (+0.25, kept in its own bucket). Nope = negative (−1). List-view edits count identically to in-deck swipes; the latest call wins.
@@ -13,7 +13,10 @@ Per listing: family tier (from the hunt rubric), displacement (cc), price, year,
## Stage 1 — Cold start
Order by the hunt's existing rubric score (family tiers + MVI / whitewalls / windshield / spokes / ABS bonuses + price fit). This is already explainable (reason strings exist in the hunt system). Deck = rubric order until ≥ 20 swipes.
## Stage 2 — Learned order
## Stage 1.5 — Cheap swipe boost (v1.x, post-launch)
Before any model exists, the importer can nudge `score` with plain swarm signals — e.g. +N when recent likes cluster in a family, small time-decay otherwise — recomputed each import, zero ML, fully explainable. Ships as v1.x if Glenn wants it; v1 itself stays pure rubric.
## Stage 2 — Learned order (Phase 3 roadmap)
- Model: logistic regression on the features above; target = like vs nope (maybes contribute softly positive). Trained on all swipe history including edits.
- Pure-Python or numpy implementation (tiny data — no heavy ML dependency). Params + training metadata live in the DB; versioned.
- Retrain nightly right after import; on-demand retrain available from the app.
@@ -27,7 +30,7 @@ Each card can show up to 2 chips: top positive contributions for that bike ("low
## Evaluation
- Track: swipes/session, like-rate trend, maybe-rate, undo rate, and "liked-item rank" (how deep liked items sat — should trend shallow).
- Optional weekly digest via Conduit if Glenn wants it (default: silent).
- Notifications (decided 2026-11-10): in-app "new since last visit" badge only — no Conduit pushes. A weekly digest remains possible later if asked for.
## Open decisions
- Maybe-handling: weak positive + separate bucket (proposed) vs pure third bucket — confirm with Glenn at Phase 2.

36
docs/SOURCES.md Normal file
View File

@@ -0,0 +1,36 @@
# SOURCES — v1 source decision
**Decision (2026-11-10): Kijiji — NS + NB windows (`kijiji-ns` / `kijiji-nb`). One site, one parser, two regional feeds.**
Chosen after the Torres engineering consultation; the evidence below is from live probes + snapshot analysis run 2026-11-10 (memo + probe scripts kept with the project's working notes).
## Candidates at a glance
| candidate | live volume | churn | field fill | photos | parse | verdict |
|---|---|---|---|---|---|---|
| **Kijiji** (ns+nb) | ~640 cards in the current window; depth beyond it (page 15 still full in NS; ≥600 NS alone) | **~18 new + 18 gone / day (NS)** | price ~100%, km 30–36%, title/loc/desc ~100% | image on **every card** at parse time | SSR HTML; transient-partial quirk **already mitigated** (retries + last-known-good, re-verified stable) | **v1 source** |
| AutoTrader.ca (ns+nb) | ~117 cruiser listings; full motorcycle view padded with ATV/SxS (34/53 and 49/64 rows from ONE dealer) | slow | 100% (structured JSON) | inline | structured `__NEXT_DATA__` JSON — the easiest parse | source #2 later (field-rich enrichment) |
| Dealer Spike (procycle / tbb) | 160 units each | none (static stock) | price only; no km/desc/year | present | HTML w/ quirks | later source (new stock) |
| Facebook Marketplace | — | — | — | — | — | **never scraped — Glenn's manual column** |
## Why Kijiji over "the easiest scrape"
Glenn's words were "the easiest to scrape." Literally easiest = AutoTrader (structured JSON, `numberOfResults` handed to you). But the deck is the product: Kijiji carries ~4–5× the live volume, real daily churn to feed a nightly deck, a photo on every card, and a private + dealer mix — and its one known flakiness is already a solved problem in the production crawler that has run daily for weeks. Engineering ease that ships a thin, slow-turning deck is not actually easy. AutoTrader returns as source #2 through the same import path (its parser already exists in the hunt pipeline).
## Evidence highlights (2026-11-10)
- Snapshot: `kijiji-ns` 320 items, `kijiji-nb` 320 items; **58/320 NS rows are NB cross-post twins** → importer dedupes.
- Live fetch: page 1 fetched twice → byte-identical 40 cards (no flake that day); **40/40 cards carry image URLs**; pages 9 and 15 still full (depth available beyond the current window).
- Images: `media.kijiji.ca` URLs are **token-free and size-rewritable** (`?rule=kijijica-640-webp` → 200; `960-webp` rewrite → 200; URL without rule → 404 — the rule is the size selector, not a token).
- Anti-bot: none observed across 15 pages of probes + production traffic.
## Scope & growth path
- **v1 mirrors the proven 8-page/province window** (~640 cards). Depth exists past it; grow only after two clean weeks, throttled — anti-bot behavior on deep sweeps is untested long-term.
- **v2** adds AutoTrader (same importer path) — dense fields complement Kijiji's private-sale mix.
- **Later** dealer inventories (new stock); the import schema is source-agnostic from day one.
## Revisit criteria
- Kijiji block / anti-bot escalation persisting ≥2 weeks → flip the order, AutoTrader first.
- Kijiji markup drift breaking the parser → importer gates fail loud (deck stays last-known-good); the crawler's `fail_streak` monitoring already covers detection.

View File

@@ -6,6 +6,7 @@ Fidelity goal (Glenn's words): "as close to Tinder as possible." Photo-first, da
### 1. Deck (default)
- Full-height card stack: the top card is draggable; the next card peeks, scaled (~95%) behind it.
- Scope (decided 2026-11-10): every active bike appears in the deck eventually — ranking reorders, never hides.
- Card anatomy: photo fills the card; bottom gradient overlay carries: year + make + model (title), price (large), sub-line: km · place · first seen. Small chips row when signals exist (e.g. "MVI", "whitewalls", "bags") — only derived from real listing data, never invented.
- Stamps: during drag, "LIKE" (green) / "NOPE" (red) / "MAYBE" (amber) stamps fade in over the card proportional to horizontal drag; rotation up to ±12° follows the finger.
- Commit: release past ~35% width (or a flick with velocity) → the card flies off; otherwise it springs back.