docs: add SOURCES, DESIGN, DATA-MODEL; record v1 decisions (Kijiji source, rank-all deck, badge-only feed); vendor hunt reference modules
This commit is contained in:
95
docs/DATA-MODEL.md
Normal file
95
docs/DATA-MODEL.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# DATA-MODEL — Bike-Tinder
|
||||
|
||||
SQLite, WAL mode, `busy_timeout`. One file: `data/bike-tinder.sqlite3`. Schema as consulted (Torres, 2026-11-10), locked for v1.
|
||||
|
||||
## Tables
|
||||
|
||||
```sql
|
||||
listings (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
region TEXT NOT NULL CHECK (region IN ('ns','nb')),
|
||||
src_id TEXT NOT NULL, -- source ad id (Kijiji id in v1)
|
||||
title TEXT NOT NULL,
|
||||
price_display TEXT, -- "$5,000"; "Please Contact"
|
||||
price_num INTEGER, -- NULL when unpackable
|
||||
km INTEGER, -- NULL when not stated (~2/3 of cards)
|
||||
location TEXT,
|
||||
url TEXT NOT NULL, -- source ad page
|
||||
img_url TEXT, -- source CDN card image
|
||||
desc TEXT, -- card description snippet
|
||||
family TEXT, -- top rubric family label (taste signal)
|
||||
score INTEGER NOT NULL DEFAULT 0, -- rubric score, recomputed each import
|
||||
status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active','gone','dup')),
|
||||
dup_of INTEGER, -- canonical row id when status='dup'
|
||||
first_seen TEXT, last_seen TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now')), updated_at TEXT,
|
||||
UNIQUE (region, src_id)
|
||||
);
|
||||
CREATE INDEX idx_listings_deck ON listings (status, score DESC);
|
||||
CREATE INDEX idx_listings_dup ON listings (dup_of) WHERE status = 'dup';
|
||||
|
||||
swipes (
|
||||
listing_id INTEGER PRIMARY KEY REFERENCES listings(id) ON DELETE CASCADE,
|
||||
action TEXT NOT NULL CHECK (action IN ('like','nope','maybe')),
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
updated_at TEXT
|
||||
);
|
||||
|
||||
import_runs (
|
||||
id, started_at, ok, partial,
|
||||
src_fetched_ns, src_fetched_nb,
|
||||
total, new, gone, dup, error
|
||||
); -- one observability row per nightly run
|
||||
|
||||
prefs (
|
||||
key TEXT PRIMARY KEY, value TEXT
|
||||
); -- deck_size, depth, etc. — tiny
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- One row per swipe; a new swipe **upserts** (a re-swiped card overwrites its action). The latest action is the truth; edits from the list view are upserts. Append-history is deliberately not kept in v1.
|
||||
- `import_runs` is the dashboard's evidence trail — every run writes one row, success or failure.
|
||||
|
||||
## State semantics
|
||||
|
||||
- **new** — `active` with no `swipes` row → the deck.
|
||||
- **seen** — has any swipe row.
|
||||
- **gone** — was `active`, then absent from ≥2 consecutive *good* imports → `status='gone'` (kept forever, swipes preserved; reappearance flips it back to `active`). Single misses and failed/partial imports never mark gone.
|
||||
- **dup** — the second province's twin of the same ad (dedupe below) → hidden from the deck, visible under its canonical row in the list.
|
||||
- **nope** — never re-enters the deck unless cleared from the list (`DELETE` swipe → back to unswiped).
|
||||
- **maybe** — soft skip: the deck query re-admits `maybe` rows after **7 days** (computed at query time — no cron needed). Editing a maybe to like/nope is just an upsert.
|
||||
- **like** — pins to the Liked ("matches") view.
|
||||
|
||||
## Deck query (one statement)
|
||||
|
||||
```sql
|
||||
SELECT * FROM listings
|
||||
WHERE status = 'active'
|
||||
AND id NOT IN (SELECT listing_id FROM swipes WHERE action = 'nope')
|
||||
AND id NOT IN (SELECT listing_id FROM swipes WHERE action IN ('like','maybe')
|
||||
AND (action = 'like' OR updated_at > datetime('now','-7 days')))
|
||||
ORDER BY score DESC, first_seen DESC
|
||||
LIMIT :deck_size;
|
||||
```
|
||||
|
||||
(Implementers: a `LEFT JOIN swipes` variant is equivalent and cheaper — the intent above is the contract. Unscored rows sort below scored ones; nothing is hidden.)
|
||||
|
||||
## Dedupe (Kijiji cross-province)
|
||||
|
||||
- Group imports by `(normalized_title, price_num)`; when a group spans both provinces, keep the more-complete row canonical (img present → km present → older row) and mark the twin(s) `status='dup', dup_of=canonical`. The twin's URL is preserved on the canonical row (extra note field / desc suffix).
|
||||
- Today's snapshot: 58/320 NS rows are twins of NB rows (dealers cross-post one ad to both provinces).
|
||||
- False merges are possible (same title + price, different bike) but rare; every merge is logged in `import_runs` and reversible in the DB. Honest v1 tradeoff.
|
||||
|
||||
## Import gates (must pass or the run aborts)
|
||||
|
||||
1. State file `ok == true` and `partial == false`.
|
||||
2. `fetched_at` within 26 hours.
|
||||
3. Non-empty `items`.
|
||||
4. On failure: log, leave the DB untouched (deck stays last-known-good).
|
||||
|
||||
## Maintenance
|
||||
|
||||
- **Backup:** nightly `sqlite3 .backup` to `data/backup/` immediately after import (house rule; the DB is the app's only mutable state).
|
||||
- **Prune:** mirrored image files for rows `gone` >30 days; rows themselves are never auto-deleted in v1.
|
||||
- **VACUUM:** optional; the DB is KV-sized (a few thousand rows).
|
||||
101
docs/DESIGN.md
Normal file
101
docs/DESIGN.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# DESIGN — Bike-Tinder
|
||||
|
||||
Architecture for v1 (single source, single user, tailnet-only PWA). Written 2026-11-10 by Alex; engineering input from the Torres consultation (same date). Items marked 【decided】 are locked for v1.
|
||||
|
||||
## Components
|
||||
|
||||
| piece | what | where |
|
||||
|---|---|---|
|
||||
| Crawler | existing `motorcycle-hunt-scan.py` (**unchanged**) — sweeps Kijiji NS+NB nightly at 08:00 | Alex's pipeline (`~/.hermes/scripts/`) |
|
||||
| Importer | reads the crawler's state JSONs, gates them, upserts into the app DB, scores, dedupes, queues images | `app/importer.py` |
|
||||
| API | FastAPI + uvicorn — deck, swipes, list, stats, images | `app/api.py` |
|
||||
| PWA | no-build single-page frontend (deck, list, matches, trends) | `app/static/` |
|
||||
| DB | SQLite (WAL mode) | `data/bike-tinder.sqlite3` (stack volume) |
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
08:00 motorcycle-hunt crawl (existing, unchanged) → state JSONs: kijiji-ns.json, kijiji-nb.json
|
||||
08:15 importer (script-only, zero LLM):
|
||||
gates → upsert → dedupe → score → gone-marking → image queue
|
||||
→ SQLite (WAL)
|
||||
Deck phone ⇄ API ⇄ SQLite
|
||||
↘ images: local mirror first, hotlink (302) fallback
|
||||
```
|
||||
|
||||
## Import design 【decided】
|
||||
|
||||
- **Source of truth = the hunt's state files, not a re-crawl.** Rationale: zero duplicate load on Kijiji; inherits the crawler's retry / partial / last-known-good / absence logic instead of re-implementing it; the format is stable and self-describing (`ok`, `partial`, `fetched_at`, per-page counts).
|
||||
- **Escape hatch:** the app vendors the Kijiji parser from `reference/motorcycle-hunt-scan.py` behind a `--direct` flag — if the hunt pipeline ever moves or dies, the importer can sweep Kijiji itself.
|
||||
- **Gates (fail loud, keep last good):** each state file must have `ok == true`, `partial == false`, `fetched_at` within 26 h, and non-empty `items`. Any failure → abort that region, log to `import_runs`, leave the DB untouched.
|
||||
- **Gone marking:** a listing absent from ≥2 consecutive *good* imports flips to `gone` (never deleted; reappearance flips it back). House discipline: a single miss — or any blocked page — never counts as "sold."
|
||||
- **Dedupe:** cross-province twins (same normalized title + price) collapse to one canonical row (`dup` status), keeping the more complete row (img > km > older). Merges are logged and reversible.
|
||||
- **Scoring:** rubric tables ported from `reference/motorcycle-hunt-rank.py` (family tiers + extras + veto) — the score is Glenn's accumulated taste, computed at import, stored on each row. See `RANKING.md`.
|
||||
- **Idempotent:** safe to re-run any time; also exposed as `POST /api/import` for manual triggers.
|
||||
|
||||
## Scheduling 【decided】
|
||||
|
||||
An in-container daily task (asyncio, 08:15 America/Halifax) inside the API process, with on-boot catch-up for missed runs. No host crontab is involved (the host user crontab is empty; the 08:00 crawl lives in Hermes cron — the app depends on neither). Every run writes an `import_runs` row; `GET /api/stats` surfaces the last run.
|
||||
|
||||
## API surface (v1)
|
||||
|
||||
| route | purpose |
|
||||
|---|---|
|
||||
| `GET /api/health` | liveness + last-import summary |
|
||||
| `GET /api/stats` | counts: active / gone / dup / swiped; last import time |
|
||||
| `GET /api/deck?limit=` | next cards (deck query below) |
|
||||
| `POST /api/swipe` `{id, action}` | record or edit like / nope / maybe (upsert) |
|
||||
| `DELETE /api/swipe/{id}` | clear a call (returns the card to the deck) |
|
||||
| `GET /api/list?filter=&q=&offset=` | full list, filters + search |
|
||||
| `POST /api/import` | manual import trigger |
|
||||
| `GET /img/{id}` | cached image file, else 302 to source CDN |
|
||||
|
||||
**Deck query:** `status='active'`, not noped, and either no swipe or a `maybe` older than 7 days — ordered `score DESC, first_seen DESC`. Unscored rows sort below scored ones but still appear — nothing is hidden.
|
||||
|
||||
## Images 【decided: hybrid mirror + hotlink】
|
||||
|
||||
- The importer pulls each new listing's card image once into `data/img/` (~100 KB avg; ≈64 MB the first day, then ~20–30 MB/month). Cards reference the local mirror; a missing file falls back to a 302 hotlink of the source CDN (verified token-free and size-rewritable).
|
||||
- Why: ad removal eventually kills CDN URLs (unverifiable until it happens — the mirror makes it a non-issue), phone-side speed, and "gone" cards stay viewable.
|
||||
- Bootstrap: today's state carries `img` on only ~3% of rows — the first import does a one-time headless re-sweep (8 pages/province via the vendored parser — 16 fetches, not 640 detail pages) to harvest card images. A placeholder renders for any listing still without a photo; it never holds up the deck. Mirrored files for rows gone >30 days are pruned.
|
||||
|
||||
## Frontend
|
||||
|
||||
No-build SPA — vanilla JS + CSS, pointer-events drag, keyboard arrows on desktop (~300 lines). A small service worker caches the shell; the API is always live. Screens, gestures, and visual anatomy: see `UI-SPEC.md`.
|
||||
|
||||
## Deployment (house constraints)
|
||||
|
||||
- **Stack dir:** `/var/mnt/DATA/podman-stacks/bike-tinder/` — `docker-compose.yml` + `Dockerfile` (`python:3.14-slim` + fastapi/uvicorn/requests), volume `./data:/data`, port `127.0.0.1:9453:8000`, `restart: unless-stopped`.
|
||||
- After adding files: `chcon -Rt container_file_t /var/mnt/DATA/podman-stacks/bike-tinder`.
|
||||
- **Systemd user unit** `podman-stack-bike-tinder.service`, cloned from the machine's `podman-stack-searxng.service` pattern: `Type=oneshot`, `RemainAfterExit=yes`, **`Delegate=yes`** (keeps the stack out of the hermes-gateway cgroup so gateway restarts can't SIGKILL it); ExecStart/ExecStop = the venv `podman-compose up -d` / `down`.
|
||||
- **Tailnet:** `tailscale serve --bg 9453` → reachable at `http://rotarran.tailf35a34.ts.net:9453`, tailnet-only. The tailnet name can't be curled from this host — first deploy verification happens from Glenn's phone.
|
||||
- **Backups:** nightly `sqlite3 .backup` into `data/backup/` right after import (the DB is the only mutable state; house rule). Restore = stop stack, swap the file, start.
|
||||
|
||||
## Security
|
||||
|
||||
Single user; **no auth in v1** — localhost bind + tailnet-only reachability. An optional `AUTH_TOKEN` env var is noted as a one-line later hardening. No secrets are required in the stack.
|
||||
|
||||
## Risks & mitigations (from consultation)
|
||||
|
||||
| risk | mitigation |
|
||||
|---|---|
|
||||
| Kijiji markup drift | parser keys on `data-testid`; failures → gates abort, deck stays last-known-good; crawler's fail/partial tracking monitors |
|
||||
| Anti-bot on deep sweeps | ship at the proven 8-page window; grow only after two clean weeks |
|
||||
| Image persistence after ad removal | local mirror (built-in) |
|
||||
| Cross-province dedupe false merges | logged, reversible, low blast radius; canonical keeps the more complete row |
|
||||
| Hunt pipeline dependency | `fetched_at` gate fails loud; vendored `--direct` parser escape hatch |
|
||||
| Dealer blast dominance in score ties | tie-break `first_seen DESC`; "hide this seller" planned as a v1.1 prefs row |
|
||||
| Rubric blindness (no family match) | rows still appear, sorted below scored ones — nothing hidden |
|
||||
|
||||
## Repo layout (target)
|
||||
|
||||
```
|
||||
app/ importer.py · api.py · static/{index.html,app.js,app.css,manifest.json,sw.js,icons}
|
||||
deploy/ docker-compose.yml · Dockerfile · podman-stack-bike-tinder.service
|
||||
reference/ copies of the hunt modules the app vendors from
|
||||
data/ runtime state (gitignored)
|
||||
docs/ this design set
|
||||
```
|
||||
|
||||
## Runbook (completed at Phase 4)
|
||||
|
||||
Deploy / update / rollback steps, log locations, import troubleshooting (gates), backup-restore drill, phone verification checklist.
|
||||
@@ -10,8 +10,9 @@ A single-source, single-user, tailnet-only Tinder-style PWA on the Rotarran: nig
|
||||
|
||||
- [x] Engineering consultation (Torres) → memo
|
||||
- [x] Design docs in this repo: SOURCES, DESIGN, DATA-MODEL, RANKING, UI-SPEC
|
||||
- [ ] Glenn's read on source choice + docs
|
||||
- **Exit criteria:** docs merged to `main`; source decision recorded.
|
||||
- [x] Glenn's calls (2026-11-10): v1 source = **Kijiji** · deck = show all, taste-ranked · new-bike feed = in-app badge only
|
||||
- [ ] Glenn's skim of the docs — build gate opens after
|
||||
- **Exit criteria:** docs merged to `main` (done 2026-11-10); source decision recorded in `SOURCES.md`.
|
||||
|
||||
## Phase 1 — Data spine
|
||||
|
||||
@@ -53,5 +54,5 @@ A single-source, single-user, tailnet-only Tinder-style PWA on the Rotarran: nig
|
||||
|
||||
## Open decisions
|
||||
|
||||
- v1 source — pending consultation evidence (see `docs/SOURCES.md`)
|
||||
- v1 source — **resolved 2026-11-10: Kijiji** (see `docs/SOURCES.md`)
|
||||
- Mockup direction — Glenn picks at Phase 2 start
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# RANKING — Bike-Tinder
|
||||
|
||||
The deck is ordered by predicted taste. Two stages: a cold start (the hunt's existing rubric), then a small learned model that never becomes a black box.
|
||||
The deck is ordered by predicted taste. Three rungs, in order: the hunt's existing rubric (v1), a cheap explainable boost from swipe clusters (v1.x), then a small learned model (Phase 3). Nothing ever becomes a black box.
|
||||
|
||||
## Signals
|
||||
- Like = strong positive (+1). Maybe = weak positive (+0.25, kept in its own bucket). Nope = negative (−1). List-view edits count identically to in-deck swipes; the latest call wins.
|
||||
@@ -13,7 +13,10 @@ Per listing: family tier (from the hunt rubric), displacement (cc), price, year,
|
||||
## Stage 1 — Cold start
|
||||
Order by the hunt's existing rubric score (family tiers + MVI / whitewalls / windshield / spokes / ABS bonuses + price fit). This is already explainable (reason strings exist in the hunt system). Deck = rubric order until ≥ 20 swipes.
|
||||
|
||||
## Stage 2 — Learned order
|
||||
## Stage 1.5 — Cheap swipe boost (v1.x, post-launch)
|
||||
Before any model exists, the importer can nudge `score` with plain swarm signals — e.g. +N when recent likes cluster in a family, small time-decay otherwise — recomputed each import, zero ML, fully explainable. Ships as v1.x if Glenn wants it; v1 itself stays pure rubric.
|
||||
|
||||
## Stage 2 — Learned order (Phase 3 roadmap)
|
||||
- Model: logistic regression on the features above; target = like vs nope (maybes contribute softly positive). Trained on all swipe history including edits.
|
||||
- Pure-Python or numpy implementation (tiny data — no heavy ML dependency). Params + training metadata live in the DB; versioned.
|
||||
- Retrain nightly right after import; on-demand retrain available from the app.
|
||||
@@ -27,7 +30,7 @@ Each card can show up to 2 chips: top positive contributions for that bike ("low
|
||||
|
||||
## Evaluation
|
||||
- Track: swipes/session, like-rate trend, maybe-rate, undo rate, and "liked-item rank" (how deep liked items sat — should trend shallow).
|
||||
- Optional weekly digest via Conduit if Glenn wants it (default: silent).
|
||||
- Notifications (decided 2026-11-10): in-app "new since last visit" badge only — no Conduit pushes. A weekly digest remains possible later if asked for.
|
||||
|
||||
## Open decisions
|
||||
- Maybe-handling: weak positive + separate bucket (proposed) vs pure third bucket — confirm with Glenn at Phase 2.
|
||||
|
||||
36
docs/SOURCES.md
Normal file
36
docs/SOURCES.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# SOURCES — v1 source decision
|
||||
|
||||
**Decision (2026-11-10): Kijiji — NS + NB windows (`kijiji-ns` / `kijiji-nb`). One site, one parser, two regional feeds.**
|
||||
|
||||
Chosen after the Torres engineering consultation; the evidence below is from live probes + snapshot analysis run 2026-11-10 (memo + probe scripts kept with the project's working notes).
|
||||
|
||||
## Candidates at a glance
|
||||
|
||||
| candidate | live volume | churn | field fill | photos | parse | verdict |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **Kijiji** (ns+nb) | ~640 cards in the current window; depth beyond it (page 15 still full in NS; ≥600 NS alone) | **~18 new + 18 gone / day (NS)** | price ~100%, km 30–36%, title/loc/desc ~100% | image on **every card** at parse time | SSR HTML; transient-partial quirk **already mitigated** (retries + last-known-good, re-verified stable) | **v1 source** |
|
||||
| AutoTrader.ca (ns+nb) | ~117 cruiser listings; full motorcycle view padded with ATV/SxS (34/53 and 49/64 rows from ONE dealer) | slow | 100% (structured JSON) | inline | structured `__NEXT_DATA__` JSON — the easiest parse | source #2 later (field-rich enrichment) |
|
||||
| Dealer Spike (procycle / tbb) | 160 units each | none (static stock) | price only; no km/desc/year | present | HTML w/ quirks | later source (new stock) |
|
||||
| Facebook Marketplace | — | — | — | — | — | **never scraped — Glenn's manual column** |
|
||||
|
||||
## Why Kijiji over "the easiest scrape"
|
||||
|
||||
Glenn's words were "the easiest to scrape." Literally easiest = AutoTrader (structured JSON, `numberOfResults` handed to you). But the deck is the product: Kijiji carries ~4–5× the live volume, real daily churn to feed a nightly deck, a photo on every card, and a private + dealer mix — and its one known flakiness is already a solved problem in the production crawler that has run daily for weeks. Engineering ease that ships a thin, slow-turning deck is not actually easy. AutoTrader returns as source #2 through the same import path (its parser already exists in the hunt pipeline).
|
||||
|
||||
## Evidence highlights (2026-11-10)
|
||||
|
||||
- Snapshot: `kijiji-ns` 320 items, `kijiji-nb` 320 items; **58/320 NS rows are NB cross-post twins** → importer dedupes.
|
||||
- Live fetch: page 1 fetched twice → byte-identical 40 cards (no flake that day); **40/40 cards carry image URLs**; pages 9 and 15 still full (depth available beyond the current window).
|
||||
- Images: `media.kijiji.ca` URLs are **token-free and size-rewritable** (`?rule=kijijica-640-webp` → 200; `960-webp` rewrite → 200; URL without rule → 404 — the rule is the size selector, not a token).
|
||||
- Anti-bot: none observed across 15 pages of probes + production traffic.
|
||||
|
||||
## Scope & growth path
|
||||
|
||||
- **v1 mirrors the proven 8-page/province window** (~640 cards). Depth exists past it; grow only after two clean weeks, throttled — anti-bot behavior on deep sweeps is untested long-term.
|
||||
- **v2** adds AutoTrader (same importer path) — dense fields complement Kijiji's private-sale mix.
|
||||
- **Later** dealer inventories (new stock); the import schema is source-agnostic from day one.
|
||||
|
||||
## Revisit criteria
|
||||
|
||||
- Kijiji block / anti-bot escalation persisting ≥2 weeks → flip the order, AutoTrader first.
|
||||
- Kijiji markup drift breaking the parser → importer gates fail loud (deck stays last-known-good); the crawler's `fail_streak` monitoring already covers detection.
|
||||
@@ -6,6 +6,7 @@ Fidelity goal (Glenn's words): "as close to Tinder as possible." Photo-first, da
|
||||
|
||||
### 1. Deck (default)
|
||||
- Full-height card stack: the top card is draggable; the next card peeks, scaled (~95%) behind it.
|
||||
- Scope (decided 2026-11-10): every active bike appears in the deck eventually — ranking reorders, never hides.
|
||||
- Card anatomy: photo fills the card; bottom gradient overlay carries: year + make + model (title), price (large), sub-line: km · place · first seen. Small chips row when signals exist (e.g. "MVI", "whitewalls", "bags") — only derived from real listing data, never invented.
|
||||
- Stamps: during drag, "LIKE" (green) / "NOPE" (red) / "MAYBE" (amber) stamps fade in over the card proportional to horizontal drag; rotation up to ±12° follows the finger.
|
||||
- Commit: release past ~35% width (or a flick with velocity) → the card flies off; otherwise it springs back.
|
||||
|
||||
Reference in New Issue
Block a user