108 lines
6.7 KiB
Markdown
108 lines
6.7 KiB
Markdown
# DATA-MODEL — Bike-Tinder
|
|
|
|
SQLite, WAL mode, `busy_timeout`. One file: `data/bike-tinder.sqlite3`. Schema as consulted (Torres, 2026-11-10), locked for v1.
|
|
|
|
## Tables
|
|
|
|
```sql
|
|
listings (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
region TEXT NOT NULL CHECK (region IN ('ns','nb')),
|
|
src_id TEXT NOT NULL, -- source ad id (Kijiji id in v1)
|
|
title TEXT NOT NULL,
|
|
price_display TEXT, -- "$5,000"; "Please Contact"
|
|
price_num INTEGER, -- NULL when unpackable
|
|
km INTEGER, -- NULL when not stated (~2/3 of cards)
|
|
location TEXT,
|
|
url TEXT NOT NULL, -- source ad page
|
|
img_url TEXT, -- source CDN card image
|
|
desc TEXT, -- card description snippet
|
|
family TEXT, -- top rubric family label (taste signal)
|
|
score INTEGER NOT NULL DEFAULT 0, -- rubric score, recomputed each import
|
|
status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active','gone','dup')),
|
|
dup_of INTEGER, -- canonical row id when status='dup'
|
|
first_seen TEXT, last_seen TEXT,
|
|
created_at TEXT DEFAULT (datetime('now')), updated_at TEXT,
|
|
UNIQUE (region, src_id)
|
|
);
|
|
CREATE INDEX idx_listings_deck ON listings (status, score DESC);
|
|
CREATE INDEX idx_listings_dup ON listings (dup_of) WHERE status = 'dup';
|
|
|
|
swipes (
|
|
listing_id INTEGER PRIMARY KEY REFERENCES listings(id) ON DELETE CASCADE,
|
|
action TEXT NOT NULL CHECK (action IN ('like','nope','maybe')),
|
|
created_at TEXT DEFAULT (datetime('now')),
|
|
updated_at TEXT
|
|
);
|
|
|
|
import_runs (
|
|
id, started_at, ok, partial,
|
|
src_fetched_ns, src_fetched_nb,
|
|
total, new, gone, dup, error
|
|
); -- one observability row per nightly run
|
|
|
|
prefs (
|
|
key TEXT PRIMARY KEY, value TEXT
|
|
); -- deck_size, depth, etc. — tiny
|
|
```
|
|
|
|
Notes:
|
|
|
|
- One row per swipe; a new swipe **upserts** (a re-swiped card overwrites its action). The latest action is the truth; edits from the list view are upserts. Append-history is deliberately not kept in v1.
|
|
- `import_runs` is the dashboard's evidence trail — every run writes one row, success or failure.
|
|
|
|
## State semantics
|
|
|
|
- **new** — `active` with no `swipes` row → the deck.
|
|
- **seen** — has any swipe row.
|
|
- **gone** — was `active`, then absent from ≥2 consecutive *good* imports → `status='gone'` (kept forever, swipes preserved; reappearance flips it back to `active`). Single misses and failed/partial imports never mark gone.
|
|
- **dup** — the second province's twin of the same ad (dedupe below) → hidden from the deck, visible under its canonical row in the list.
|
|
- **nope** — never re-enters the deck unless cleared from the list (`DELETE` swipe → back to unswiped).
|
|
- **maybe** — soft skip: the deck query re-admits `maybe` rows after **7 days** (computed at query time — no cron needed). Editing a maybe to like/nope is just an upsert.
|
|
- **like** — pins to the Liked ("matches") view.
|
|
|
|
## Deck query (one statement)
|
|
|
|
```sql
|
|
SELECT * FROM listings
|
|
WHERE status = 'active'
|
|
AND id NOT IN (SELECT listing_id FROM swipes WHERE action = 'nope')
|
|
AND id NOT IN (SELECT listing_id FROM swipes WHERE action IN ('like','maybe')
|
|
AND (action = 'like' OR updated_at > datetime('now','-7 days')))
|
|
ORDER BY score DESC, first_seen DESC
|
|
LIMIT :deck_size;
|
|
```
|
|
|
|
(Implementers: a `LEFT JOIN swipes` variant is equivalent and cheaper — the intent above is the contract. Unscored rows sort below scored ones; nothing is hidden.)
|
|
|
|
## Dedupe (Kijiji cross-province)
|
|
|
|
- Group imports by `(normalized_title, price_num)`; when a group spans both provinces, keep the more-complete row canonical (img present → km present → older row) and mark the twin(s) `status='dup', dup_of=canonical`. The twin's URL is preserved on the canonical row (extra note field / desc suffix).
|
|
- Today's snapshot: 58/320 NS rows are twins of NB rows (dealers cross-post one ad to both provinces).
|
|
- False merges are possible (same title + price, different bike) but rare; every merge is logged in `import_runs` and reversible in the DB. Honest v1 tradeoff.
|
|
|
|
## Import gates (must pass or the run aborts)
|
|
|
|
1. State file `ok == true` and `partial == false`.
|
|
2. `fetched_at` within 26 hours.
|
|
3. Non-empty `items`.
|
|
4. On failure: log, leave the DB untouched (deck stays last-known-good).
|
|
|
|
## Maintenance
|
|
|
|
- **Backup:** nightly `sqlite3 .backup` to `data/backup/` immediately after import (house rule; the DB is the app's only mutable state).
|
|
- **Prune:** mirrored image files for rows `gone` >30 days; rows themselves are never auto-deleted in v1.
|
|
- **VACUUM:** optional; the DB is KV-sized (a few thousand rows).
|
|
|
|
## P2 additions — frozen 2026-11-10 (production build)
|
|
|
|
- **New columns** (guarded, idempotent migrations in `db.init_db`: `ALTER TABLE ... ADD COLUMN` when missing):
|
|
- `listings.year INTEGER` — parsed from title at import (`rubric.year_from_title`); NULL when absent.
|
|
- `listings.seller_type TEXT` — `'dealer'` | `'private'` | NULL. Filled ONLY from a real page signal (the hunt image backfill / a later enrichment pass); never inferred from titles.
|
|
- `listings.extras TEXT` — JSON array of rubric-matched extra labels (`MVI/cert, whitewalls, spokes, windshield, bags, ABS`); UI displays at most 3.
|
|
- **Importer:** fills the three new fields; all P1 behavior unchanged (gates, upsert, dedupe, gone-marking, idempotence, counts 640/560/80).
|
|
- **Deck feed + cursor pagination (production):** semantics unchanged (no swipe, or `maybe` older than 7 days; `active` only); order `score DESC, first_seen DESC, id DESC`. Cursor = opaque `"<score>|<first_seen>|<id>"` of the last returned row; next page = strictly-after in that tuple order (OR-chain on the three keys). `GET /api/deck?limit&cursor` → `{items, next_cursor, remaining}`. Cursor stays correct while swipes delete rows underneath it.
|
|
- **Badge:** `prefs['badge_seen_at']` (ISO). `new` = active rows with `first_seen` > COALESCE(badge_seen_at,'0'); `POST /api/badge/seen` stamps now.
|
|
- **Media mirror:** files at `data/img/{listing_id}.webp`; `GET /img/{id}` → file (200) → `img_url` (302) → 404 (UI placeholder). Downloads via `app/imgfetch.py` (polite: ~1 req/s, skip-existing, resumable). `img_url` enrichment happens upstream in the hunt state (`motorcycle-hunt-img.py`; atomic writes; rescans keep curated imgs) and lands via import.
|
|
- **Dedupe refinement (stretch, targeted):** for cross-province groups with >2 members, replace collapse-to-one with `min(ns,nb)` completeness pairing; surplus siblings stay active. Expected dup ≈ 77 (vs 80). Verify counts and report.
|