Bike-Tinder/docs/DATA-MODEL.md

# DATA-MODEL — Bike-Tinder

SQLite, WAL mode, `busy_timeout`. One file: `data/bike-tinder.sqlite3`. Schema as consulted (Torres, 2026-11-10), locked for v1.

## Tables

```sql
listings (
  id             INTEGER PRIMARY KEY AUTOINCREMENT,
  region         TEXT NOT NULL CHECK (region IN ('ns','nb')),
  src_id         TEXT NOT NULL,                    -- source ad id (Kijiji id in v1)
  title          TEXT NOT NULL,
  price_display  TEXT,                             -- "$5,000"; "Please Contact"
  price_num      INTEGER,                          -- NULL when unpackable
  km             INTEGER,                          -- NULL when not stated (~2/3 of cards)
  location       TEXT,
  url            TEXT NOT NULL,                    -- source ad page
  img_url        TEXT,                             -- source CDN card image
  desc           TEXT,                             -- card description snippet
  family         TEXT,                             -- top rubric family label (taste signal)
  score          INTEGER NOT NULL DEFAULT 0,       -- rubric score, recomputed each import
  status         TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active','gone','dup')),
  dup_of         INTEGER,                          -- canonical row id when status='dup'
  first_seen     TEXT,  last_seen TEXT,
  created_at     TEXT DEFAULT (datetime('now')),  updated_at TEXT,
  UNIQUE (region, src_id)
);
CREATE INDEX idx_listings_deck ON listings (status, score DESC);
CREATE INDEX idx_listings_dup  ON listings (dup_of) WHERE status = 'dup';

swipes (
  listing_id  INTEGER PRIMARY KEY REFERENCES listings(id) ON DELETE CASCADE,
  action      TEXT NOT NULL CHECK (action IN ('like','nope','maybe')),
  created_at  TEXT DEFAULT (datetime('now')),
  updated_at  TEXT
);

import_runs (
  id, started_at, ok, partial,
  src_fetched_ns, src_fetched_nb,
  total, new, gone, dup, error
);                                                 -- one observability row per nightly run

prefs (
  key TEXT PRIMARY KEY, value TEXT
);                                                 -- deck_size, depth, etc. — tiny
```

Notes:

- One row per swipe; a new swipe **upserts** (a re-swiped card overwrites its action). The latest action is the truth; edits from the list view are upserts. Append-history is deliberately not kept in v1.
- `import_runs` is the dashboard's evidence trail — every run writes one row, success or failure.

## State semantics

- **new** — `active` with no `swipes` row → the deck.
- **seen** — has any swipe row.
- **gone** — was `active`, then absent from ≥2 consecutive *good* imports → `status='gone'` (kept forever, swipes preserved; reappearance flips it back to `active`). Single misses and failed/partial imports never mark gone.
- **dup** — the second province's twin of the same ad (dedupe below) → hidden from the deck, visible under its canonical row in the list.
- **nope** — never re-enters the deck unless cleared from the list (`DELETE` swipe → back to unswiped).
- **maybe** — soft skip: the deck query re-admits `maybe` rows after **7 days** (computed at query time — no cron needed). Editing a maybe to like/nope is just an upsert.
- **like** — pins to the Liked ("matches") view.

## Deck query (one statement)

```sql
SELECT * FROM listings
WHERE status = 'active'
  AND id NOT IN (SELECT listing_id FROM swipes WHERE action = 'nope')
  AND id NOT IN (SELECT listing_id FROM swipes WHERE action IN ('like','maybe')
                 AND (action = 'like' OR updated_at > datetime('now','-7 days')))
ORDER BY score DESC, first_seen DESC
LIMIT :deck_size;
```

(Implementers: a `LEFT JOIN swipes` variant is equivalent and cheaper — the intent above is the contract. Unscored rows sort below scored ones; nothing is hidden.)

## Dedupe (Kijiji cross-province)

- Group imports by `(normalized_title, price_num)`; when a group spans both provinces, keep the more-complete row canonical (img present → km present → older row) and mark the twin(s) `status='dup', dup_of=canonical`. The twin's URL is preserved on the canonical row (extra note field / desc suffix).
- Today's snapshot: 58/320 NS rows are twins of NB rows (dealers cross-post one ad to both provinces).
- False merges are possible (same title + price, different bike) but rare; every merge is logged in `import_runs` and reversible in the DB. Honest v1 tradeoff.

## Import gates (must pass or the run aborts)

1. State file `ok == true` and `partial == false`.
2. `fetched_at` within 26 hours.
3. Non-empty `items`.
4. On failure: log, leave the DB untouched (deck stays last-known-good).

## Maintenance

- **Backup:** nightly `sqlite3 .backup` to `data/backup/` immediately after import (house rule; the DB is the app's only mutable state).
- **Prune:** mirrored image files for rows `gone` >30 days; rows themselves are never auto-deleted in v1.
- **VACUUM:** optional; the DB is KV-sized (a few thousand rows).

## P2 additions — frozen 2026-11-10 (production build)

- **New columns** (guarded, idempotent migrations in `db.init_db`: `ALTER TABLE ... ADD COLUMN` when missing):
  - `listings.year INTEGER` — parsed from title at import (`rubric.year_from_title`); NULL when absent.
  - `listings.seller_type TEXT` — `'dealer'` | `'private'` | NULL. Filled ONLY from a real page signal (the hunt image backfill / a later enrichment pass); never inferred from titles.
  - `listings.extras TEXT` — JSON array of rubric-matched extra labels (`MVI/cert, whitewalls, spokes, windshield, bags, ABS`); UI displays at most 3.
- **Importer:** fills the three new fields; all P1 behavior unchanged (gates, upsert, dedupe, gone-marking, idempotence, counts 640/560/80).
- **Deck feed + cursor pagination (production):** semantics unchanged (no swipe, or `maybe` older than 7 days; `active` only); order `score DESC, first_seen DESC, id DESC`. Cursor = opaque `"<score>|<first_seen>|<id>"` of the last returned row; next page = strictly-after in that tuple order (OR-chain on the three keys). `GET /api/deck?limit&cursor` → `{items, next_cursor, remaining}`. Cursor stays correct while swipes delete rows underneath it.
- **Badge:** `prefs['badge_seen_at']` (ISO). `new` = active rows with `first_seen` > COALESCE(badge_seen_at,'0'); `POST /api/badge/seen` stamps now.
- **Media mirror:** files at `data/img/{listing_id}.webp`; `GET /img/{id}` → file (200) → `img_url` (302) → 404 (UI placeholder). Downloads via `app/imgfetch.py` (polite: ~1 req/s, skip-existing, resumable). `img_url` enrichment happens upstream in the hunt state (`motorcycle-hunt-img.py`; atomic writes; rescans keep curated imgs) and lands via import.
- **Dedupe refinement (stretch, targeted):** for cross-province groups with >2 members, replace collapse-to-one with `min(ns,nb)` completeness pairing; surplus siblings stay active. Expected dup ≈ 77 (vs 80). Verify counts and report.

## P3 additions — frozen 2026-11-10 (personalization)

New tables (guarded, idempotent creation; fresh AND existing DBs migrate in place):

```sql
swipe_events (                       -- append-only strike log; the learning source of truth
  id          INTEGER PRIMARY KEY AUTOINCREMENT,
  listing_id  INTEGER NOT NULL REFERENCES listings(id) ON DELETE CASCADE,
  action      TEXT NOT NULL CHECK (action IN ('like','nope','maybe')),
  round       INTEGER NOT NULL DEFAULT 1,   -- 1 = first verdict; 2+ = second look (engine caps influence at round 2)
  section_id  INTEGER,                      -- section it happened in (NULL for list edits / legacy)
  created_at  TEXT DEFAULT (datetime('now'))
);
CREATE INDEX idx_events_listing ON swipe_events (listing_id, round);
CREATE INDEX idx_events_time    ON swipe_events (created_at);

sections (
  id              INTEGER PRIMARY KEY AUTOINCREMENT,
  status          TEXT NOT NULL DEFAULT 'open' CHECK (status IN ('open','report','closed')),
  size            INTEGER NOT NULL,
  profile_version INTEGER,                  -- taste_profile.version used at compose
  recipe          TEXT,                     -- JSON summary {new, return, wildcard, caps_applied, note}
  created_at      TEXT DEFAULT (datetime('now')),
  closed_at       TEXT
);

section_items (
  section_id  INTEGER NOT NULL REFERENCES sections(id) ON DELETE CASCADE,
  listing_id  INTEGER NOT NULL REFERENCES listings(id) ON DELETE CASCADE,
  slot        TEXT NOT NULL CHECK (slot IN ('new','return','wildcard')),
  position    INTEGER NOT NULL,
  PRIMARY KEY (section_id, listing_id)
);

taste_profile (
  version    INTEGER PRIMARY KEY AUTOINCREMENT,
  created_at TEXT DEFAULT (datetime('now')),
  source     TEXT NOT NULL CHECK (source IN ('agent','mock','fallback')),
  payload    TEXT NOT NULL                  -- JSON: lead, leanings, changes, chips, adjustments, wildcards
);

agent_runs (
  id         INTEGER PRIMARY KEY AUTOINCREMENT,
  started_at TEXT, finished_at TEXT,
  ok         INTEGER NOT NULL DEFAULT 0,
  source     TEXT, model TEXT, cost_note TEXT, error TEXT,
  profile_version INTEGER
);
```

Semantics:

- **Learning = `swipe_events` only** (append-only). A cleared swipe keeps its history — clearing removes the *UI verdict*, not the record. **UI truth = `swipes`** (latest verdict / cleared), unchanged from P1–P2.
- Round = prior events for that listing + 1, assigned at write time. Rounds cap at 2 for engine influence (any later events weigh as round 2; v1 returns never produce rounds >2 by design).
- **Migration seeding:** when `swipe_events` is empty and `swipes` has rows → insert one round-1 event per swipe row (created_at copied). Only-when-empty, hence idempotent.
- **Sections:** at most ONE `open` section at a time. The deck = that section's `section_items` in `position` order, filtered to items undecided at read. Exhausted = no item undecided → `status='report'`. Forge = close current + open new. First-ever read composes a cold section (rubric-weighted, w=0).
- **Returns eligibility (v1 contract):** a verdict exists (like/maybe/nope in `swipes`); listing `active`; NOT in either of the previous two sections; no round≥2 `nope` ever (rested); not already returned once (round cap). Selection prefers maybe > like > nope; nope candidates must be "surprising" (points ≥55 or net-positive family). Max 5 returns per section, ≤2 of them nopes.
- **`points` is computed, not stored** (deterministic from events + profile; a cache is allowed only if it never changes semantics).
- **prefs:** `forge_mode` ('ondemand' default | 'strict'). Existing keys unchanged.
- **Legacy deck semantics:** `/api/deck` (cursor paging; 7-day maybe re-admit) stays as a read-only fallback for debugging; sections are the production flow.

## B1 additions — frozen 2026-11-10 (photo experience v2)

- **New columns** (guarded, idempotent migrations in `db.init_db`):
  - `listings.imgs TEXT` — JSON array of gallery image URLs (page order, deduped by image id; entry 0 = the established primary). Absent/empty → single-image fallback to `img_url`.
  - `swipes.last_photo_idx INTEGER` — the photo index on screen when the card was struck (set by `POST /api/swipe`'s optional `photo_idx`; NULL when unknown). Drives UI-SPEC §11.5 rotation: a re-shown listing leads at `(idx+1) % n`.
- **Importer:** carries `imgs` from hunt state (state gains `imgs[]` beside `img`; the hunt image tool fills both in the ONE page fetch it already makes — `img` = established primary, `imgs[0]` = that primary, remainder = gallery in page order).
- **Media mirror:** `data/img/{id}_{n}.webp` per `imgs[n]` (polite ~1 req/s, skip-existing, resumable — same as P2); `GET /img/{id}?i=n` → mirrored file → CDN 302 → 404. Legacy `/img/{id}` unchanged.
- **API payload additions:** deck/section items and list rows carry `n_photos` (int) and `last_photo_idx` (int|null); `POST /api/swipe` accepts optional `photo_idx` (int) and persists it.
- **Rotation is deck-only** (list thumbs + dossier hero = photo 0).