No bikes showing on prod. #3

Closed
opened 2026-11-15 10:54:04 -04:00 by MacKwnoX · 6 comments
Owner

There is currently no bikes showing in the deck on prod. Like you to find out, update the issue on why, before building a fix.

There is currently no bikes showing in the deck on prod. Like you to find out, update the issue on why, before building a fix.
Collaborator

Picked this up. Here is the diagnosis, before any fix.

What is happening. The swipe deck is empty because the call that feeds it, GET /api/section, has been failing with a server error (500) since Nov 14. The list, matches, and search screens are unaffected.

Timeline.

  • Nov 14, 08:27 AST — the app composed your current batch (section 57, 25 cards), which included "2016 CRF150F" (#298) as a new card and "2011 yamaha stryker" (#306) as a second look.
  • Nov 14, 08:31 AST — the nightly import ran and correctly marked 29 delisted Kijiji ads as gone, including both of those two.
  • Since then, every deck request has returned 500. First hit: Nov 14, 12:22 AST. You hit it again this morning (several attempts between 10:50 and 10:53), then filed this issue.

Root cause. When serving a batch, the app builds each card's dossier data, including the "why this rank" breakdown. That builder looks the card up in the current pool of listings; for a card whose ad died after the batch was composed and which was never swiped, the lookup comes back empty and the code indexes into it anyway (a missing null check). One such card crashes the entire batch request. Listing #298 is exactly that card, and it is reproducible right now: GET /api/listing/298 returns 500, while GET /api/listing/306 (also delisted, but swiped, so it survives the lookup) returns 200.

Why it did not self-heal. The batch only recomposes by itself when it is fully drained of cards — this one still has cards. "Run agent now" (you tried it this morning) refreshes the taste profile but does not rebuild the batch. And the failing request shows no hint of any of this; the deck just looks empty.

Separate finding (not the cause of this issue). This morning's 08:00 sweep flagged every source as not-ok (Kijiji returned no pages, and the sweep ran unusually slowly). The 08:31 nightly import then correctly refused to import, by design (stats: last import failed). So no fresh stock arrived today — but that did not break the deck. The sweep's own tooling tracks this and raises a loud flag on a second consecutive failure.

Next. The fix is a guard so a delisted card degrades quietly (no rank breakdown for it) instead of taking the whole deck down. I will build and verify it on the dev branch and report back here with the commit and test results. Nothing deploys until you reply deploy.

Picked this up. Here is the diagnosis, before any fix. **What is happening.** The swipe deck is empty because the call that feeds it, `GET /api/section`, has been failing with a server error (500) since Nov 14. The list, matches, and search screens are unaffected. **Timeline.** - Nov 14, 08:27 AST — the app composed your current batch (section 57, 25 cards), which included "2016 CRF150F" (#298) as a new card and "2011 yamaha stryker" (#306) as a second look. - Nov 14, 08:31 AST — the nightly import ran and correctly marked 29 delisted Kijiji ads as gone, including both of those two. - Since then, every deck request has returned 500. First hit: Nov 14, 12:22 AST. You hit it again this morning (several attempts between 10:50 and 10:53), then filed this issue. **Root cause.** When serving a batch, the app builds each card's dossier data, including the "why this rank" breakdown. That builder looks the card up in the current pool of listings; for a card whose ad died after the batch was composed and which was never swiped, the lookup comes back empty and the code indexes into it anyway (a missing null check). One such card crashes the entire batch request. Listing #298 is exactly that card, and it is reproducible right now: `GET /api/listing/298` returns 500, while `GET /api/listing/306` (also delisted, but swiped, so it survives the lookup) returns 200. **Why it did not self-heal.** The batch only recomposes by itself when it is fully drained of cards — this one still has cards. "Run agent now" (you tried it this morning) refreshes the taste profile but does not rebuild the batch. And the failing request shows no hint of any of this; the deck just looks empty. **Separate finding (not the cause of this issue).** This morning's 08:00 sweep flagged every source as not-ok (Kijiji returned no pages, and the sweep ran unusually slowly). The 08:31 nightly import then correctly refused to import, by design (stats: last import failed). So no fresh stock arrived today — but that did not break the deck. The sweep's own tooling tracks this and raises a loud flag on a second consecutive failure. **Next.** The fix is a guard so a delisted card degrades quietly (no rank breakdown for it) instead of taking the whole deck down. I will build and verify it on the dev branch and report back here with the commit and test results. Nothing deploys until you reply `deploy`.
Collaborator

Fixed in beeace2 on dev.

What changed. One guard in the ranking-breakdown builder (app/taste.py): a card whose listing has left the active pool now degrades quietly (no rank-breakdown lines) instead of raising. Nothing else changes, and this also fixes the matching 500 when opening the dossier for such a listing from the list screen.

Checks.

  • app.selfcheck: 36/36 passed — includes a new regression test that composes a batch, delists one of its cards, and asserts the deck keeps serving (test_issue3_section_survives_midsection_delist).
  • Before/after on a copy of the live database (the live file was never touched):
    • before: GET /api/section → 500; GET /api/listing/298 → 500
    • after: GET /api/section → 200 (25 cards served, section 57 intact); GET /api/listing/298 → 200 (empty rank breakdown); POST /api/section/forge → 200 (fresh 25-card batch)
  • No UI changes, so the full fe_walk_v11 suite was not needed; the endpoint checks above cover the affected paths.

Once deployed, your current batch comes back as-is — including the two delisted ads; Forge next 25 will swap in a fresh batch whenever you want one.

Ready to deploy — reply deploy here and it ships; reply hold to pause.

Fixed in `beeace2` on `dev`. **What changed.** One guard in the ranking-breakdown builder (`app/taste.py`): a card whose listing has left the active pool now degrades quietly (no rank-breakdown lines) instead of raising. Nothing else changes, and this also fixes the matching 500 when opening the dossier for such a listing from the list screen. **Checks.** - `app.selfcheck`: **36/36 passed** — includes a new regression test that composes a batch, delists one of its cards, and asserts the deck keeps serving (`test_issue3_section_survives_midsection_delist`). - Before/after on a copy of the live database (the live file was never touched): - before: `GET /api/section` → 500; `GET /api/listing/298` → 500 - after: `GET /api/section` → 200 (25 cards served, section 57 intact); `GET /api/listing/298` → 200 (empty rank breakdown); `POST /api/section/forge` → 200 (fresh 25-card batch) - No UI changes, so the full `fe_walk_v11` suite was not needed; the endpoint checks above cover the affected paths. Once deployed, your current batch comes back as-is — including the two delisted ads; Forge next 25 will swap in a fresh batch whenever you want one. Ready to deploy — reply `deploy` here and it ships; reply `hold` to pause.
Author
Owner

Had an internet outage last night, it’s the reason for the failing. Just to confirm if this happens again we have a safe guard for it?

Had an internet outage last night, it’s the reason for the failing. Just to confirm if this happens again we have a safe guard for it?
Collaborator

Yes - and this morning gave a live demonstration of one of the layers.

First, one correction on the cause. The deck didn't fail because of the outage. It went dark on Nov 14, 08:31 AST, when the nightly import marked 29 listings as gone, including two cards sitting in your open batch (both had stopped appearing in the scan's results for two consecutive sweeps). The outage was not involved in that.

What the outage did hit, and how it was absorbed. This morning's 08:00 scan couldn't reach any source - all six failed to fetch (the connection here was out right over that window, roughly 08:00-08:24). From there, by design:

  • A failed scan changes nothing: every source keeps its last known-good catalogue, and a single bad sweep is recorded quietly (only two in a row raise a flag).
  • The 08:31 import refused the stale feed outright: it logged the run as failed and did not touch a single listing - no false "gone" marks came out of today.
  • The "gone" rule itself is outage-proof: an ad must be missed by two consecutive good sweeps before it can be marked gone, so one bad network day can never delist anything. And if an ad shows up again, it flips back to active automatically.

The crash class itself is what the fix closes. Once beeace2 is deployed, any card whose listing disappears mid-batch - outage, deletion, anything - can no longer take the deck down; it just drops its "why this rank" lines and the batch keeps serving. A regression test locks that in.

One refinement to my earlier diagnosis while I was checking: the two cards' Kijiji pages are live right now, so they may have just fallen out of the scan's search window rather than being deleted. It doesn't change the fix or the crash math - and the gone -> active round trip is already handled automatically.

Bottom line: worst case next time is a stale-but-correct deck that catches up on the next good sweep. The fix is ready and still waiting on one word - reply deploy here and it ships.

Yes - and this morning gave a live demonstration of one of the layers. **First, one correction on the cause.** The deck didn't fail because of the outage. It went dark on Nov 14, 08:31 AST, when the nightly import marked 29 listings as gone, including two cards sitting in your open batch (both had stopped appearing in the scan's results for two consecutive sweeps). The outage was not involved in that. **What the outage did hit, and how it was absorbed.** This morning's 08:00 scan couldn't reach any source - all six failed to fetch (the connection here was out right over that window, roughly 08:00-08:24). From there, by design: - A failed scan changes nothing: every source keeps its last known-good catalogue, and a single bad sweep is recorded quietly (only two in a row raise a flag). - The 08:31 import refused the stale feed outright: it logged the run as failed and did not touch a single listing - no false "gone" marks came out of today. - The "gone" rule itself is outage-proof: an ad must be missed by two consecutive *good* sweeps before it can be marked gone, so one bad network day can never delist anything. And if an ad shows up again, it flips back to active automatically. **The crash class itself** is what the fix closes. Once `beeace2` is deployed, any card whose listing disappears mid-batch - outage, deletion, anything - can no longer take the deck down; it just drops its "why this rank" lines and the batch keeps serving. A regression test locks that in. One refinement to my earlier diagnosis while I was checking: the two cards' Kijiji pages are live right now, so they may have just fallen out of the scan's search window rather than being deleted. It doesn't change the fix or the crash math - and the gone -> active round trip is already handled automatically. Bottom line: worst case next time is a stale-but-correct deck that catches up on the next good sweep. The fix is ready and still waiting on one word - reply `deploy` here and it ships.
Author
Owner

Deploy

Deploy
Collaborator

Deployed — the fix is live on the stack.

  • main and release are at 7088304 (fix beeace2 merged in).
  • Image rebuilt, containers re-raised by the stack unit; /api/health returns 200.
  • /api/section is serving again: your batch (section 57, 25 cards) is back.
  • Swipe ledger untouched: 236 swipes before and after (8 like / 13 maybe / 215 nope).

If anything looks off, file it here.

Deployed — the fix is live on the stack. - `main` and `release` are at `7088304` (fix `beeace2` merged in). - Image rebuilt, containers re-raised by the stack unit; `/api/health` returns 200. - `/api/section` is serving again: your batch (section 57, 25 cards) is back. - Swipe ledger untouched: 236 swipes before and after (8 like / 13 maybe / 215 nope). If anything looks off, file it here.
Sign in to join this conversation.
No Label
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: MacKwnoX/Bike-Tinder#3