fix(scraper): stop silent crawl failures leaving no recipes #17

Merged

TimCane merged 3 commits from fix/crawl-failure-reported-as-success into main

2026-07-19 17:41:30 +01:00

TimCane commented

2026-07-19 17:40:24 +01:00

(Migrated from github.com)

Summary

a refused request ended the crawl enumeration, which records the run as a complete success over however few documents it managed first: the run reads green, its cursor is discarded, and the truncation is never retried
crawlers now throw, so the run records Failed and keeps the cursor to resume from
a process killed mid-crawl left its run Running for good, which seeding reads as a crawl already in flight and skips: those runs are now cancelled at startup
normalising shared one queue and one worker with crawling, so it waited behind a crawl that runs for hours and recipes stayed unsearchable long after being fetched; it now has its own server and queue
separate servers rather than two workers: a second worker on a shared queue would take a second crawl, which is the overlap one worker exists to prevent

Test plan

dotnet test green: 83 unit, 48 integration
new unit tests cover both crawlers failing the run on a refused request, and still ending cleanly when the catalogue is genuinely exhausted
new integration tests cover an abandoned run being cancelled with its cursor intact, and a finished run being left alone
redeploy and confirm the orphaned gousto run clears and a crawl reaches more than 5 documents

## Summary - a refused request ended the crawl enumeration, which records the run as a complete success over however few documents it managed first: the run reads green, its cursor is discarded, and the truncation is never retried - crawlers now throw, so the run records Failed and keeps the cursor to resume from - a process killed mid-crawl left its run Running for good, which seeding reads as a crawl already in flight and skips: those runs are now cancelled at startup - normalising shared one queue and one worker with crawling, so it waited behind a crawl that runs for hours and recipes stayed unsearchable long after being fetched; it now has its own server and queue - separate servers rather than two workers: a second worker on a shared queue would take a second crawl, which is the overlap one worker exists to prevent ## Test plan - `dotnet test` green: 83 unit, 48 integration - new unit tests cover both crawlers failing the run on a refused request, and still ending cleanly when the catalogue is genuinely exhausted - new integration tests cover an abandoned run being cancelled with its cursor intact, and a finished run being left alone - [ ] redeploy and confirm the orphaned gousto run clears and a crawl reaches more than 5 documents