feat(scraper): crawl a source that has never completed one #13

Merged

TimCane merged 2 commits from feat/seed-first-crawl into main

2026-07-19 16:40:06 +01:00

TimCane commented

2026-07-19 16:37:20 +01:00

(Migrated from github.com)

Summary

A fresh deployment shipped an empty database and crawled nothing until its scheduled night. With the sources now staggered that is up to a week of the MCP server answering "no recipes" - deploy on a Monday and Gousto arrives Wednesday, HelloFresh the following Sunday.

The scraper now queues a crawl at startup for any source with no completed run.

The condition is on runs, not rows, which is what makes it safe to leave on:

State at startup	Behaviour
never crawled	seed - the fresh-deployment case
last run succeeded	nothing - a redeploy must not start surprise traffic
last run failed or cancelled	seed - resumes from its saved cursor
last run still marked running	nothing - would duplicate work in flight

Queued rather than run inline, so startup does not block on a crawl that takes hours. Config-gated with Sources:<slug>:CrawlOnStartup, default true, so Sources__hellofresh__CrawlOnStartup=false deploys quiet.

Test plan

122 tests pass (77 unit, 45 integration); the four branches above are unit tested
Verified against the live stack rather than trusting the unit tests:
- Redeploy case - restarted with both sources holding successful runs: schedules registered, nothing queued
- Fresh deploy case - cleared run history and restarted: Queued a first crawl of gousto and ... of hellofresh, two jobs in Hangfire, and the worker picked one up immediately
- Interrupted case - stopping mid-crawl recorded the run as Cancelled (not Failed, not stuck Running), and the next boot queued it again

## Summary A fresh deployment shipped an empty database and crawled nothing until its scheduled night. With the sources now staggered that is up to a week of the MCP server answering "no recipes" - deploy on a Monday and Gousto arrives Wednesday, HelloFresh the following Sunday. The scraper now queues a crawl at startup for any source with no completed run. The condition is on **runs, not rows**, which is what makes it safe to leave on: | State at startup | Behaviour | |---|---| | never crawled | seed - the fresh-deployment case | | last run succeeded | nothing - a redeploy must not start surprise traffic | | last run failed or cancelled | seed - resumes from its saved cursor | | last run still marked running | nothing - would duplicate work in flight | Queued rather than run inline, so startup does not block on a crawl that takes hours. Config-gated with `Sources:<slug>:CrawlOnStartup`, default true, so `Sources__hellofresh__CrawlOnStartup=false` deploys quiet. ## Test plan - 122 tests pass (77 unit, 45 integration); the four branches above are unit tested - Verified against the live stack rather than trusting the unit tests: - **Redeploy case** - restarted with both sources holding successful runs: schedules registered, nothing queued - **Fresh deploy case** - cleared run history and restarted: `Queued a first crawl of gousto` and `... of hellofresh`, two jobs in Hangfire, and the worker picked one up immediately - **Interrupted case** - stopping mid-crawl recorded the run as Cancelled (not Failed, not stuck Running), and the next boot queued it again