chore: stagger the source crawls across the week #12

Merged

TimCane merged 1 commit from chore/stagger-crawl-schedules into main

2026-07-19 16:31:10 +01:00

TimCane commented

2026-07-19 16:29:32 +01:00

(Migrated from github.com)

Summary

HelloFresh crawls Sunday 03:00 UTC, Gousto Wednesday 03:00 UTC, set per source in the scraper's appsettings.json
Normalising is unchanged - hourly for every source, independent of crawling
Record the reasoning on SourceOptions.Schedule and in PLAN.md, and correct the pace figure there: a full HelloFresh pass is about 3,070 requests and an hour and three quarters, not the 20 minutes originally estimated from the HAR's cuisine-filtered total

Why stagger: one Hangfire worker runs jobs serially, so a shared night queued Gousto behind a nearly two-hour HelloFresh pass. Spreading them also keeps each session from a single VPN exit shorter, which matters given Cloudflare already blocks some Proton exits outright. Recipe catalogues do not change fast enough to need the same night.

Also documents that the cron is UTC - 03:00 UTC is 04:00 British Summer Time, so it is not "3am local" for half the year.

Test plan

117 tests pass (72 unit, 45 integration)
Started the scraper against the live stack and confirmed the config binds, rather than trusting the file: log reports Scheduled gousto: crawl '0 3 * * 3' and Scheduled hellofresh: crawl '0 3 * * 0'
Checked Hangfire's stored state directly - recurring-job:crawl:gousto next runs Wed 2026-07-22 03:00 UTC, crawl:hellofresh next runs Sun 2026-07-26 03:00 UTC

## Summary - HelloFresh crawls Sunday 03:00 UTC, Gousto Wednesday 03:00 UTC, set per source in the scraper's `appsettings.json` - Normalising is unchanged - hourly for every source, independent of crawling - Record the reasoning on `SourceOptions.Schedule` and in PLAN.md, and correct the pace figure there: a full HelloFresh pass is about 3,070 requests and an hour and three quarters, not the 20 minutes originally estimated from the HAR's cuisine-filtered total Why stagger: one Hangfire worker runs jobs serially, so a shared night queued Gousto behind a nearly two-hour HelloFresh pass. Spreading them also keeps each session from a single VPN exit shorter, which matters given Cloudflare already blocks some Proton exits outright. Recipe catalogues do not change fast enough to need the same night. Also documents that the cron is **UTC** - 03:00 UTC is 04:00 British Summer Time, so it is not "3am local" for half the year. ## Test plan - 117 tests pass (72 unit, 45 integration) - Started the scraper against the live stack and confirmed the config binds, rather than trusting the file: log reports `Scheduled gousto: crawl '0 3 * * 3'` and `Scheduled hellofresh: crawl '0 3 * * 0'` - Checked Hangfire's stored state directly - `recurring-job:crawl:gousto` next runs Wed 2026-07-22 03:00 UTC, `crawl:hellofresh` next runs Sun 2026-07-26 03:00 UTC