fix(scraper): stop restarts queueing duplicate crawls #20

Merged

TimCane merged 2 commits from fix/duplicate-startup-crawls into main

2026-07-19 19:06:14 +01:00

TimCane commented

2026-07-19 18:56:52 +01:00

(Migrated from github.com)

Summary

skip the startup seed when hangfire already holds a crawl of the source (enqueued, fetched, processing, or scheduled for retry): the interrupted job survives a restart and resumes by itself, so every redeploy was adding another full crawl to the queue
turn on the sliding invisibility timeout: crawls run for hours, and on the fixed 30 minute timeout a job still being worked goes back on offer, littering its history with ghost processing entries

Test plan

dotnet test: 107 passed, including 10 new PendingCrawlTests over a fake IMonitoringApi
delete the piled-up duplicate jobs from the dashboard (#2-#5 and all but one unstarted crawl per source), then redeploy and confirm the startup log says 'Skipped seeding' instead of queueing another

## Summary - skip the startup seed when hangfire already holds a crawl of the source (enqueued, fetched, processing, or scheduled for retry): the interrupted job survives a restart and resumes by itself, so every redeploy was adding another full crawl to the queue - turn on the sliding invisibility timeout: crawls run for hours, and on the fixed 30 minute timeout a job still being worked goes back on offer, littering its history with ghost processing entries ## Test plan - dotnet test: 107 passed, including 10 new PendingCrawlTests over a fake IMonitoringApi - [ ] delete the piled-up duplicate jobs from the dashboard (#2-#5 and all but one unstarted crawl per source), then redeploy and confirm the startup log says 'Skipped seeding' instead of queueing another