fix: make the first live crawl actually work #10

Merged

TimCane merged 3 commits from fix/scraper-db-address into main

2026-07-19 16:10:48 +01:00

TimCane commented

2026-07-19 16:08:58 +01:00

(Migrated from github.com)

Summary

Everything here was found by running the first real crawl behind the VPN. All four would have shipped silently.

Gousto detail slug. List entries carry a category-prefixed path (/chicken-recipes/chicken-date-tamarind-curry), so Trim('/') produced chicken-recipes/... and every detail fetch 404'd. The first crawl stored 5 list pages and zero recipes while reporting success. The detail payload's own url has no prefix, which is exactly what the old tests used - they were written against the wrong sample. Slug tests are now driven by the captured list page
Run counts never persisted. documents_fetched and documents_changed were set on the returned object after CompleteAsync, so every stored run read 0 - a crawl that fetched nothing looked identical to one that worked
Postgres unreachable from the VPN namespace. gluetun replaces Docker's resolver with its own DNS-over-TLS, so db does not resolve there. Turning DoT off would fix names by sending every public lookup through the host resolver, outside the tunnel - so both services connect to a pinned address instead. This also fixes the MCP container, where a host resolv.conf carrying ndots:0 plus a search domain broke short names independently
Published ports are configurable. A running devcontainer already holds 5205/5206
.env.example notes that gluetun refuses an IPv6 interface address unless the daemon has IPv6 enabled - which is how the very first up failed

Verified against the live stack

Tunnel up, exit IP confirmed London GB before any crawl ran
Routing checked directly: 172.28.0.10 goes out eth0, 1.1.1.1 goes out tun0
crawl gousto --max 6 -> 6 fetched, 5 changed, no fetch failures, 5 real recipe documents stored
normalize gousto -> 5 done, 6 not applicable, 0 failed
Queried through the MCP server over HTTP: search_recipes for "steak" returned two real Gousto recipes with prep times and calories; get_recipe returned allergens, pantry items, 8 steps, and the caveat that Gousto publishes no quantities
115 tests pass (70 unit, 45 integration)

## Summary Everything here was found by running the first real crawl behind the VPN. All four would have shipped silently. - **Gousto detail slug.** List entries carry a *category-prefixed* path (`/chicken-recipes/chicken-date-tamarind-curry`), so `Trim('/')` produced `chicken-recipes/...` and every detail fetch 404'd. The first crawl stored 5 list pages and zero recipes while reporting success. The detail payload's own `url` has no prefix, which is exactly what the old tests used - they were written against the wrong sample. Slug tests are now driven by the captured list page - **Run counts never persisted.** `documents_fetched` and `documents_changed` were set on the returned object after `CompleteAsync`, so every stored run read 0 - a crawl that fetched nothing looked identical to one that worked - **Postgres unreachable from the VPN namespace.** gluetun replaces Docker's resolver with its own DNS-over-TLS, so `db` does not resolve there. Turning DoT off would fix names by sending every public lookup through the host resolver, outside the tunnel - so both services connect to a pinned address instead. This also fixes the MCP container, where a host `resolv.conf` carrying `ndots:0` plus a search domain broke short names independently - **Published ports are configurable.** A running devcontainer already holds 5205/5206 - `.env.example` notes that gluetun refuses an IPv6 interface address unless the daemon has IPv6 enabled - which is how the very first `up` failed ## Verified against the live stack - Tunnel up, exit IP confirmed London GB before any crawl ran - Routing checked directly: `172.28.0.10` goes out `eth0`, `1.1.1.1` goes out `tun0` - `crawl gousto --max 6` -> 6 fetched, 5 changed, no fetch failures, 5 real recipe documents stored - `normalize gousto` -> 5 done, 6 not applicable, 0 failed - Queried through the MCP server over HTTP: `search_recipes` for "steak" returned two real Gousto recipes with prep times and calories; `get_recipe` returned allergens, pantry items, 8 steps, and the caveat that Gousto publishes no quantities - 115 tests pass (70 unit, 45 integration)