A production scraper for Mariano's that collects the full catalogue for a configured store, checks the store before and after the run, validates every record against the required 22-field schema and only then publishes. Built to run on Azure twice a week without anyone watching it.
The store is resolved by its Kroger location ID and checked against the configured chain and ZIP before any product is collected and again after the last one. Every price below came from a request carrying that store's ID; a mismatch stops the run before publication.
Unique records per category from Marianos_Category_URLs.csv. Click a category to see its subcategories.
Share of records with a value in each field. Empty strings, never nulls. Fields the rules require to stay empty at the scraper stage are marked, and so are fields the source does not provide.
A stratified sample from the run, exactly as written to JSON.
| Product | Weight | Price | UPC | Store brand |
|---|
The sample file uses all 22 fields in the required order. The report holds total and per-category counts, duplicates, field completeness, runtime, request counts and the validation result.
Every run is validated as a whole before it can replace the last good dataset. A blocked, thin, partial or wrong-store run is kept for diagnosis and the previous good file stays where the pipeline reads it.
One Bicep template and one command install everything in your subscription. No secrets in code, no keys in config: a managed identity reads Key Vault, pulls the image and writes to Blob Storage.
Blob lease per store. A second start exits; runs never overlap.
Location ID, chain and ZIP must match before collection.
Category → subcategory → term units, paced, retried, each one checkpointed.
Same store check after the last unit.
Schema, duplicates, yields, canaries, price context, completion.
Only a passing run replaces latest/.