Dataset and methodology changelog
Dated changes to coverage, data access and measurement rules. Changes to targets or rules can affect comparisons across dates; see observation dates on each dataset.
- Corrected the content check for the two anti-bot challenge targets: it now looks for the page's success heading. Successful (2xx) responses on those targets recorded before the correction are excluded from success rates, because they can't be re-checked; blocked responses remain counted.
- Expanded the API target set with GOV.UK organisations (server-rendered), crates.io serde (JavaScript), Node.js About (CDN), data.gouv.fr licences (French), and RFC 9110 (large document). Target additions do not imply observations are already available.
- Authentication-unavailable responses are excluded from success rates where they represent unavailable measurement access, rather than a target response. Existing provider-specific treatment remains unchanged.
- Added Scrapfly observations to the API benchmark; vendor presence does not imply every mode has measurements.
- Unavailable premium responses and measurement-limit responses are excluded from success rates rather than treated as vendor failures. Legacy 60-second premium timeouts remain excluded.
- Restricted current completion comparisons to fetched observations on matching dates with valid public-run counts; carried-forward and unknown-date evidence is not a current recommendation.
- Added daily API history by UTC date, provider, target and exact mode. Missing days remain unavailable; the public attempt CSV reproduces the displayed window.
- Added Dataset structured data to the Store open-data page and downloadable Store observation history.
- Added public API attempt CSV downloads.
- Clarified recorded billing events and list-tier pricing: rates per 1,000 events are not universally prices per result. Unknown amounts, zero rates and explicit free models remain distinct.
- Added per-tool observation dates and carried-forward status to tables and downloads; dataset file dates do not date each tool’s evidence.
- Added matched ScraperAPI versus ScrapingBee comparisons.
- Expanded website grouping with domain-based target discovery. Personal-contact harvesting tools remain excluded.