Crawler policy

Clear behavior. Stable identity.

This policy describes how InfoPrice identifies and operates its public retail-information crawlers.

01

Identification

Our crawlers use stable, documented product tokens and User-Agent strings. InfoPriceBot identifies scheduled automated crawling; InfoPriceFetcher identifies explicit on-demand retrieval.

02

robots.txt

InfoPriceBot is designed to respect robots.txt directives for automated crawling. Site owners can allow or disallow the published product tokens through standard robots.txt rules.

03

HTTP response handling

InfoPrice honors Retry-After, backs off after HTTP 429 responses, and applies backoff after repeated 5xx responses.

04

Rate control

InfoPrice applies host-level request rate and concurrency controls. Limits are adapted operationally rather than represented by an unsupported universal numerical promise.

05

Public-data scope

Normal crawler activity is limited to publicly accessible retail information. It avoids authentication, account, checkout, form submission, and transactional workflows unless specifically authorized.

06

Privacy

Public crawler identity never includes customer or tenant identifiers, internal job IDs, user email addresses, proxy credentials, internal infrastructure details, or internal tracing identifiers.

07

IP verification

Published crawler network prefixes are available from ips.json. If no prefixes appear, production ranges have not yet been configured and traffic cannot be verified by that mechanism.