Responsible data
How we collect, and what we won’t.
Gristmill collects publicly available pages at polite rates and strips personal data it does not need. This page is a placeholder policy to adapt, not legal advice.
Not legal advice
Whether scraping a source is allowed depends on the site, the data, your use and your jurisdiction. Template text only. Have your own counsel review your policy and each source.
01
robots.txt
We read robots.txt for every source before a run and honour disallow rules and crawl-delay for our user agent. Sources that opt out are not offered as datasets.
User-agent: *
Disallow: /account/
Crawl-delay: 5
02
Personal data (PII)
Datasets are built around businesses, products and listings. Names, emails and phone numbers of private people are dropped or hashed before delivery. We do not sell profiles of individuals.
reviewer_name → dropped
email → dropped
review_text → kept (PII scrubbed)
03
Rate limits
Each source gets a request budget and backoff on errors, so a run never looks like an attack. Busy hours are avoided where we can.
max_concurrency: 4
requests_per_min: 60
backoff: exponential
04
No logins, no paywalls
We only collect pages anyone can see without an account. We do not bypass logins, paywalls or CAPTCHAs.
auth_required: false
paywalled: skip
05
Takedowns
Site owners can ask us to stop collecting from their site. We pause the source within two business days while we review.
contact: /contact
subject: takedown
06
Retention
Raw pages are deleted after parsing. Delivered rows stay in your workspace until you delete them or close the account.
raw_html: 7 days
rows: until deleted
Identify us, or block us.
Our crawler sends a stable user agent with a link back to this page. Block it in robots.txt and we stop.
User-Agent: GristmillBot/1.0 (+https://gristmill.example/responsible-data)
# placeholder value, replace with your own