CRAWLER POLICY
source-trace-bot
WHAT THIS IS
GEOscanAI's Source Trace product asks AI engines real questions, then fetches the pages those engines cited or that independently corroborate what they said, to build a public record of which sources actually back a claim. This fetcher — identified by the User-Agent below — is what does that fetching. It is not a search-index crawler and does not crawl your site generally; it fetches a specific, small set of URLs per research question, once, to read the page text for that one question.
SourceTracePipeline/1.0 (+https://geoscanai.co/source-trace-bot; research/product prototype; respects robots.txt; no login/cookies)
WHAT IT FETCHES, AND WHY
Candidate URLs come from two places: links an AI engine cited directly in its answer, and a small number of supplementary search results for the same question. Each page is fetched once, its text is matched against the specific claims extracted from the engines' answers, and short, attributed quotes (with a link back to the original page) are what appear on the resulting trace page — never the full fetched text. The fetch exists to verify attribution, not to republish or index your content.
ROBOTS.TXT
robots.txt is checked before every fetch and honoured as a hard block — not a suggestion, not overridden under any condition. If your robots.txt disallows this user agent, or disallows all bots, the page is never fetched. This is checked once per host per research run, not once per URL.
RATE LIMITING
At most one request to your host at a time, with a minimum 1.5 second gap between requests to the same host — or your site's own declared Crawl-delay, whichever is longer, if your robots.txt sets one.
HOW TO BLOCK IT
Add a rule to your robots.txt for the user-agent token SourceTracePipeline:
User-agent: SourceTracePipeline Disallow: /
This is checked and honoured before any fetch, as above.
CONTACT
Questions, takedown requests, or anything else about this fetcher: hello@geoscanai.co