What is GPTBot?
Quick Summary
GPTBot is OpenAI's background web crawler used to discover and index public web pages for training data.
Detailed Explanation
GPTBot is OpenAI's background indexer. It parses public HTML documents and text blocks to expand model training sets (like GPT-4 and subsequent models). GPTBot follows standard robots.txt directives and ignores pages containing noindex meta configurations.
Track crawler traffic in real-time
Find out exactly when chat models or search bots are reading your content. Try AIReady free.