Three separate decisions
- FetchMay the crawler request the page?You control: robots.txt, CDN, firewall
- Index or retrieveDoes the search product store or pull the page?Platform decides
- RecommendDoes the answer name or recommend you?Platform decides, from the sources it finds
1. Crawl access
OpenAI documents three agents. OAI-SearchBot surfaces websites in ChatGPT’s search features and follows robots.txt. GPTBot collects content that may be used for model training; disallowing it is the training opt-out. ChatGPT-User fetches pages when a user asks, and robots.txt may not apply to it. These are independent choices, so you can allow search while opting out of training.
Check the effective rules: a bot-specific group in robots.txt replaces the wildcard group for that bot, so it can silently drop restrictions you meant to keep. If a CDN or firewall blocks unfamiliar traffic, compare it against the IP ranges OpenAI publishes before allowing it. OpenAI says robots.txt changes for search can take about 24 hours to take effect.
2. Pages that answer the question
- One page per service you want to be found for, titled in the customer’s words.
- Named facts: who does the work, their credential, the address or service area, hours and how to book.
- Plain HTML for the core answer, so it is readable without scripts.
3. References elsewhere
Answers often cite directories, associations, news and review sites. Make sure the profiles you control are accurate and link to the right page. Do not buy mentions or reviews.
4. A documented test
Write the question with the city in it, note the date, whether search was on, and your location settings. Save the answer and every cited URL. Repeat after changes. Our checklist has a worksheet for this.
Primary sources
- OpenAI: Overview of OpenAI crawlers
- OpenAI Help: ChatGPT search
- Google Search Central: Optimizing for generative AI in Search
Checked on 18 November 2026. Platform behaviour changes; we re-check these references when we revise the page.