Conclusion First
In generative search, for your content to “appear in the answer,” it often has to complete two steps first: GeneralSearch finds you, then Fetch reads you, and only then does it come down to GEO making the model dare to cite you.
Therefore, GEO is not just about “writing more like an answer”; it is also about turning the website into a stably crawlable, parseable, snippet-reusable, and attributable knowledge interface.
If you break the problem into three engineering questions: findable (GeneralSearch) → readable and read correctly (Fetch) → cited accurately and traceably (GEO), you will locate growth levers and blocking points faster.
Key Takeaways
- GeneralSearch determines the candidate pool: if you are not in the top results/candidate sources, models usually will not “go to the trouble of reading you.”
- Fetch determines usability: 403s, challenge pages, forced login, and pure CSR (no body text in HTML) can directly make you “read as empty.”
- GEO determines citation rate: AI prefers extractable answer blocks, clear entities, and verifiable evidence over long marketing narratives.
- Search crawlers vs user-triggered Fetch must be managed separately: for example, OpenAI’s OAI-SearchBot (search) and ChatGPT-User (user-triggered access) serve different purposes; Perplexity also distinguishes PerplexityBot from Perplexity-User, and user-triggered fetch often does not follow robots.
- robots allowing access does not mean the server lets it through: the real blocking points are often WAF/CDN bot management and rate limiting.
- Generative search performs “multiple searches”: Google publicly describes that AI Overviews/AI Mode may use query fan-out (multiple related searches) to find supporting pages; covering sub-questions and synonymous expressions will matter more.
- Observability is a core capability of GEO: build a loop with logs (Fetch success rate/status codes/latency) + a fixed question set (coverage rate/citation rate/statement consistency), rather than looking at clicks only.
- To “avoid being displayed,” do not rely on robots alone: noindex/preview control has strict prerequisites (crawlers must be able to access the page to read noindex).
1. Why Bring Fetch and GeneralSearch into the GEO Discussion
In UME’s framework, the goal of technical GEO is not “creating an llms.txt,” but turning the site into a “knowledge interface” that generative engines can stably crawl, correctly parse, reuse in snippets, and reliably attribute.
Fetch and GeneralSearch happen to correspond to the two most easily overlooked pieces of “foundation engineering” in the generative retrieval chain:
- GeneralSearch: responsible for “finding candidate sources” (discovery and recall). On different platforms it may be called
web_search, Search API, query fan-out, or a more general “search tool.” - Fetch: responsible for “opening and reading a URL” (a single request/crawl/parse). Google’s definition of “fetchers” is very close: like
wget, they usually represent a single request initiated by a user.
GEO’s practical constraint is: you cannot optimize only “content expression” while ignoring “whether you are found and whether you can be read.” Because the default strategy of many generative systems is: search first, then pick a few links to Fetch.
2. Three Concepts in One Table
| Concept | Role in the chain | Typical failure signal | Lever you can optimize directly | Recommended core metrics |
|---|---|---|---|---|
| GeneralSearch | Discover/recall candidate pages (enter the candidate pool) | “AI never mentions you/always picks competitor sources” | SEO foundation, topic coverage, internal links and Hub, title/summary engineering | Candidate pool coverage, top-N entry rate, topic visibility |
| Fetch | Open the URL and obtain parseable content | 403/429, challenge pages, empty-shell HTML, timeouts, overly long redirect chains | robots + WAF/CDN allowlisting, SSR/pre-rendering, performance and stability, canonicalization | Fetch success rate, 200 share, TTFB/latency, visible body text parse ratio |
| GEO | Make content selected and cited in the form of “evidence/answer blocks” | Crawled but not cited; inaccurate citations; severe de-branding | Answer engineering, entity engineering, evidence engineering, Schema, citable anchor blocks | Citation rate, Share of Answer, statement consistency, correction loop time |
This table can serve as an on-site “issue attribution” framework: not being cited ≠ content is bad; it may simply be stuck at GeneralSearch or Fetch.
3. The Default Path of Generative Search: GeneralSearch → Fetch → Generate → Cite
Abstracting the typical behavior of generative systems into pseudocode makes it easier to understand what you need to optimize:
User question Q
↓
GeneralSearch(Q) → return candidate URL list (possibly multiple rounds of query fan-out)
↓
Select top-K URLs
↓
Fetch(URL) → get HTML/body text → parse/chunk/re-rank
↓
Generate answer → select evidence snippets → cite/attributeThere are two key points:
- Search becomes “multiple searches”
Google explicitly mentions that AI Overviews/AI Mode may use query fan-out: searching multiple routes around subtopics and then aggregating supporting pages. For content teams, this means: betting on one main keyword is not enough; you must cover sub-questions, synonymous phrasings, and comparison-type questions. - Fetch is often “small in number and picky”
Generative systems usually do not “deep-crawl your whole site” like traditional search engines; instead, they pick a small number of pages to read around the current question. This is also why UME keeps emphasizing P0: do not let AI crawlers hit 403s; robots allowing access does not mean the server allows it.
4. Get Selected by GeneralSearch: Candidate Pool Engineering
4.1 What you really need to win is the “eligibility to be selected as a source”
In UME’s framing, GEO “inclusion” is more like entering the candidate pool of the generative retrieval chain and being reliably cited in answers.
This means: SEO is still the foundation. Google also makes clear: the best practices for AI features remain the existing SEO fundamentals, with no additional “mystery optimization” required; but the prerequisite is that pages can be indexed, can produce snippets, and meet technical requirements.
4.2 Content and structure strategies on the GeneralSearch side
With the goal of “making it easier for systems to find you,” the executable actions UME recommends are:
- Build topic Hubs (aggregation pages) instead of scattered articles: use a Hub to connect concepts, tutorials, comparisons, cases, and FAQs into a traversable knowledge network.
- Turn keyword lists into “question graphs”: What/Why/How/Compare/Metrics/Pitfalls, and write users’ real question phrasings into H2/H3 headings.
- Titles and summaries must be directly extractable: generative engines first look at a small number of signals (URL, title, description/summary), and quality affects “whether it is worth Fetching.”
- Cover synonyms and aliases: especially entity names, terminology, English abbreviations, and industry colloquialisms, to avoid “searchable but not matching you.”
5. Make Fetch Able to Read You—and Read You Correctly: The P0 Checklist for Technical GEO
5.1 Treat Fetch as a “more fragile crawler/browser”
Google describes fetchers as: like wget, usually making only a single request; such tools often have lower expectations for retries and fault tolerance.
And in the AI ecosystem, these “user-triggered access” agents/fetchers may even not be bound by robots:
- OpenAI explains that ChatGPT-User is user-triggered access, not used for automated crawling, and robots rules may not apply.
- Perplexity’s documentation also states: Perplexity-User supports user actions, and “because it is a user-requested fetch, it generally ignores robots.txt.”
The lesson for content teams: you cannot just “allowlist a certain bot”; you must also make pages reliably readable for “browser-like access.”
5.2 P0: Don’t let AI bots hit 403
UME lists this as the first item in its technical GEO checklist: robots.txt allowing access does not mean the server allows it; WAF/CDN bot rules are the common blocking points.
Implementation actions (by priority):
- Bucket logs by UA: count 200/301/403/429/5xx.
- Locate the blocking layer: application (Nginx) → CDN → WAF → bot management/challenge pages.
- Use “IP range + UA” combinations for allowlisting: allow public IP ranges while still rate-limiting abnormal frequencies (especially since Perplexity’s documentation provides WAF configuration ideas).
5.3 P0: Core content must be “visible in the HTML”
Technical GEO explicitly requires: body text, definitions, key tables, and FAQs must be visible in the HTML; pure CSR must at least have SSR/pre-rendering as a fallback.
Engineering acceptance methods (without complex tools):
- Whether key paragraphs are visible in
view-source: - Whether the HTML fetched by
curlis the body text, rather than an empty shell/challenge page
5.4 Keep redirect chains within 1–2 hops
UME points out: different crawling agents have different retry/fault-tolerance capabilities, and you cannot assume they are all like Googlebot; key pages should reach a final 200 within at most 1–2 redirects.
5.5 Use HTTP headers to make “updatability” engineering, not a slogan
In its crawling infrastructure documentation, Google mentions support and recommendations for caching and mechanisms such as ETag and Last-Modified, which affect crawl efficiency and update recognition.
In the GEO context, this is part of “freshness engineering”: making it easier for systems to confirm that you are the latest version, rather than repeatedly crawling old content.
6. Make the Model Dare to Cite You: Answer Engineering × Entity Engineering × Evidence Engineering
UME’s inclusion logic emphasizes: generative systems typically go through discovery → crawling → parsing → chunking → indexing → retrieval → re-ranking → generation → citation; every link has controllable levers.
To turn “read” into “cited,” the most effective approach is not writing another article, but engineering content into reusable components: SSOT → entity pages → evidence cards → answer blocks → distribution and evaluation loop.
6.1 Answer blocks are the smallest delivery unit of GEO
Recommended answer block structure (usable directly as UME’s on-site writing standard):
- Question sentence (H3): write the heading as the user’s question
- Short answer (1–2 sentences): can be taken away directly
- Key points (3–5): why/how/what to watch out for
- Boundary conditions: applicable and not applicable
- Evidence slot: data, cases, sources (pointing to evidence cards)
6.2 Entity consistency determines whether AI “recognizes you”
Content engineering emphasizes: naming and relationships for organizations/products/authors/terms must be stable and machine-readable; entity definition drift is a common cause of GEO failure.
Recommended minimum entity assets:
- Organization (organization page)
- Product/Service (product/service page)
- Glossary (terminology page)
- Author/Team (author/team page)
6.3 Evidence cards determine whether AI “trusts you”
Evidence card template (reusable component):
- Conclusion sentence
- Data/screenshots/comparisons
- Definitions: time range, metric definitions, sample
- Method: how it was obtained, constraints
- Source: traceable citations
- Update: last update time and owner
7. Fault Tree: Is It Stuck at GeneralSearch, Fetch, or GEO?
| Symptom | More likely root cause | Quick verification | Resolution action |
|---|---|---|---|
| AI always picks competitor sources | Cannot get into the GeneralSearch candidate pool | Look at source distribution with a fixed question set; check indexing and Hub coverage | Build a Hub + subpage matrix; add synonymous phrasings; improve visibility |
| Sometimes mentions you, sometimes not | Candidate pool fluctuation + insufficient evidence | Check whether stable entity pages/evidence slots exist | Unify naming on entity pages; add evidence cards to key conclusions |
| AI visits your site but crawls empty content | Fetch failure or parse failure | Check logs for 403/challenge pages/JS; verify with curl/view-source | Allow through WAF/CDN; SSR/pre-rendering; remove challenge pages |
| Cited but the link cannot locate the passage | Missing citable anchors | Check whether paragraph ids/table of contents exist | Add anchor ids to key paragraphs; adopt a “cite this section” strategy |
| Cites you but states it incorrectly | Unclear entities/boundaries or version drift | Use a regression question set to compare statement consistency | Defensive GEO: boundary conditions, version logs, correction loop |
8. Monitoring and Metrics: Make GEO an Auditable System
8.1 You need two types of “observation panels”
- Fetch panel (site availability)
- Fetch success rate (share of 200s)
- Share of 403/429/5xx
- Average latency/TTFB
- Distribution of redirect chain length
- Citation panel (answer share)
- Question coverage
- Citation rate
- Share of Answer (the proportion in which you appear/are cited in answers)
- Statement consistency (consistency for the same question across time/models)
8.2 Use trackable parameters to identify “return visits brought by AI search”
OpenAI’s publisher FAQ mentions that referral URLs from ChatGPT search automatically carry utm_source=chatgpt.com, making them easy to track in analytics tools.
We recommend incorporating this into your UTM rules and channel dashboards, and building a separate report for the funnel “answer citation → return visit → conversion.”
9. Permissions, Control, and Boundaries: robots, noindex, and “User-Triggered Fetch”
This section is critical in GEO practice, because many teams mistakenly treat robots as the “final control switch.”
9.1 Managing crawlers by purpose is the new normal
Taking OpenAI as an example:
- OAI-SearchBot is used for search result presentation;
- GPTBot is used to crawl data that may enter training;
- ChatGPT-User is used for user-triggered access, and robots rules may not apply; robots updates may take about 24 hours to take effect.
Perplexity is similar:
- PerplexityBot is used for search results;
- Perplexity-User supports user actions and “because it is a user-requested fetch, generally ignores robots.txt.”
Conclusion: you need to split “search presentation,” “training use,” and “user-triggered access” into separate strategy decisions, then implement them through robots/WAF/page controls.
9.2 To avoid being displayed, don’t rely on robots alone
OpenAI’s FAQ mentions: even if you block OAI-SearchBot, if the system obtains the URL from third-party search or other pages, it may still display only the “link and title”; if you do not want this, you need noindex—but the prerequisite is that crawlers can access the page to read the tag.
Google’s noindex documentation also states clearly: for noindex to take effect, the page must not be blocked by robots.txt; otherwise crawlers cannot see noindex and the page may still appear.
Evidence and Boundaries
Evidence sources
- UME’s definition of technical GEO and its P0/P1 engineering checklist (crawlable, parseable, citable, observable).
- UME’s breakdown of the GEO inclusion chain and the “crawled → cited → converted” framework.
- OpenAI’s official distinction between the purposes of OAI-SearchBot, GPTBot, and ChatGPT-User, and the robots effective delay.
- OpenAI’s publisher FAQ on “how to appear in ChatGPT search,” “noindex and display control,” “utm_source=chatgpt.com,” and Atlas accessibility recommendations (ARIA).
- Google’s definitions of crawlers and fetchers, its explanation of user-triggered fetchers, and technical attributes such as caching/protocols.
- Google’s explanation of AI features (query fan-out, multi-round search, SEO fundamentals still apply).
- Perplexity’s description of PerplexityBot and Perplexity-User purposes, and its note that user-triggered fetch often ignores robots.
Applicable and non-applicable boundaries
- Applicable: content websites, B2B corporate sites, tools/knowledge bases, documentation centers, and brand/product teams that want to improve “being cited/being represented.”
- Not applicable or requiring caution: pages with forced login, heavy personalization, and highly dynamic content where SSR/pre-rendering is not desired; and assets that, for copyright/compliance reasons, clearly must not be crawled/displayed (define authorization boundaries and a noindex strategy first).
Terminology Definitions
- GEO (Generative Engine Optimization): an optimization system for generative AI/conversational search, with the goal of making models more inclined to cite you and attribute correctly when answering related questions.
- GeneralSearch: the search tool capability that generative systems use to “find candidate web pages/data sources” (names differ across platforms).
- Fetch: the crawl/read tool capability that generative systems use to “open a URL and read its content”; behaviorally similar to what Google calls fetchers (representing a single user request).
- Candidate pool: the set of sources that can be retrieved and re-ranked in the generative retrieval chain; without entering the candidate pool, citation is out of the question.
- Answer Block: the smallest semantic unit that can be independently extracted and reused (short answer + key points + boundaries + evidence slot).
- Evidence Card: componentizing credibility (definitions, method, source, timestamp, owner) so that AI is more willing to cite.
Key Entity List
- Brand and site: UME, growume.com
- Core methodologies: GEO, technical GEO, content engineering, SSOT, entity pages, evidence cards, answer blocks, Share of Answer, observability
- Key technical objects: robots.txt, WAF/CDN, SSR/pre-rendering, Schema.org, ETag/Last-Modified, UTM
- Key ecosystem entities: OpenAI (OAI-SearchBot, GPTBot, ChatGPT-User), Perplexity (PerplexityBot, Perplexity-User), Google (AI Overviews, AI Mode, fetchers, Google-Extended)
发表回复