Conversations about AI search optimization usually start with writing style. Lead with the answer, turn headings into questions, add llms.txt. All are easy, but few have been shown to work.
To be cited, a page must first be readable and indexed. Bot blocks, changed URLs and outdated figures creep in through everyday updates. So the foundation is not a one-off rewrite but an operation that keeps pages citable.
This article separates what has been verified from what has not, then covers check frequency, automation and cost. Overall search monitoring is covered in "Search Console and GA4 monitoring automation", and structured data upkeep in "Automating structured data maintenance".
Main sources for this article: Google Search Central's guide to generative AI features (updated 10 July 2026), Search Console Help, Pew Research Center (2025), three Ahrefs studies (2025–2026), two SE Ranking studies (llms.txt and AI traffic) and the European Commission's announcement of 9 December 2025. All were checked on 1 November 2026.
| What you will learn | Short answer |
|---|---|
| What has changed | Fewer clicks, more official measurement (section 1) |
| How citation works | Readable → indexed → candidate → cited (section 2) |
| What has evidence | Not llms.txt; research comes with conditions (section 3) |
| Why pages drop out | Six patterns, such as bot blocks and stale facts (section 4) |
| What to watch and fix | Weekly reports and quarterly test questions (sections 5–8) |
| Cost and when to continue | Three tiers, plus scale-up and stop rules (sections 11, 14) |
1. What is happening: fewer clicks, more ways to measure
More searches now show AI answers, those answers reduce clicks, and in 2026 official measurement began to arrive. The figure below lists the main events.
The drop in clicks varies by study
Pew Research Center analyzed the March 2025 browsing of 900 US adults. Of 68,879 searches, 18% produced an AI summary. People clicked a traditional result on 8% of visits with a summary, against 15% without one. Links inside the summary were clicked on 1% of visits.
Ahrefs' re-run found that, in December 2025, an AI Overview went with a 58% lower click-through rate for the top-ranking page. Its April 2025 study had found 34.5%. This is a desktop correlation, and Ahrefs sells SEO tools.
Google, for its part, says clicks from pages with AI Overviews tend to be "higher quality", without publishing figures. The table below collects numbers you can use for judgement.
| Metric | Figure | Source and period | Caveat |
|---|---|---|---|
| Clicks on traditional results | 8% (15% without) | Pew, March 2025 | 900 US adults |
| Clicks on summary links | 1% | Pew, March 2025 | Share of visits |
| Top-result CTR difference | −58% | Ahrefs, Dec 2025 | Desktop, correlation |
| Searches with AI Overviews | 25.11% | Conductor, Nov 2025 | 21.9M searches; varies by sector |
| AI share of traffic | 1.08% | Conductor, same | 13,770 domains in 10 sectors |
| AI share of traffic | 0.32% | SE Ranking, Jan–Apr 2026 | 101,574 sites |
All of them agree that AI referrals are growing but still small. Measure your own numbers before deciding how much to invest.
What small traffic shares mean
SE Ranking reports AI's share of traffic rising from 0.02% in 2024 to 0.32% in 2026. Conductor reports 1.08% overall and 2.8% in IT. Visits with no referrer are not counted, so the real figure may be somewhat higher.
For a site with 20,000 sessions a month, that means 64 sessions at 0.32%, 216 at 1.08% and 560 at 2.8%. So the realistic stance is: no big budget, but do not break anything.
2. How citation works: four gates
Most AI search products pull information from the web or a search index when composing an answer. This is called RAG (retrieval-augmented generation) or grounding. Google's guide says AI Overviews and AI Mode rely on its core Search ranking and quality systems.
A single question is also split into several related searches, known as query fan-out. A page therefore passes four gates in order. The figure below shows how far operations can protect it.
Failing gate 1 or 2 means zero visibility. Gates 3 and 4 are probabilistic, and even good pages are not always cited.
High rankings do not guarantee citations
Ahrefs' March 2026 study analyzed 863,000 keyword results and about 4 million URLs. Of AI Overview citations, 37.9% also ranked in the organic top 10, 31.2% ranked 11–100 and 31.0% ranked beyond 100. Ahrefs put the figure at about 76% in July 2025.
But both its parsing and the model (Gemini 3) changed in between. What can be said is that ranking well helps but is not sufficient.
Each service has its own entrance
Outside Google, each service has its own bots and its own index. The table below shows the differences confirmed in official documentation.
| Service | Search and answer bots | Training bot | Official measurement |
|---|---|---|---|
| Googlebot | Google-Extended (control token) | Search Console | |
| ChatGPT | OAI-SearchBot, ChatGPT-User | GPTBot | No official report |
| Claude | Claude-SearchBot, Claude-User | ClaudeBot | No official report |
| Perplexity | PerplexityBot, Perplexity-User | Says none for training | No official report |
| Copilot / Bing | Bingbot | Could not confirm | Bing Webmaster Tools |
Sources are the OpenAI, Anthropic, Perplexity, Google and Bing documents. A page can be readable to Google but not to ChatGPT.
3. What has been verified, and what has not
Many methods are sold under the names LLMO (large language model optimization) and GEO (generative engine optimization). Google's position is that AEO and GEO are still SEO for the search experience. The table below sorts the main claims by strength of evidence.
| Claim | Evidence | Verdict |
|---|---|---|
| llms.txt increases citations | Google ignores it; no correlation | No evidence |
| AI-specific schema is needed | Google says not needed | Not for Google |
| Content should be chunked | Google says not needed | Not for Google |
| Statistics and quotes raise visibility | Peer-reviewed, narrow conditions | Conditional |
| Newer pages get cited more | Varies; weak for Google | Conditional |
| Bought mentions increase citations | Google says spam systems apply | Avoid |
| Blocking bots reduces citations | Clear from docs and mechanics | Certain |
The only "certain" row is the simple one: blocked pages are not read. Most ways of increasing citations are conditional or unverified.
llms.txt: stating plainly that there is no evidence
llms.txt is a proposal to place a Markdown summary of a site for AI use. It is not an established standard. Google's guide says AI text files are not needed to appear, are harmless, and that Google Search ignores them.
SE Ranking's study covered about 300,000 domains, of which 10.13% had the file. Neither statistics nor a machine learning model showed any link between the file and citation frequency. Removing the variable actually improved the model's accuracy.
This is correlational and does not look inside each service, and giving AI tools a map of developer documentation may still be useful. But the sales line "it increases citations" has no evidence behind it, as of November 2026.
How to read "up to 40%"
The GEO paper, accepted at KDD 2024, is the source of the often-quoted "up to 40%". According to its tables, adding quotations raised a word-based metric by about 41%, statistics by about 31% and citing sources by about 28%. Keyword stuffing lowered it by about 8%.
However, most tests used a simulated setup where GPT-3.5 summarized the top five search results. A 200-query test on the real Perplexity gave mixed results. Pages ranked fifth tended to gain while those ranked first lost. The fair reading is only that "text showing evidence may have an advantage".
Freshness: it matters, but not everywhere
Ahrefs' study analyzed 16.975 million cited URLs. AI assistants such as ChatGPT tended to cite newer pages than organic search. The figure below compares average days since publication.
AI Overviews alone are not newer than organic results, and even the cited pages average about 2.9 years old. The study's author is also cautious about frequent updating. There is no need to force changes on old pages that are still correct.
4. Six ways pages drop out of citations
When citations fall, the cause is not necessarily AI's whims. The table below sets out six patterns, how to notice each and how to fix it.
| Pattern | What happens | How to notice | Fix |
|---|---|---|---|
| Bot blocks | CDN or WAF stops search bots | Logs, status checks | Separate rules by purpose |
| Rendering | Text appears only after JS | Check HTML without JS | Serve text from the server |
| URL changes | Old URL 404s; the source vanishes | Broken link checks | 301 redirects |
| Stale facts | Prices or rules have changed | Review-date tracking | Update facts and date |
| Fake updates | Only the date is new | Compare diffs and dates | Show real change dates |
| Duplication | Similar pages split the signal | Pages seen per question | Merge with a 301 |
Most come from internal changes or neglect. So start checking from your own site, before looking at AI answers.
Pattern 1: your CDN or WAF is stopping bots
On 1 July 2025, Cloudflare began asking new domains about AI crawlers at sign-up, blocking them by default without permission. Its July 2026 post says that choosing to block training also stops bots that combine search and training, such as Googlebot and BingBot. It also announced new defaults from 15 September 2026.
Checking robots.txt will not reveal this. OpenAI and Perplexity advise allowing their published IP ranges in your WAF. Security staff may have blocked them in good faith, so review the change history too.
Pattern 2: text appears only after JavaScript runs
In Vercel's analysis (December 2024), the major AI crawlers did not execute JavaScript. Treat this as dated. Even so, serving the text in your HTML avoids the problem whatever the specification says.
The same analysis found 34.82% of ChatGPT's fetches and 34.16% of Claude's hit 404 pages, against 8.22% for Googlebot. Redirect upkeep helps here too.
Patterns 3 to 6: quiet results of internal changes
Redesigns, content merges and plugin updates all move the foundation of citations. See "301 redirects in a website redesign" for URLs and "Content audit before a redesign" for merge decisions.
5. Watching: how to build measurement
Use three layers: official reports, your own logs and fixed test questions. The table below sets out what to watch and where.
| What | Where | Frequency | Limit |
|---|---|---|---|
| Google impressions | Search Console AI report | Weekly | Clicks in another report |
| Bing / Copilot citations | Bing AI Performance | Weekly | Position unknown |
| Bot access | Server logs | Weekly | Verify IPs |
| AI referrals | GA4 source report | Weekly | Misses no-referrer visits |
| Appearing in answers | Test questions | Quarterly | Changes every run |
Reliability runs from top to bottom. The first four fit into 15 minutes a week.
Search Console's generative AI performance report
Google announced this report on 3 June 2026 and rolled it out to all sites by 31 August. It shows impressions in AI Overviews and AI Mode by page, country, device and period. Search Labs experiments are not included.
If you cannot see it, your impressions may be too few, or you may have excluded your site in the generative AI control. The main metric is impressions, and it does not tell you which questions you appeared for.
AI Performance in Bing
Bing Webmaster Tools has shown citation counts in public preview since 10 February 2026, along with cited pages and a sample of grounding queries. In June, intents and citation share began rolling out.
Microsoft says these counts do not indicate placement or importance. They still show which pages are cited, for what kinds of phrases.
Measure test questions by appearance rate, not rank
SparkToro's study ran 12 prompts 60 to 100 times per service, 2,961 runs in total. The chance of ChatGPT or Google's AI returning the same list of brands twice in 100 runs was under 1%. The same order was rarer still.
"Third in AI" means almost nothing. What is worth tracking is the appearance rate: how many runs out of several mention you. Google also warns that no third-party tool has access to its internal ranking or AI systems.
As a DevNJ estimate, start with 10 questions on 3 services, 5 runs each: 150 runs, or 2 to 3 hours by hand. Five runs are noisy, so treat changes of 20 points or less as no change. Answers differ by country and language, so record European and Japanese audiences separately.
6. Fixing: deciding which pages to update
Rewriting every page on a schedule is not realistic. Focus on pages whose facts change and pages close to citations or revenue. The table below gives a guide by page type.
| Page type | Review frequency | Trigger | Owner |
|---|---|---|---|
| Pricing, services | On change and quarterly | Price or scope change | Business lead |
| Regulation explainers | Quarterly | Law change, regulator news | Subject reviewer |
| Articles using statistics | Every six months | New annual data | Editor |
| How-to guides | Every six months | Interface or spec change | Practitioner |
| Opinion articles | Yearly | Premises change | Editor |
| Case studies | Yearly | Permission, new numbers | Sales lead |
These frequencies are DevNJ estimates, not statistics. Pages with no named reviewer eventually go stale.
Dates should show real changes only
Google's guidance lists changing a page's date when the content has not substantially changed as something to avoid. It also says removing lots of old content to improve rankings will not work.
Sitemap lastmod is used only when consistently accurate. Updates to main content, structured data or links count as significant; a copyright year does not. Keep visible and structured dates consistent.
A CMS that bumps the date on every save is a common trap. Make "updated" a field people set only for significant changes.
Fix or write new?
If a URL has a track record, fix it first to keep its standing and links. Record which figure you changed, when and on what basis.
7. Writing: easy to verify beats "AI-friendly"
Google says you do not need to write in a special way for AI. Bing, on the other hand, recommends headings, tables, FAQs, data and sources. Since the advice differs, aim for what suits both: text that lets readers verify the facts. Give figures a source, period and conditions; define terms briefly in the first paragraph; state the conditions of comparisons. None of this is wasted if AI behavior changes.
A page with nothing original has no reason to be cited
Google's guide emphasizes a unique point of view and first-hand experience. Your own quote ranges, frequent enquiries and real failures are original information that even a small company has.
Mass generation, by contrast, needs care. Google's spam policies name generating many pages with AI without adding value as an example of scaled content abuse. If you produce at volume, build in a pre-publication quality gate.
Check how AI describes your company
Being cited does not mean being described correctly. In a Tow Center study (March 2025), eight AI search tools gave incorrect answers to more than 60% of queries about citing news articles. An EBU and BBC study was reported as finding significant issues in 45% of about 3,000 answers, and sourcing problems in 31%.
Both concern news, and later models may differ. Still, record whether test answers get your name, prices and service area right. If errors persist, consolidate that information on your site; a correction may not be picked up.
8. What to automate: machines and people
Judging facts cannot be handed to machines. The figure below divides the work into three.
Hand the right-hand three to a machine and plausible errors get published. The overall approach is covered in "Site operations automation starts with an inventory".
Minimum automated checks
Against your list of key pages, check these three things daily or on each publish.
# 1. Status when fetched with a search bot's name
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://example.com/page
# 2. Is the text in the server's HTML? (search for one sentence)
curl -s https://example.com/page | grep -c "A distinctive sentence from the page"
# 3. Has robots.txt changed since last time?
curl -s https://example.com/robots.txt | diff - robots.previous.txt
The first only claims a User-Agent, so it cannot reproduce a WAF that judges by IP. Use server logs to see what real bots received, and match them against the IP ranges each company publishes.
A daily, weekly, monthly and quarterly table
The table below is an example that works for small and mid-sized sites.
| Frequency | Task | Method | Time |
|---|---|---|---|
| Daily, on publish | Status, text, robots | Automated, alerts | Only on alert |
| Weekly | AI report, Bing, logs, GA4 | Auto tally, review | 15 minutes |
| Monthly | Pages with large swings | Candidates, review | 1 hour |
| Quarterly | Expired reviews, test questions | People | Half a day |
Times are DevNJ estimates. Speed (Core Web Vitals) monitoring can reuse the setup in our separate article.
9. Allow or block: separate search from training
"Should AI read our site?" is not one choice. Use in search answers and use for model training can be controlled separately with many services. The table below lists the means by goal.
| Goal | Means | Caution |
|---|---|---|
| Appear in ChatGPT search | Allow OAI-SearchBot | WAF must allow it too |
| Stop OpenAI training | Disallow GPTBot | Independent of search |
| Stop Gemini training etc. | Disallow Google-Extended | No effect on Google Search |
| Leave Google's AI features | Search Console exclude control | Lose traffic and impressions |
| Limit some snippets | data-nosnippet, max-snippet | Affects normal search too |
A policy of "appear in search but stop training" can be written as follows. This is only an example.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
ChatGPT-User and Perplexity-User fetch pages at a user's request, so each company says robots.txt may not apply. Blocking does not mean nothing is ever read.
Decide carefully on Google's exclude control
Excluding your site in the generative AI control removes it from AI features, including as grounding for answers. Google says this does not affect ranking elsewhere in Search. But it does not stop training, and similar content from others keeps appearing.
For a business site that wants more enquiries, the benefit of excluding is usually small. Before excluding, estimate the impact from your generative AI report impressions.
10. Differences between Europe and Japan
For sites running in both, the differences below make decisions easier.
| Item | EU and the Netherlands | Japan |
|---|---|---|
| AI Overviews launch | Dutch: May 2025 | August 2024 |
| Competition law | Commission investigation, Dec 2025 | JFTC reported as investigating |
| Copyright and training | Machine-readable opt-out | Debate centres on Article 30-4 |
| Labeling AI content | AI Act Article 50 (from Aug 2026) | No equivalent general duty confirmed |
| Unit of review | By country and language | Mainly Japanese, Japan |
This table is not legal advice.
Europe: investigation, opt-out, AI Act
On 9 December 2025, the European Commission opened a formal investigation into the terms under which Google uses publisher content for AI Overviews and similar features. One issue is whether publishers can refuse without leaving Search. There is no legal deadline.
On copyright, Article 4 of the DSM Directive lets rights holders reserve text and data mining rights by machine-readable means for online content. What counts as machine-readable divides the courts. TDMRep is one method, but it is not a W3C standard.
Article 50(4) of the AI Act requires disclosure for AI-generated text published to inform the public on matters of public interest. It does not apply where there is human review or editorial control and someone holds editorial responsibility. A law firm's note says Article 50 has applied since 2 August 2026. If you publish generated articles with no human review, ask a lawyer.
Japan: news publishers and competition law
The Japan Fair Trade Commission (JFTC) published ver. 2.0 of its generative AI report on 16 April 2026. It is also reported to be looking at AI search's use of news articles, and to have begun surveying about 370 news organizations in July 2026. We have found no decision yet that directly changes how ordinary businesses run their sites.
Multilingual sites
If your Japanese and English versions disagree on a number, the answer depends on which one the AI reads. For the translation process, see "Bilingual site translation automation".
11. Cost range: three tiers
The table below shows DevNJ estimates, not statistics. Outsourced work is priced at €80 an hour, and yen is converted at €1 = ¥175.
| Tier | Scope | Setup | Monthly work | Tools per month |
|---|---|---|---|---|
| A Manual | 12 checks in a spreadsheet | €320–640 (¥56,000–112,000) | €160–320 (¥28,000–56,000) | €0 |
| B Semi-automated | Auto checks, weekly tallies | €1,280–2,560 (¥224,000–448,000) | €320–640 (¥56,000–112,000) | €0–30 (¥0–5,250) |
| C With paid tools | Test tools, update tracking | €3,200–6,400 (¥560,000–1,120,000) | €640–1,280 (¥112,000–224,000) | €30–300 (¥5,250–52,500) |
Tier A assumes 4 to 8 hours of setup and 2 to 4 hours a month, B 16 to 32 and 4 to 8, and C 40 to 80 and 8 to 16. Time spent checking facts is not included. On sites with many legal or pricing pages, that becomes the largest cost.
Examples of paid tool prices
The table below shows tools that run test questions, where we could confirm official prices.
| Tool | Plan | Monthly (as listed) | Included |
|---|---|---|---|
| Otterly.AI | Lite / Standard / Premium | $29 / $189 / $489 | 15 / 100 / 400 prompts |
| Otterly.AI add-ons | AI Mode, Gemini, Claude | $9–$439 | Per service |
| Semrush | SEO / Starter / Pro+ (monthly) | €134 / €190 / €286 | Top two: 50 / 100 daily prompts |
Prices were checked on 1 November 2026 and vary with tax and billing terms. Peec AI, Ahrefs Brand Radar and others are left out because we could not confirm official figures.
Given SparkToro's findings, using a tool's rank display for decisions is risky. If you use one, follow the appearance rate over time.
How much is reasonable?
At 20,000 sessions a month with 1% from AI, that is 200 sessions. At a 1% enquiry rate, that is two enquiries a month. Depending on deal size, anything above tier B may be hard to recover.
That said, many of the 12 checks also help normal SEO and prevent redesign accidents. For outsourcing against in-house, see "Outsource, in-house or automate?".
12. Common mistakes
These are patterns, not specific companies.
| Mistake | Why it happens | What to do instead |
|---|---|---|
| Adding llms.txt and stopping | Easy to explain | Check items 1–6 first |
| Updating all dates at once | Oversimple reading of freshness data | Update only what changed |
| Judging by tool rankings | The number looks clear | Use appearance rate, official reports |
| Blocking all AI bots to be safe | Search and training conflated | Set policy by purpose |
| Publishing AI drafts unchecked | Wanting to save time | Name a fact reviewer |
If a vendor "guarantees citations", ask how. Only the AI provider could guarantee results. Comparing updated and untouched pages over 4 to 8 weeks only supports "it may have helped".
13. The first 90 days
Here are the steps in order. One person can manage them in one or two hours a week (DevNJ estimate).
- Week 1: see where you stand. Check the Search Console AI report and exclude control, and register with Bing Webmaster Tools.
- Week 2: confirm the route. Check robots.txt, CDN and WAF settings, and bot access in server logs.
- Weeks 3–4: choose your pages. Pick about 20 pages close to revenue or with changing facts, and check their HTML text and redirects.
- Weeks 5–6: add review dates. Record an owner and next review date for each page, and fix outdated figures.
- Weeks 7–8: fix dates and policy. Align CMS dates and lastmod with real changes, and write down your bot policy.
- Weeks 9–10: automate the watching. Run the three checks from section 8 daily and put weekly records in one format.
- Weeks 11–12: try test questions. Run 10 questions on 3 services, 5 times each, then decide how to continue using section 14.
The order matters. If you rewrite before you measure, you cannot tell what worked.
14. Worked examples (fictional)
Three fictional businesses show how decisions differ. None represents a real company.
| Item | Manufacturer A, Japan | Freelancer B, Netherlands | E-commerce C, Europe |
|---|---|---|---|
| Size | 10,000 sessions a month | 3,000 sessions a month | 150,000 sessions a month |
| Main readers | Japanese businesses | Residents of the Netherlands | Consumers across the EU |
| Suggested tier | A to B | A | C |
| Priority pages | Specs, case studies | Pricing, scope | Products, delivery, returns |
| Bot policy | Allow search; training internal call | Allow search | Allow search; training for legal |
A has high enquiry value, so tier B may pay back. B can manage by hand and should first match their KVK (Dutch Chamber of Commerce) details. For C, accurate product data feeds directly into AI answers.
Rules for scaling up and stopping
Set the rules in advance. The table below is an example; adjust the numbers to your business.
| Situation | Decision |
|---|---|
| AI above 2% of traffic, with enquiries | Move up a tier |
| AI below 0.5% for two quarters | Stay at tier A |
| Fact reviews miss their deadlines | Reduce the page list |
| Nobody opened the paid tool for three months | Cancel it |
A stop rule keeps the budget from growing every time a new term appears.
15. The 12-point checklist
Items 1–6 cover "readable and indexed", 7–10 "correct", and 11–12 "watching".
- 1. robots.txt treats search bots as intended
- 2. CDN and WAF do not stop allowed bots (confirmed in logs)
- 3. Search Console's generative AI control matches your policy
- 4. Key pages are indexed and snippets are not disabled
- 5. Text is present in HTML without JavaScript
- 6. Changed URLs have 301s, with no broken links to key pages
- 7. Pages with changing facts have an owner and an unexpired review date
- 8. Visible date, dateModified and lastmod match and reflect real changes
- 9. Structured data matches the visible text
- 10. Numbers agree across languages, and hreflang is reciprocal
- 11. AI report, Bing, logs and GA4 are recorded weekly
- 12. Appearance rate and description errors are recorded quarterly
If items 1–6 are missing, start there.
16. Summary
- No method can guarantee a citation. Google itself says meeting the requirements does not guarantee being shown
- AI referrals range from 0.32% to 1.08% across studies and are still small. Measure your own numbers before setting a budget
- There is no evidence that llms.txt increases citations. Google states that Search ignores it
- Operations can protect four things: readable, indexed, correct and honestly dated. Most of the six drop-out patterns come from internal changes
- Combine Search Console, Bing, logs and test questions. AI "rankings" change every run, so use the appearance rate
- Decide on search and training bots separately
- Keep fact checks, publishing, dates and bot policy with people, and automate the rest
For outside help building this, please contact DevNJ. We run Japanese and English websites from The Hague.
Sources
- Optimizing for generative AI features — Google: the guide (updated 10 July 2026)
- AI features and your website — Google: requirements and controls
- Generative AI performance reports — Google: launch and rollout
- Generative AI performance report — Google: impressions scope
- Search generative AI control — Google: exclude control and effects
- Creating helpful content — Google: standards on dates and authors
- Build a sitemap — Google: lastmod
- Byline dates — Google: date display and structured data
- Spam policies — Google: scaled content abuse
- Common crawlers — Google: Google-Extended scope
- AI Overviews in Dutch — Google: Dutch support (20 May 2025)
- Web Vitals — web.dev: Core Web Vitals thresholds
Other providers
- Crawlers — OpenAI: bot purposes
- Crawlers — Anthropic: bot purposes
- Crawlers — Perplexity: bot purposes
- AI Performance — Microsoft Bing: metrics and recommendations
- New AI Visibility Insights — Microsoft Bing: added features
- AI crawler defaults — Cloudflare: July 2025 change
- Your site, your rules — Cloudflare Blog: bots combining search and training (1 July 2026)
Studies and research
- AI summaries and clicks — Pew Research Center: click behavior
- Update: AI Overviews Reduce Clicks by 58% — Ahrefs: December 2025 data
- AI Overviews Reduce Clicks by 34.5% — Ahrefs: April 2025 study
- AI Overview citations and the top 10 — Ahrefs: overlap with rankings
- AI Assistants Prefer to Cite Fresher Content — Ahrefs: age comparison
- LLMs.txt study — SE Ranking: about 300,000 domains
- AI traffic research study — SE Ranking: AI traffic share
- AEO / GEO Benchmarks Report — Conductor: traffic and appearance rates
- GEO: Generative Engine Optimization — arXiv: KDD 2024 paper (full text)
- AI recommendation consistency — SparkToro: variance in answers
- Eight AI search engines — Columbia Journalism Review: citation accuracy
- News integrity in AI assistants — NPR: report on the EBU and BBC study
- The rise of the AI crawler — Vercel: December 2024 analysis
- llms.txt proposal: the proposal
Regulation
- Google and online content for AI — European Commission: announcement of 9 December 2025
- Directive (EU) 2019/790 — EUR-Lex: Article 4 rights reservation
- LAION Round 2 — Kluwer Copyright Blog: case law commentary
- TDM Reservation Protocol — W3C Community Group: community report
- Article 50 — EU AI Act Explorer: text
- AI Act transparency in force — Goodwin: application commentary
- Generative AI report ver. 2.0 — JFTC: published 16 April 2026
- JFTC survey of news organizations — Kumamoto Nichinichi Shimbun (Kyodo): press report
- AI Overviews launch in Japan — Current Awareness: launch in Japan (August 2024)
All sources were checked on 1 November 2026. Specifications change often, so check the official documents before implementing. Costs, effort, frequencies and decision rules are DevNJ's working estimates, not statistics. Statements on regulation are general information, not legal advice.
FAQ
- How do I get cited in Google AI Overviews? Is special optimization needed?
- Google says there are no additional technical requirements or special optimizations for AI Overviews or AI Mode. A page must be indexed and eligible to show with a snippet. Since 2026, the site must also be set to 'include' in Search Console's generative AI control. None of this guarantees a citation.
- Does adding llms.txt increase AI search citations?
- We found no evidence that it does. Google states that Google Search ignores such files. SE Ranking's study of about 300,000 domains found no correlation between having llms.txt and citation frequency. We cannot rule out other AI services using it in future, but it is a low-priority task.
- Where can I see my visibility in AI search?
- Search Console's generative AI performance report shows Google impressions by page, country and device. AI Performance in Bing Webmaster Tools shows citation counts and cited pages. ChatGPT and others have no official report, so use server logs, referral data and a fixed set of test questions.
- How often should I update articles to stay cited by AI?
- There is no universal answer. For pages whose facts change, such as prices or regulations, checking when something changes and once a quarter is often enough. In Ahrefs' study, URLs cited by AI assistants averaged about 2.9 years old. Google warns against changing dates when the content has not substantially changed.
- Can I block AI training but still appear in AI search?
- With many services, yes. OpenAI lets you control GPTBot (training) and OAI-SearchBot (search) separately, and Anthropic and Perplexity publish purpose-specific bots too. Google-Extended limits uses such as Gemini training, and it does not affect inclusion or ranking in Google Search.
- How much does AI search maintenance cost?
- As a DevNJ estimate, manual checks take 4 to 8 hours to set up and 2 to 4 hours a month, or €320 to €640 upfront at €80 an hour. A semi-automated setup is €1,280 to €2,560 upfront, and paid tools plus automation €3,200 to €6,400. AI referrals are still small, so it makes sense to build this into normal SEO work.
Planning a site that has to work in both markets?
Get in touch