Google updates · News analysis

AI-Only Content or a Better Format? How to Audit the Difference

Machine-readable content is not automatically a “shadow web.” The key audit question is whether important claims remain publicly verifiable and consistent across formats.

Quick answer

Machine-readable content is not automatically a “shadow web.” The key audit question is whether important claims remain publicly verifiable and consistent across formats.

Contents

Chris Green’s “shadow web” concern is not simply that websites are making content easier for machines to read. It is that a second, less visible layer of information could emerge: one in which claims presented to AI systems differ from what people can verify on a public website.

That distinction matters. A Markdown response generated from a public page is not the same thing as an unlinked endpoint containing a different price or policy. And while a mismatch is a real governance problem, it is not automatically evidence of a search-policy violation or an AI-provider penalty.

The concern: a second information layer for AI

In an article published on October 22, 2026, SEO consultant Chris Green argues that machine-friendly versions of public information differ in kind from information made available only to AI systems. He discusses formats and mechanisms including llms.txt, agent-specific Markdown, information endpoints, structured feeds, and Cloudflare Markdown for Agents.

Green’s concern is that businesses may create material claims for AI systems that are harder for customers, journalists, or other observers to check. He also asks whether AI providers might eventually scrutinize or trust such claims differently. That future outcome is a hypothesis, not an established provider-wide rule.

The practical question is therefore not “Is this file for AI?” It is: Can a person verify its material claims, and do those claims agree with the public source?

Different format or different facts? The distinction that matters

FirstGlance classifies machine-accessible content by its relationship to the public source—not by its file extension or delivery method.

  1. Same source, alternate presentation. A system converts public HTML, or a shared source, into Markdown or another machine-readable format. Check that the conversion preserves the important claims and context.
  2. Additional but publicly verifiable detail. A feed or endpoint adds useful structure or detail, and the same material information is also available on an accessible public page.
  3. Separate AI-only information. Material claims appear only in a file, feed, or endpoint intended for machine access. This creates a verification and ownership risk; it does not, by itself, prove a policy violation.
  4. Audience-dependent contradiction. People and machines receive materially different claims. If content is deliberately varied to manipulate search rankings and mislead users, that may fit Google’s Search-specific definition of cloaking. The actual implementation and intent matter.

Consider a fictional refund example:

CasePublic pageMachine-accessible versionFirstGlance interpretation
A: format conversion“Refunds within 30 days.”Markdown conversion says “Refunds within 30 days.”Same claim, different presentation. Check the conversion.
B: public supporting detailA public FAQ says “Refunds within 30 days”; a linked policy explains eligibility and exclusions.A feed repeats the terms and links to that policy.Potentially useful structure, provided the terms are compatible.
C: unverified AI-only claim“Refunds within 30 days.”An unlinked endpoint says “Refunds within 60 days.”A material conflict to escalate and correct. This alone does not establish cloaking or a penalty.
D: deliberate audience splitPeople see one offer.A crawler or agent receives another offer specifically to influence search visibility.Could meet Google’s cloaking definition in a Search context, depending on the facts and intent.

The examples are illustrative, not evidence of ranking effects or AI-provider penalties. Their point is that a change in format is different from a change in facts.

What Google’s current guidance does—and does not—say

Google’s published guidance is specific to Google Search. It says Google does not use llms.txt or other special AI text files for Search visibility, including its generative Search features; maintaining them neither helps nor harms Google Search rankings. Google also says that AI Overviews and AI Mode do not require special structured data or additional technical requirements beyond Search eligibility and established practices.

That does not mean every AI system works the same way, or that machine-readable formats have no use. It means there is no basis for presenting llms.txt as a Google ranking lever. Google’s guidance points site owners back to established Search practices. For structured data in particular, Google says markup should accurately represent content visible on the page and should not mislead.

Cloudflare Markdown for Agents: a converted view of the source page

Cloudflare’s Markdown for Agents is a useful example of why file format alone is a poor risk test. When enabled, a client can request text/markdown; Cloudflare fetches the origin HTML and converts it on the fly. The documented output can include metadata from supported meta tags, converted body text, and the source page’s JSON-LD.

Cloudflare says its preprocessing removes non-content elements such as navigation, headers, footers, scripts, and styles, while JSON-LD is preserved separately. Its documentation describes conversion from HTML only and a 6 MiB decompressed-HTML limit, as of October 9, 2026.

This is a transformation of a page, not inherently a separate AI-only content estate. But a conversion is not a guarantee of semantic equivalence. Check the actual response for important text, context, links, metadata, structured data, and update state.

SEO history and the actual policy boundaries

Google’s documented milestones. Announcement dates are shown separately from later integration into core ranking systems.

Green’s concern draws on earlier SEO manipulation. The history is worth handling precisely: Google says Panda was announced in 2011 and became part of its core ranking systems in 2015; Penguin was announced in 2012 and integrated into core systems in 2016. Google announced the Helpful Content system in 2022, and in March 2024 said it had evolved into its core ranking systems.

On March 5, 2024, Google also announced its scaled-content-abuse spam policy. This is not simply another name for Panda or Penguin. Google defines scaled content abuse by its purpose and value: producing many pages primarily to manipulate Search rankings rather than help users. The policy applies whether content is made by automation, people, or both.

Google’s cloaking definition is also narrower than “content in a machine-readable format.” It concerns presenting different content to users and search engines with the intent to manipulate rankings and mislead users. A Markdown conversion, feed, or structured format is not cloaking merely because machines can consume it. Assess what differs, who receives it, and why.

Do AI providers distrust AI-only information? What is known and unknown

Public documentation offers some relevant signals, but not a universal answer. OpenAI describes ChatGPT Search selection in terms including meaning, intent, relevance, recency, and safety. It documents OAI-SearchBot as a crawler used to surface websites in ChatGPT Search.

Those statements do not establish a general penalty, distrust, or deprioritization of information available only to AI systems. The reviewed public sources do not establish a provider-wide rule of that kind. Nor can public documentation fully reveal private retrieval or ranking systems. Green’s concern about future validation or scrutiny should therefore be read as a question, not a confirmed outcome.

A practical audit: keep public and machine-readable claims aligned

A FirstGlance editorial workflow for checking material claims across public pages and machine-readable versions; not an official platform test.

The FirstGlance checklist below is an editorial audit framework, not an official Google or provider test. Use it for material assertions such as prices, warranties, eligibility rules, product capabilities, and service areas.

  • Record the public source: capture the canonical URL and the wording visible to readers.
  • Find every machine-readable version: check JSON-LD, feeds, APIs, Markdown responses, llms.txt, and other endpoints.
  • Track ownership and origin: note the content owner, last-verified date, and system or source used to generate each representation.
  • Compare meaning, not phrasing: equivalent wording is fine; incompatible prices, dates, conditions, or capabilities are not.
  • Flag risk clearly: record missing public corroboration, contradictions, stale information, orphaned endpoints, or structured data unsupported by visible page content.
  • Test negotiated formats: request both the ordinary HTML response and the machine-format response. Compare important claims, context, links, metadata, structured data, and freshness.
  • Assign remediation: give every mismatch an owner and a correction date, then rerun the check after material content changes.

Where feasible, generate alternate formats from a maintained source of truth. If an endpoint contains extra detail, publish the customer-relevant substance on an accessible public page as well. This makes the information easier to verify and easier to maintain; it is a governance recommendation, not a promise of better rankings.

The middle ground: make useful information accessible and accountable

Machine-readable content can be a useful presentation of public information. The risk rises when important claims are separated from public verification, left without an owner, or contradicted across audiences. Those conditions call for review, but they should not be confused with proof of cloaking or an undisclosed AI-provider penalty.

The proportionate response is to keep material claims consistent, publicly verifiable, and maintainable. That approach addresses the real audit problem without treating every AI-friendly format as suspicious.

Sources and references

  1. www.searchenginejournal.com/why-we-should-fear-the-shadow-web-were-building-for-ai/593930
  2. chrisgreenseo.substack.com/p/shadow-web-for-ai
  3. developers.google.com/search/docs/fundamentals/ai-optimization-guide
  4. developers.google.com/search/docs/appearance/ai-features
  5. developers.google.com/search/docs/appearance/structured-data/sd-policies
  6. developers.google.com/search/docs/essentials/spam-policies
  7. developers.google.com/search/docs/appearance/ranking-systems-guide
  8. developers.google.com/search/blog/2024/03/core-update-spam-policies
  9. developers.cloudflare.com/fundamentals/reference/markdown-for-agents
  10. developers.cloudflare.com/changelog/post/2026-02-12-markdown-for-agents
  11. developers.cloudflare.com/changelog/post/2026-10-09-markdown-for-agents-in-process-conversion
  12. openai.com/transparency-and-content-moderation
  13. developers.openai.com/api/docs/bots

About the author

Edvardas Čeilitka

SEO audit and consulting specialist

Independent SEO audit and consulting specialist behind First Glance SEO Audit research, practical guides and implementation recommendations.

Professional profile