
What AI crawlers can actually read on a website — and what robots.txt does and doesn't control
AI crawlers read what your server publicly returns. robots.txt is a public request that cooperating crawlers follow — not access control, and not noindex. What each control actually governs, with sources.
ELIXIR Creative