85 launches, and 13 of them ship a robots.txt nobody wrote

A thread came past this week that is genuinely good: one founder, a billion impressions and a million clicks out of SEO, and the whole playbook laid out folder by folder. Domains, tech SEO, indexing, backlinks, social. No course at the end. I would follow most of it.

Its last folder is called AI search, it is seven lines long, and it opens with the line that started this post: same game as SEO. Then it says to ask ChatGPT for the best tool in your category and buy your way onto whichever pages it cites.

That tactic is real. The premise is not. Classic SEO is a fight over position: everyone is in the index, and you argue about the order. The answer engines run on permission, granted or refused per user agent, in a file most founders have never opened and some are no longer writing themselves. So instead of arguing about it, I counted.

The sample, and how it was read

Hacker News publishes its Show HN list as JSON, so the sample defines itself: every Show HN submission live on that list on 2 September 2026. That is 126 submissions, 123 of them carrying a URL, posted between 28 August and 2 September, and 85 distinct domains once duplicates collapse. Indie builders, launching this week, which is exactly the person the thread is written for.

Then four fetches per domain: the robots file, the llms file, the homepage, and a parse of the robots file that follows the actual grouping rule rather than searching for a word. Nothing here needs a tool or a login.

curl -s https://example.com/robots.txt
curl -s https://example.com/llms.txt | head -5
curl -s https://example.com/ | grep -o 'application/ld+json'
curl -s https://example.com/ | grep -o '<h1'

Thirteen files nobody wrote

Fifty-six of the 85 serve a real robots.txt. Twenty-nine serve nothing. Fifteen of the 56 tell GPTBot to go away, and this is where it gets interesting: thirteen of those fifteen files are the same file. Same preamble, same eight blocked agents, same order, byte for byte, on thirteen unrelated products.

It is Cloudflare’s managed robots.txt. Turn the setting on and Cloudflare generates the file for you, and if your origin already serves one, their documentation says it will prepend our managed robots.txt before your existing robots.txt, combining both into a single response. Prepend, not replace. Your file is still down there, underneath, saying whatever it said before.

Which is how nine of those thirteen ended up publishing two policies at once. Here is the bottom of one of them, unedited, after Cloudflare’s block of eight Disallow: / groups:

# END Cloudflare Managed Content

# BentoPDF Robots.txt
# https://www.bentopdf.com

# Allow all crawlers
User-agent: *
Allow: /

Allow all crawlers, in a file whose first half blocks eight of them. That is not a criticism of the founder, who wrote a correct file and then never saw what was served. It is the whole point: on nine of these thirteen domains the published crawling policy is not the one anybody typed. Two more had no origin file at all, so Cloudflare made one from scratch. One serves an HTML page at /robots.txt, which is its own separate bug. The last is jsr.io, whose own contribution is a single sitemap line.

What the block actually costs, agent by agent

Here is the part that would not fit in seven lines. Being blocked is not one thing, because these crawlers are not one thing. Each vendor now runs a separate agent per job, and the vendors document it themselves.

Seven of the eight are training or dataset crawlers. Exactly one, Google-Extended, reaches into a live answer: Google’s own reference says it governs training and for grounding, providing content from the Google Search index to the model at prompt time, in Gemini Apps and Vertex. It also says, in the same paragraph, that Google-Extended does not affect inclusion in Google Search and is not a ranking signal.

So the managed block is a coherent position, not a mistake: keep me out of the training runs, leave me in the answers. Cloudflare says so literally, in a line at the top of every one of those thirteen files: Content-Signal: search=yes, ai-train=no, use=reference. If that is your position, the file is right and you should keep it.

The problem is that nobody in this sample chose it in a way I can see from outside, and nine of them wrote the opposite sentence in the same document.

The agents that actually feed the answers were never in the file

Now the number that made this article worth writing. Across all 56 robots files, the count of sites blocking OAI-SearchBot is zero. Claude-SearchBot, zero. PerplexityBot, zero. Not one site in the sample has closed the door the answers actually come through, including the thirteen that never meant to close anything.

OpenAI’s crawler page puts the consequence in one sentence: sites that are opted out of OAI-SearchBot will not be shown in ChatGPT’s search answers, though can still appear as navigational links. Anthropic’s support page, updated 7 April 2026, says the same about Claude-SearchBot and adds Claude-User for the fetch a person triggers by asking. Perplexity publishes the same split and states outright that PerplexityBot is not used to crawl content for AI foundation models.

Which turns the thread’s seven lines into something actionable. If you want out of the training corpora, block GPTBot and ClaudeBot and you are done, at no cost to your citations. If you want into the answers, the agents to name are the ones nobody in this sample named.

The one file in eighty-five I would have shipped as is

Eleven of the 56 wrote their own AI rules, and all eleven named at least one search-side agent, which tells you these were read decisions rather than defaults. One went further than I have. generativeide.com splits its file into AI Search & Retrieval (Allow) and AI Training (Disallow), explains the intent in a comment, and then documents the exact failure this article is about, four lines from the top:

# NOTE: If Cloudflare "AI Scraper Protection" is active in your dashboard, it
# prepends a managed block that overrides the per-bot Allow rules below.
# Fix: Cloudflare Dashboard -> Security -> Bots -> AI Scrapers & Crawlers -> Disable.

They wrote that comment because it happened to them, and the file served today has nothing above it, so they went and turned the setting off. Their file allows nine names on the answering side, Googlebot and Bingbot included, blocks the six that build corpora, and says why in the file itself.

Seven domains out of the 85 name all three roles for both OpenAI and Anthropic, so seven people have read the same documentation. Five of those seven then allow everything, which is a decision and a fine one. Two run the split: training blocked, answering allowed, nothing accidental in either direction. Two out of eighty-five.

llms.txt stopped being a rumour while nobody was looking

Twenty-four of the 85 serve a real llms.txt: a plain markdown file at the root saying what the product is and listing the pages that matter. I opened all 24 and every one is genuine, from 1,150 bytes to 112,144. More than a quarter of a random week of indie launches, for a convention that did not exist two years ago.

Whether the engines read it is still unproven and I am not going to pretend otherwise. What it costs is ten minutes, and what it produces is the one clean, structured description of your product that a retrieval system does not have to reverse engineer out of your nav bar. Mine is 53 lines and it is the highest ratio of possible upside to effort on this whole list.

Three of the 24 publish an llms.txt inviting the models in and block ClaudeBot in the same breath. Two of those three are deliberate training-only blocks and read fine. The third is the CDN again.

The on-page half the thread also skips

While I had the homepages, I checked the things the thread lists and the things it does not. Of the 78 domains whose homepage answered, 41 ship JSON-LD and 37 do not, 24 have no canonical, 14 serve no h1 in the HTML before JavaScript runs, and 7 have no meta description. The thread covers the canonical, the h1 and the description. It never once says the words structured data, which is the gap I would close first, because it is the only place you get to state plainly what a page is in a format built to be parsed.

1. 01JSON-LD on every indexable page: Article with author and a real dateModified, Person for the author with sameAs pointing at every profile you own, Organization on the home page. The sameAs links are how a model confirms that the name on your site and the name on your GitHub are one entity rather than three weak ones.
2. 02An author page and an About page whose first sentence is a flat declarative description of you. A model will lift that sentence verbatim when someone asks who you are, and if it does not exist it writes its own.
3. 03Clusters instead of a weekly orphan: one pillar page per topic, five to ten satellites, each linking up and sideways. A model retrieves chunks, not sites, so every section has to survive being read alone, out of context.
4. 04A free tool on your domain. One calculator or checker collects more natural links in a year than a month of outreach emails, and nobody has to be paid.
5. 05Image hygiene past the alt text: descriptive filenames, WebP, and hero images at 1200px or wider with max-image-preview:large, or Google Discover will not show the card at all.
6. 06Prune once a year. Any page with no impressions and no inbound links after twelve months is diluting the rest of the site. Noindex it or fold it into the nearest survivor.

What this site does, so you can check it against me

Same four fetches, run against alexsssaint.com. The robots file names 18 agents explicitly and allows every one of them, because a portfolio and a free tool are worth more quoted than withheld, and each named group repeats the disallow list so naming a bot never opens the cockpit. There is an llms.txt. Every post carries BlogPosting JSON-LD with the author, the date and the cover. There is an IndexNow key at the root and a script that pings it after a deploy that changed something, which is the only way I know to get a new page in front of Bing the same day rather than the same month.

That is not a brag, it is the control group. If you run the four commands on my domain and on yours and the outputs differ, the article did its job.

Sixty seconds on your own domain

Run the four fetches at the top of this post against your own site, and read the robots output with two questions in mind. First: is there anything above your own first line? If there is, your CDN is writing policy for you, and the fix is a toggle in its dashboard, not an edit to your file. Second: does any group name an agent whose job is search rather than training? If not, the answer engines are reading you on a default nobody chose.

The thread’s advice to ask ChatGPT for the best tool in your category and go find the pages it cites is still the right second move. This is the first one, it takes a minute, and 29 of 85 people shipping this week have not done it because there is nothing at that URL to read.