Should publicly-funded media stop AI from reading and using its journalism? It sounds like a question about copyright and intellectual property, but it’s not. It’s a question about purpose.
The purpose of public media
The Public Media Alliance (which advocates for public media and draws its membership from the most influential public media outlets from around the world) defines the purpose of public media as:
… to provide a variety of quality content that is universally accessible to a diverse audience on a national level. This includes providing reliable information to the public so that they can participate in society in a meaningful way.
They go on to say:
Ultimately, public service media organisations are intended to inform, educate, and entertain. They should enable citizens to access and interact with free, independent, engaging and relevant content whether they are in rural or urban environments, irrespective of economic status or technology.
That last point “irrespective of technology” is key, because we’re in the middle of the biggest information revolution in human history since the invention of the printing press.
The information revolution
A generation ago, most people got their news from newspapers, radio, and TV. Today, they get it from a device and increasingly by asking questions answered by AI.
When someone asks AI (or search) a question, it answers from information it can access. Content an AI system can’t read is content a growing share of the public won’t see.
robots.txt
That brings me to robots.txt: a plain text file on a website with instructions telling bots what they can or can’t read.
It’s what determines what content search and AI bots will crawl and index, and as a result be visible to their users. And it’s publicly visible to anyone to look at. (You can find it by adding “/robots.txt” to any domain name.)
I recently noticed a public broadcaster blocked an AI agent I created for compiling summaries of global media coverage of issues I’m interested in.
So I checked what 7 public media had in their robots.txt: ABC, SBS, BBC, PBS, NPR, CBC, and for old times’ sake (as an ex-Kiwi) RNZ.
I was surprised by what I found.
Some read like a time capsule referring to long-dead content and platforms. Some cited commercial advertising arrangements as reasons for reluctantly allowing specific advertising bots access – all in plain sight.
Most blocked some or many of the search and AI agents commonly used today.
Most showed little strategic thought or intent.
Here in Australia, the ABC and SBS shared obscure entries from past copying and pasting, but had radically different block-lists with no obvious coherent rationale: one blocked 17 AI crawlers and the other just 2.
The exception was the BBC, which appeared the most thoughtful (even including plain-English guidance for AI agents at the top), but ironically the least aligned with the purpose of public media.
Its robots.txt prevents use of its content for training, for search, for creating news summaries, or for business use. That’s a defensible position for a commercial publisher protecting its content, but it’s devastating for public access to its publicly-funded journalism. (The BBC even threatened legal action against AI startup Perplexity for ignoring its robots.txt and using its content without permission.)
Even if the BBC didn’t want its content used for training, it could still allow its content to searched and cited, but it doesn’t.
Not surprisingly, research by the Institute for Public Policy Research found that ChatGPT sourced more information from GB News, Al Jazeera, and Marie Claire than from the BBC:
While ChatGPT appears to be respecting the BBC’s wishes, this comes at a cost to the public: the UK’s most popular and trusted news outlet is absent from the country’s most widely used AI tool.
The research also found the BBC absent in answers from Google’s Gemini, which generates the “AI overview” in Google Search results. Because of Google Search’s ubiquity, it’s probably the most widely used AI answer engine even if many of its users don’t even realise they’re using AI.
It’s a strategic decision, not a technical one
What access you give to search and AI bots is a strategic decision, not a technical one. It can’t be left to content management system’s defaults or whoever looks after the website.
For public media, it’s hard to defend blocking search and AI if they are to live up to their purpose of making their content accessible regardless of technology. In an age when people reach for search and AI for answers to questions, it’s tantamount to taking yourself off air.
