If your business runs Facebook or Instagram ads, walking away from Meta to avoid its new Commercial Terms is not a realistic option, and Meta offers no other way to decline them. What you can still set is which of Meta’s crawlers your robots.txt asks to stay out. Our recommendation for most UK businesses: block Meta-ExternalAgent, the crawler Meta names for AI model training, and leave its ads crawler (Meta-ExternalAds), its Meta AI search crawler (Meta-WebIndexer) and its link-preview crawler (FacebookExternalHit) alone, and put the change live now, with a dated note of what you chose. These rules express crawling preferences: they do not reject Meta’s terms or remove content already collected.
The revised Commercial Terms replace a version dated 6 March 2024 that had no crawling clause at all. They cover any business or commercial use of Meta’s products, which the terms define to include ads and marketing messages, product catalogues, managing a Page and using Ads Manager or Meta Business Suite. Carrying on with any of those once the terms take effect counts as agreement, and pausing your ads while you keep managing a Page does not count as declining. The new clause lets Meta, or anyone acting for it, crawl, copy and periodically re-crawl pages associated with a business’s products or services, and “create test credentials” to reach them. Nieman Lab first reported the clause on 15 October.
The first published version also let Meta use that content to train its products. The training wording has now gone from the UK English text. The right to crawl has stayed, and the terms say nothing about robots.txt. Meta says website owners can still use robots.txt to signal what its crawlers may read, in its crawler documentation and in statements to journalists, while the contract itself is silent. Set your file on purpose and keep the evidence.
Meta has put its new terms on hold
Meta told business users about the update in early October, in a Business Help Center post and a notice in Meta Business Suite. Its stated aim is to cut overlap between agreements: the Product Catalog Terms move into the Commercial Terms, and the Ad Creative Generative AI Terms move into the Self-serve Ad Terms, Unite.ai reported. Meta then moved the start date from 30 October to 20 November 2026, and the UK English previews of the Commercial Terms and the Self-serve Ad Terms gave the new date in a notice at the top, though their own effective-date lines still read 30 October 2026. Both carry the same crawling clause. On 28 October, according to the CMA, Meta announced a pause to the rollout pending a consultation with industry groups. No new start date has been set, and Meta said its existing terms remain in place until then, Nieman Lab reported.
The postponement came amid objections from publishers and talks between Meta and the UK regulator. A Competition and Markets Authority spokesperson said on 23 October, Nieman Lab reported, that “Meta has now paused implementation until 20 November”, and that the CMA would keep scrutinising the terms. The CMA has also sent Meta a formal letter, according to the same report. Meta told Nieman Lab the extension was to “allow more time for us to explain to our partners and stakeholders” what is changing, including that content owners can still use robots.txt to signal preferences. On 30 October the CMA opened an investigation under the Competition Act 1998 into whether Meta’s commercial terms breach competition law, and it says no decision has been made.
Advertisers have granted crawling rights before. Search Engine Journal pointed out that the current Self-serve Ad Terms, dated 22 August 2025, already let Meta crawl an advertiser’s domain once it places an ad order, with research and training among the listed uses. The revised clause reaches further: the permission now comes with any business use of Meta’s products, so a company that manages a Facebook Page and never buys an ad is covered. The clause also takes in pages made available through other users’ use of Meta’s products, and the European Publishers Council has raised the concern that this could cover articles people share.
The crawling clause covers any business using Meta’s tools, and reports disagreed on AI training
The clause Nieman Lab published on 15 October ended with a sentence letting Meta use crawled content to provide, develop, improve, research and train its products. Three legal experts told Nieman Lab the wording could support an argument that Meta may scrape news sites and train its AI on them. Matt Topic, a media and intellectual property lawyer at Loevy & Loevy, said publishers “should assume that it would be enforceable”. James Grimmelmann of Cornell Law School added that test credentials would technically let Meta sign up as an ordinary user to get past registration walls.
Meta spokesperson Andy Stone replied on 16 October that the update was about pulling accurate product information from third-party sites for ad and content recommendations, and had nothing to do with scraping news sites, PPC Land reported. That afternoon, US time, Meta published a revised clause. The training sentence was gone, and the crawling was now tied to developing, improving, researching and maintaining advertising and business products, plus the integrity and security of Meta’s products.
Reports then split. Unite.ai said on 17 October, citing archived captures, that the sentence had been removed. PPC Land checked the UK English preview on the evening of 18 October and found it still there, word for word. On 19 October Search Engine Journal found the same split in the Self-serve Ad Terms, with the UK English preview still carrying the sentence and the US English one without it. By 20 October Relevant Audience saw the revised wording on the UK English pages. The UK English previews of both documents now show the revised clause, with no use of the word “train” and no mention of robots.txt.
The reports describe different wording in different language editions at different times. None of them establishes why.
The edit has not settled the legal argument. Topic told Nieman Lab that “develop” is broad enough to include training and that “business product” does not seem to be a defined term. Grimmelmann said the new purposes language is “still broad enough to drive a truck through”, and that the edit leaves alone the wording that would let Meta crawl entire sites.
robots.txt is the control Meta points to, and it has gaps
Meta’s answer to the objections has been robots.txt. A spokesperson told Nieman Lab on 17 October that the first version had not made clear that robots.txt preferences are respected, and that news sites can still switch the crawler off that way. Meta’s crawler documentation lists five user agents and is specific about which ones can be refused.
If the table extends beyond the screen, scroll sideways to view all columns.
| Crawler (robots.txt token) | What Meta says it does | robots.txt, by Meta’s account | Our default for a UK advertiser |
|---|---|---|---|
Meta-ExternalAgent (meta-externalagent) |
What Meta says it doesCrawls for uses such as training foundation AI models, or improving products by indexing content directly | robots.txt, by Meta’s accountNo exception listed | Our default for a UK advertiserBlock |
Meta-ExternalAds (meta-externalads) |
What Meta says it doesCrawls for uses such as improving advertising and other business-related products and services | robots.txt, by Meta’s accountNo exception listed | Our default for a UK advertiserKeep access (leave it unnamed; check your wildcard rules) |
Meta-WebIndexer (meta-webindexer) |
What Meta says it doesImproves Meta AI search results; Meta says allowing it helps Meta AI cite and link to your content | robots.txt, by Meta’s accountNo exception listed | Our default for a UK advertiserKeep access (leave it unnamed; check your wildcard rules) |
Meta-ExternalFetcher (meta-externalfetcher) |
What Meta says it doesFetches individual links at a user’s request, including for agentic AI tasks | robots.txt, by Meta’s accountMay bypass robots.txt | Our default for a UK advertiserNo rule (Meta says it may ignore one) |
FacebookExternalHit (facebookexternalhit) |
What Meta says it doesBuilds the title, description and image preview for links shared on Facebook, Instagram and Messenger | robots.txt, by Meta’s accountMay bypass robots.txt during security or integrity checks | Our default for a UK advertiserKeep access (blocking it can stop previews being created or refreshed) |
The first gap is in the table. By Meta’s own account, two of the five may ignore robots.txt: Meta-ExternalFetcher, which fetches pages a user asks for, and FacebookExternalHit, during security or integrity checks. For most advertisers we recommend keeping FacebookExternalHit, because it gathers and caches the preview card shown when someone shares your link, in an ad or a post.
The second gap is that Meta-ExternalAgent is a mixed-use crawler. Meta says it crawls for training foundation models or for improving products by indexing content directly, under one robots.txt token. A robots.txt rule for it cannot let the indexing through and refuse the training.
The third gap is legal. A robots.txt rule is a request. Benjamin Sobel of the University of Wisconsin told Nieman Lab that compliance is voluntary and that, “unless it’s a legally enforceable preference, it can still be ignored all the same”, and the experts Nieman Lab consulted said Meta could argue that terms a business has accepted outweigh its robots.txt. Meta says the update gives it no new rights that supersede those preferences; the terms do not say so. The clause is not yet in force, and the question is open. A business whose website content is a product in its own right should take legal advice.
Google has made the same point about the standard itself. Its Gary Illyes said on 21 October 2026 that robots.txt, as RFC 9309 defines it, is an access preference protocol: it governs which URLs a crawler may request and says nothing about what happens to content once it has been crawled, as Search Engine Journal reported. An IETF draft he co-wrote would add a Content-Usage rule to robots.txt for stating how crawled content may be used. It is still a Working Group document, has not been sent for IESG review and expires on 20 February 2027 unless revised, and the separate vocabulary draft that defines a train-ai label says it does not reflect Working Group consensus.
The fourth is time. Meta says its crawlers may cache robots.txt for up to 24 hours, and a block only changes crawling from then on. It does nothing about pages already collected.
Few sites name the training crawler. In Zyte’s State of Web Access 2026 study of 11,100 top landing pages, 4.4% named meta-externalagent in a Disallow rule, and a further 27.2% blocked all bots with a wildcard rule, which catches it too. A default WordPress robots.txt names no Meta crawler at all.
What should UK marketing teams do next?
Setting your Meta crawler robots.txt rules is an afternoon’s work for whoever looks after the website. Do it now rather than wait for a new start date:
- Save the terms you are agreeing to. Open the UK English previews of the Commercial Terms and the Self-serve Ad Terms, save each as a dated PDF, and read the crawling paragraph in the section on licences and rights you give Meta. Save the Business Help Center notice too, and save fresh copies when Meta sets a new start date. Meta has changed the clause once already.
- Read your robots.txt. Load yourdomain.co.uk/robots.txt and look for
meta-externalagent. If no group names it, the training crawler follows yourUser-agent: *group, if you have one, and can crawl every path that group does not disallow; with neither, robots.txt does not restrict it at all. Check the file your site actually serves, on every subdomain you use, because CMS plugins and CDNs can add or rewrite rules. - Block the training crawler by name. Add this group, or edit the existing one if your file already names the crawler, and leave the rest of the file as it is. Keep it as its own group. If an existing group names
meta-externalagentalongside other crawlers, remove only its User-agent line there and leave the other crawlers’ rules alone. Then remove any Allow lines from other groups for this crawler, because groups that name the same crawler are combined:User-agent: meta-externalagent Disallow: / - Check the rules for the crawlers you want to keep. Under the robots.txt standard, RFC 9309, a crawler that finds a group with its own name obeys that group and ignores the wildcard, and several groups that name the same crawler are combined. A group that names Meta-ExternalAds or Meta-WebIndexer replaces your wildcard rules for that crawler, so a group holding only
Allow: /would lift restrictions such as a block on/wp-admin/or checkout pages. Left unnamed, they follow the rules you already have, so check that your wildcard group does not disallow your whole site or the landing pages you advertise. - Keep the ads crawler if you advertise. Meta-ExternalAds crawls to improve advertising and business products, the purpose the revised clause now names. Meta’s crawler documentation does not say what blocking it changes for your campaigns, so for an advertiser we treat keeping it as the safer default. A brand or publisher that keeps a Page but buys no ads can consider adding it to the block, after checking which Meta business tools it uses. If your results depend on Meta reading your landing pages and product data, the rest of that set-up is in our Meta Ads UK guide.
- Keep Meta-WebIndexer and FacebookExternalHit. Meta says allowing Meta-WebIndexer helps Meta AI cite and link to your content, and FacebookExternalHit builds your link previews. OpenAI splits its search and training bots the same way, as our GEO guide explains, and Google offers Google-Extended for Gemini training and grounding without affecting Search inclusion, covered in our AI Overviews opt-out post.
- Check what a new login can see. The clause lets Meta create test credentials, and on 15 October Meta declined to tell Nieman Lab whether they would get past paywalls or other paid content. If you run a members’ area or gated downloads, look at what a free registration unlocks. The robots.txt standard itself says the file is no substitute for security, so anything private needs a login and permissions behind it.
- Publish now and check your logs. Meta allows up to 24 hours for a robots.txt change to take effect. A few days after publishing, search your server or CDN logs for
meta-externalagent; a crawler that honours the rule should stop requesting the blocked pages, though logs cannot show what happens to content already collected. Run a few key URLs through Meta’s Sharing Debugger to confirm link previews still build, and preview an ad to check its landing page loads. Meta sets no robots.txt deadline and you can change the file at any time; doing it first means your choice is in place, and on record, when the terms start. Changing robots.txt does not decline the terms. - Write the decision down. Record the date, the robots.txt text, the terms PDF you saved and who signed it off. If the clause is ever argued over, you will want to show when you published the rule and what it said, although the record does not prove when Meta read it and cannot limit every later use of your content on its own. Agencies should agree the policy with each client in writing, because it touches both the client’s content and the Meta products its ads rely on.
What to watch next
Watch for the new start date Meta sets after its consultation, and for the CMA’s investigation, whose case page plans initial information gathering until January 2027. When the revised terms take effect, compare the live text with the copy you saved. Then watch whether Meta writes its robots.txt assurance into the terms. Movement for an Open Web has filed a complaint with the CMA and the European Commission asking for interim measures. The European Publishers Council has written to the European Commission again, asking it to seek an undertaking from Meta to postpone the contested provisions if interim measures cannot come before 20 November, and the UK’s News Media Association, which asked ministers and regulators to intervene, called the delay “a pause, not a solution”. The European Commission told Nieman Lab that it “welcomes Meta’s announcement” of the postponement and is committed to ensuring that Meta complies with EU law, including competition rules and the Digital Markets Act. Separately, PPC Land reported on 15 October that an undated Meta notice told advertisers it would switch on product browsing in existing ads from 28 October unless they opt out; the notice does not list the formats, placements or objectives affected, and the date is separate from the terms. Closer to home, check Meta’s crawler documentation every few months for new user agents or changed descriptions, and check how your existing rules apply to any new name. Our guide to robots.txt and AI crawlers covers the tokens Google, OpenAI, Anthropic and Perplexity publish.
The real lesson from Meta’s terms update
Meta changed a contract that every business on its platforms accepts through continued use, changed it again within two days of the first report, showed different wording in different language editions, then moved the start date and put the whole update on hold. None of that needed your signature. The robots.txt file on your own server is one of the few parts of the arrangement you control, and a policy on AI crawlers, covering every vendor, belongs in routine website upkeep. Meta Connect 2026 showed the company leaning hard on product data and AI agents, so the question of what its crawlers read will come back.
To have your crawler rules reviewed, or to see what Meta and the other AI platforms can read on your site, talk to AIWIZ.