Information correct as of 21 October 2026. Meta has already rewritten the crawling clause once since publishing it, so read the version your own account shows before 30 October.
If your business runs Facebook or Instagram ads, walking away from Meta to avoid its new Commercial Terms is not a realistic option, and Meta offers no other way to decline them. What you still decide is what Meta’s crawlers may read on your website. Our recommendation for most UK businesses: block Meta-ExternalAgent, the crawler Meta names for AI model training, leave its ads and Meta AI search crawlers alone, and have the change live by 28 October with a dated note of what you chose.
The revised terms take effect on 30 October 2026 and replace a version dated 6 March 2024 that had no crawling clause at all. They cover any business or commercial use of Meta’s products, which the terms define to include ads and marketing messages, product catalogues, managing a Page and using Ads Manager or Meta Business Suite. Carrying on with any of those after 30 October counts as agreement, and pausing your ads while you keep managing a Page does not count as declining. The new clause lets Meta, or anyone acting for it, crawl, copy and periodically re-crawl pages associated with a business’s products or services, and “create test credentials” to reach them. Nieman Lab first reported the clause on 15 October.
The first published version also let Meta use that content to train its products. The training wording has now gone from the UK English text. The right to crawl has stayed, and the terms say nothing about robots.txt. Meta says robots.txt still decides what its crawlers read, but that assurance lives in statements to journalists while the contract is silent. Set your file on purpose and keep the evidence.
Meta’s new Commercial Terms take effect on 30 October
Meta told business users about the update in early October, in a Business Help Center post and a notice in Meta Business Suite. Its stated aim is to cut overlap between agreements: the Product Catalog Terms move into the Commercial Terms, and the Ad Creative Generative AI Terms move into the Self-serve Ad Terms, Unite.ai reported. Both revised documents take effect on 30 October, and the UK English Self-serve Ad Terms preview carries the same crawling clause as the Commercial Terms preview.
Advertisers have granted crawling rights before. Search Engine Journal pointed out that the current Self-serve Ad Terms, dated 22 August 2025, already let Meta crawl an advertiser’s domain once it places an ad order, with research and training among the listed uses. The change on 30 October is reach. The permission now comes with any business use of Meta’s products, so a company that manages a Facebook Page and never buys an ad is covered. The clause also takes in pages made available through other users’ use of Meta’s products, and the European Publishers Council has raised the concern that this could cover articles people share.
The crawling clause covers any business using Meta’s tools, and reports disagreed on AI training
The clause Nieman Lab published on 15 October ended with a sentence letting Meta use crawled content to provide, develop, improve, research and train its products. Three legal experts told Nieman Lab the wording could support an argument that Meta may scrape news sites and train its AI on them. Matt Topic, a media and intellectual property lawyer at Loevy & Loevy, said publishers “should assume that it would be enforceable”. James Grimmelmann of Cornell Law School added that test credentials would technically let Meta sign up as an ordinary user to get past registration walls.
Meta spokesperson Andy Stone replied on 16 October that the update was about pulling accurate product information from third-party sites for ad and content recommendations, and had nothing to do with scraping news sites, PPC Land reported. That afternoon, US time, Meta published a revised clause. The training sentence was gone, and the crawling was now tied to developing, improving, researching and maintaining advertising and business products, plus the integrity and security of Meta’s products.
Reports then split. Unite.ai said on 17 October, citing archived captures, that the sentence had been removed. PPC Land checked the UK English preview on the evening of 18 October and found it still there, word for word. On 19 October Search Engine Journal found the same split in the Self-serve Ad Terms, with the UK English preview still carrying the sentence and the US English one without it. By 20 October Relevant Audience saw the revised wording on the UK English pages. When we opened the UK English previews of both documents on 21 October 2026, each showed the revised clause, and neither contained the word “train” or any mention of robots.txt.
Both sets of reports appear to have been accurate when they were made. Meta seems to have updated its language editions on different days, which left UK businesses for roughly three days with a preview that still held the training wording Meta told reporters it had removed.
The edit has not settled the legal argument. Topic told Nieman Lab that “develop” is broad enough to include training and that “business product” is not a defined term. Grimmelmann said the new purposes language is “still broad enough to drive a truck through”, and that the edit leaves alone the wording that would let Meta crawl entire sites.
robots.txt is the control Meta points to, and it has gaps
Meta’s answer to the objections has been robots.txt. A spokesperson told Nieman Lab on 17 October that the first version had not made clear that robots.txt preferences are respected, and that news sites can still switch the crawler off that way. Meta’s crawler documentation lists five user agents and is specific about which ones can be refused.
If the table extends beyond the screen, scroll sideways to view all columns.
| Crawler (robots.txt token) | What Meta says it does | robots.txt, by Meta’s account | Our default for a UK advertiser |
|---|---|---|---|
Meta-ExternalAgent (meta-externalagent) |
What Meta says it doesCrawls for uses such as training foundation AI models, or improving products by indexing content directly | robots.txt, by Meta’s accountNo exception listed | Our default for a UK advertiserBlock |
Meta-ExternalAds (meta-externalads) |
What Meta says it doesCrawls for uses such as improving advertising and other business-related products and services | robots.txt, by Meta’s accountNo exception listed | Our default for a UK advertiserAllow (leave it unnamed) |
Meta-WebIndexer (meta-webindexer) |
What Meta says it doesImproves Meta AI search results; Meta says allowing it helps Meta AI cite and link to your content | robots.txt, by Meta’s accountNo exception listed | Our default for a UK advertiserAllow (leave it unnamed) |
Meta-ExternalFetcher (meta-externalfetcher) |
What Meta says it doesFetches individual links at a user’s request, including for agentic AI tasks | robots.txt, by Meta’s accountMay bypass robots.txt | Our default for a UK advertiserNo rule (Meta says it may ignore one) |
FacebookExternalHit (facebookexternalhit) |
What Meta says it doesBuilds the title, description and image preview for links shared on Facebook, Instagram and Messenger | robots.txt, by Meta’s accountMay bypass robots.txt during security or integrity checks | Our default for a UK advertiserAllow (a block breaks link previews) |
The first gap is in the table. By Meta’s own account, two of the five may ignore robots.txt: Meta-ExternalFetcher, which fetches pages a user asks for, and FacebookExternalHit, during security or integrity checks. Blocking FacebookExternalHit would cost you more than it saves, because it builds the preview card whenever someone shares your link, in an ad or a post.
The second gap is that Meta-ExternalAgent is a mixed-use crawler. Meta says it crawls for training foundation models or for improving products by indexing content directly, under one robots.txt token. You cannot let it index and refuse the training.
The third gap is legal. A robots.txt rule is a request. Benjamin Sobel of the University of Wisconsin told Nieman Lab that following one is voluntary and gives a site no enforceable right, and the experts Nieman Lab consulted said Meta could argue that terms a business has accepted outweigh its robots.txt. Meta says the update gives it no new rights that supersede those preferences; the terms do not say so. The clause is not yet in force, no court has tested it, and the question is open. This post is not legal advice, and a business whose website content is a product in its own right should take some.
The fourth is time. Meta says its crawlers may cache robots.txt for up to 24 hours, and a block only changes crawling from then on. It does nothing about pages already collected.
Few sites have made a choice either way. In Zyte’s State of Web Access 2026 study of 11,100 top landing pages, 4.4% named meta-externalagent in a Disallow rule. A default WordPress robots.txt names no Meta crawler at all.
What should UK marketing teams do next?
Setting your Meta crawler robots.txt rules is an afternoon’s work for whoever looks after the website. Get it done before 28 October:
- Save the terms you are agreeing to. Open the UK English previews of the Commercial Terms and the Self-serve Ad Terms, save each as a dated PDF, and read the crawling paragraph in the section on licences and rights you give Meta. Meta has changed it once already.
- Read your robots.txt. Load yourdomain.co.uk/robots.txt and look for
meta-externalagent. If it is not there, the training crawler follows yourUser-agent: *group, and unless that group blocks the whole site, Meta-ExternalAgent can crawl it. Check the file your site actually serves, on every subdomain you use, because CMS plugins and CDNs can add or rewrite rules. - Block the training crawler by name. Add this group and leave the rest of the file as it is:
User-agent: meta-externalagent Disallow: / - Do not name the crawlers you want to keep. Under the robots.txt standard, RFC 9309, a crawler that finds a group with its own name obeys that group and ignores the wildcard. An Allow group for Meta-ExternalAds or Meta-WebIndexer would switch off your existing wildcard rules for them, such as a block on
/wp-admin/or checkout pages. Left unnamed, they follow the rules you already have. - Keep the ads crawler if you advertise. Meta-ExternalAds crawls to improve advertising and business products, the purpose the revised clause now names. Meta does not document what blocking it changes for your campaigns, so an advertiser risks more by blocking it than it protects. A brand or publisher that keeps a Page but buys no ads can reasonably add it to the block. If your results depend on Meta reading your landing pages and product data, the rest of that set-up is in our Meta Ads UK guide.
- Keep Meta-WebIndexer and FacebookExternalHit. Meta says allowing Meta-WebIndexer helps Meta AI cite and link to your content, and FacebookExternalHit builds your link previews. OpenAI splits its search and training bots the same way, as our GEO guide explains, and Google keeps training separate through Google-Extended, covered in our AI Overviews opt-out post.
- Check what a new login can see. The clause lets Meta create test credentials, and Meta has not said which crawler would use them or whether they would reach paid content. If you run a members’ area or gated downloads, look at what a free registration unlocks. The robots.txt standard itself says the file is no substitute for security, so anything private needs a login and permissions behind it.
- Publish by 28 October and check your logs. Meta allows up to 24 hours for a robots.txt change to take effect. A few days after publishing, search your server or CDN logs for
meta-externalagent; requests to blocked paths should stop. Run a few key URLs through Meta’s Sharing Debugger to confirm link previews still build, and preview an ad to check its landing page loads. The 28 October date is our safety margin. Meta sets no robots.txt deadline, you can change the file after 30 October too, and an early change does not stop you accepting the terms by carrying on. - Write the decision down. Record the date, the robots.txt text, the terms PDF you saved and who signed it off. If the clause is ever argued over, you will want to show that you refused training on purpose, and when. Agencies should agree the policy with each client in writing, because it affects both the client’s content and its ad performance.
What to watch next
On 30 October the preview becomes the live text, so compare it with the copy you saved. Then watch whether Meta writes its robots.txt assurance into the terms; until it does, the promise sits in emails to reporters. The European Publishers Council has written to the European Commission asking for urgent scrutiny, and the UK’s News Media Association has asked ministers and regulators to intervene. Neither has a date attached. Closer to home, check Meta’s crawler documentation every few months for new user agents or changed descriptions, because a robots.txt file only covers the names it contains.
The real lesson from Meta’s terms update
Meta changed a contract that every business on its platforms accepts by default, changed it again within two days of the first report, and put the change into its language editions at different speeds. None of that needed your signature. The robots.txt file on your own server is one of the few parts of the arrangement you control, and a policy on AI crawlers, covering every vendor, belongs in routine website upkeep. Meta Connect 2026 showed the company leaning hard on product data and AI agents, so the question of what its crawlers read will come back.
To have your crawler rules reviewed, or to see what Meta and the other AI platforms can read on your site, talk to AIWIZ.