MARKETING · · 7 min read
Meta’s 20 November ad terms are paused. Can it still crawl your website?
Meta’s new advertiser terms, which would have let it crawl business websites, won’t start on 20 November after all. Here’s what was paused, what still applies, and the simple checks worth doing anyway.
Ami · Paid Media Specialist
On this page
QUICK ANSWER
Nothing changes on 20 November. Meta has paused its updated Commercial Terms and ad terms, including the new clause that would have let it crawl websites linked to business accounts, and the existing terms stay in place until further notice. There’s no new date.
That doesn’t mean Meta can’t read your website today. Its crawlers already visit most sites, and the current ad terms already cover content you supply in ads. The useful move is to decide which Meta crawlers you’re happy with, check your robots.txt, and review the AI features in your ad account.
What Meta was planning to change
In October, Meta told advertisers it was updating four sets of terms: the Meta Commercial Terms, the Self-Serve Ad Terms, the Product Catalog Terms and the Ad Creative Generative AI Terms. According to CIO, they were first due on 30 October, then pushed back to 20 November.
Two changes drew most of the attention:
- Website crawling. The new Commercial Terms included a clause allowing Meta to crawl websites associated with a business’s products or services, not just the pages you link in an ad.
- Responsibility for AI. A new AI features section would have made advertisers solely responsible for what Meta’s AI tools create or change in their ads.
Publishers and regulators pushed back hard. As Search Engine Journal reported, courts in Norway and Brazil issued orders and the UK’s competition regulator opened an investigation.
What Meta has actually said
Around 5 November, Meta posted a notice saying the updates “are now paused and will not take effect on November 20, 2026”. A Meta spokesperson said the company is “pausing the changes as we continue to engage with stakeholders and take our time to get this right.”
A few things are worth reading carefully:
- It’s a pause, not a withdrawal. Meta hasn’t said the changes are gone, only that they won’t start on 20 November.
- The current terms continue. The versions already in force stay in effect until Meta says otherwise.
- No country is singled out. The notice, as reported, doesn’t limit the pause to any region, so Australian advertisers are in the same position as everyone else.
Key point: if you got an email or Ads Manager banner about new terms starting 20 November, you don’t need to accept or do anything by that date.
Book a 20 minute callCan Meta crawl your website right now?
Yes, in a few ways, and most of them are normal.
Search Engine Journal’s review of the current terms found that the Self-Serve Ad Terms already let Meta access, analyse and crawl the content you provide when you place an ad, and the Ad Creative Generative AI Terms let it crawl URLs you give it. So the landing pages in your ads are already covered. The paused clause would have reached further, to your business’s wider web presence.
Separately, Meta runs several crawlers across the web. Its web crawler documentation lists them:
| Crawler | What Meta says it’s for |
|---|---|
| facebookexternalhit | Building link previews (title, description, image) when someone shares your page on Facebook, Instagram or Messenger |
| meta-externalagent | Uses such as training AI models or improving products by indexing content |
| meta-externalads | Improving advertising and other business products |
| meta-webindexer | Improving Meta AI search results, so it can cite and link to your content |
| meta-externalfetcher | Fetching a link when a person asks for it |
Meta says you can control these with a Disallow rule in robots.txt, and that changes can take up to 24 hours to be picked up. It also says two of them, facebookexternalhit and meta-externalfetcher, may bypass robots.txt in some situations, such as security checks or fetching a link a person asked for.
Should you block any of them?
For most small businesses, blocking everything from Meta is a mistake. If you block facebookexternalhit, links to your site shared on Facebook and Instagram can show up without a proper title or image, which makes them far less likely to be clicked. That hurts your organic social and your ads.
A more measured approach:
- Keep facebookexternalhit allowed, so shared links look right.
- Keep meta-webindexer allowed if you’d like Meta AI to be able to cite and link to you when people ask it questions.
- Decide deliberately about meta-externalagent. This is the one that covers AI training, and it’s the same call we walk through in should you block AI from reading your website. If your content is the thing you sell, blocking it is reasonable. If your site exists to win enquiries, it usually isn’t worth worrying about.
- Be careful with meta-externalads. Meta doesn’t say blocking it affects your ads, but it doesn’t say it doesn’t, either. If you run Meta ads, we’d leave it alone unless you have a specific reason.
To opt out of training, the rule in your robots.txt looks like this:
- Line 1: User-agent: meta-externalagent
- Line 2: Disallow: /
Type your domain followed by /robots.txt into a browser to see what your file says now. If your site sits behind Cloudflare or uses a platform setting to block AI crawlers, those can block Meta’s bots regardless of what your robots.txt says, so check those too.
The AI ad features are still worth reviewing
The paused liability clause was about AI features, and those features haven’t gone anywhere. Meta keeps pushing Advantage+ automation, including AI-generated text variations and image enhancements. A Meta spokesperson has said advertisers can opt out of individual Advantage+ features, so you don’t have to take all or nothing.
Whatever the terms say, you are the one whose name is on the ad. Under Australian Consumer Law, your advertising mustn’t mislead, whether a person or a tool wrote it. That’s general information, not legal advice, and the ACCC’s advertising guidance is the place to start.
Practical habits that help:
- Check which Advantage+ creative enhancements are switched on in each campaign, not just the account default.
- Turn off anything that changes wording, prices or product images in ways you haven’t approved.
- Preview ads in every placement before they go live, and spot check them while they run.
- Keep a simple record of what you approved, especially for offers and claims.
We cover how Meta ads work, what to spend and why creative matters in our guide to Facebook and Instagram ads for Melbourne small businesses.
What to watch next
Meta says it will share an update when it has more to say. The pressure from regulators in Europe and the UK hasn’t gone away, so the next version may look different. When new terms do arrive, read the crawling and AI sections before you click accept, and give yourself time to adjust your robots.txt and campaign settings first.
It’s also part of a wider pattern. Google is doing something similar with ads inside its AI answers, which we covered in Google ads are now inside AI Mode in Australia. The platforms want more of your content and more control of your ads, and the businesses that do well are the ones who make those choices on purpose.
A quick checklist
- Ignore any 20 November deadline for now. The new terms are paused.
- Look at your robots.txt and any Cloudflare or platform AI crawler settings.
- Keep facebookexternalhit allowed so shared links display properly.
- Decide on meta-externalagent based on whether your content is your product.
- Review Advantage+ creative settings campaign by campaign.
- Watch for Meta’s next update and read it before accepting.
Questions
Did Meta’s new terms start on 20 November 2026?
No. Meta paused its updated Commercial Terms, Self-Serve Ad Terms, Product Catalog Terms and Ad Creative Generative AI Terms, and the existing versions stay in effect until further notice.
Does Meta crawl my website already?
Most likely. Meta’s crawlers build link previews, index content for Meta AI and may collect content for training, and the current ad terms already cover content and URLs you supply in ads.
How do I stop Meta using my website to train AI?
Add a Disallow rule for meta-externalagent to your robots.txt. Leave facebookexternalhit allowed, or links to your site shared on Facebook and Instagram may lose their previews.
Will blocking Meta’s crawlers affect my ads?
Meta doesn’t say. We’d keep meta-externalads and facebookexternalhit allowed if you advertise on Facebook or Instagram, and only block the training crawler if that’s what you want.
Not sure what your site is telling Meta, Google and AI tools, or whether your ad settings are working for you? Our [digital marketing team](ServiceMarketing.dc.html) can review your Meta account and website together, and our free [website check](WebsiteCheck.dc.html) covers the SEO basics in a few minutes.
Book a 20 minute call
WRITTEN BY
Ami · Paid Media Specialist
Runs Google Ads and Meta campaigns that turn budget into enquiries, and reports on what actually works.
