Google asks sites to let its AI agent past firewalls
By Jonathan DavidEditorial standards
Google published a technical guide on 17 November asking website owners to explicitly allow Google-Agent, the name its agents use when they browse the web and take actions for a user, in their CDNs, web application firewalls and bot detection services. Because the visit is requested by a person, Google says fetchers of this kind generally ignore robots.txt, so a site's firewall, not its robots.txt file, decides whether the agent gets in. When we requested the homepages of 50 big UK brands from a cloud server on 18 November, 16 refused every request, including the ones sent under a normal Chrome browser's name.
What did Google publish?
The new page, Allowing Google-Agent traffic, describes Google-Agent as the name used by agents hosted on Google infrastructure that navigate the web and perform actions at a user's request. It asks sites to allow that traffic explicitly so that legitimate requests are not blocked, while keeping their other security controls. Google's crawling changelog dates the addition to 17 November, and Search Engine Roundtable reported it the same morning.
Google gives three ways to recognise the agent. Its user agent string contains "Google-Agent". Its requests come from the IP ranges in a file called user-triggered-agents.json, which listed 20 address blocks, 12 IPv4 and 8 IPv6, when we read it on 18 November. And Google is experimenting with signing some requests under the Web Bot Auth protocol, with the identity agent.bot.goog, so a site can check the signature against published keys.
Google's current pages do not say which of its products send these requests, or in which countries. When the name first appeared in March, Google gave Project Mariner as an example, as Search Engine Roundtable reported.
20
IP address blocks in Google's user-triggered-agents.json file, 12 IPv4 and 8 IPv6
File read 18 November 2026; it was generated on 17 November and Google updates it
Why doesn't robots.txt control it?
Google lists Google-Agent among its user-triggered fetchers, and says that because a user requested the fetch, these fetchers generally ignore robots.txt rules. A robots.txt file that welcomes Google's crawlers therefore says nothing about whether the agent can load your pages. That is decided by whatever sits in front of the site: the CDN, the firewall and any bot management service.
Google's troubleshooting list is specific about where blocks happen. An allow rule has to be evaluated before broad blocks on cloud and data centre networks, managed bot rules and category-level blocks on AI agents. It has to skip CAPTCHAs and JavaScript challenges, including in-page CAPTCHAs on search or form submission. It has to cover more than the homepage: search, filters, detail pages, add to cart, booking and the API calls behind them. Per-IP rate limits need to allow for many users sharing the same Google addresses, and proxies and load balancers have to pass on the visitor's IP, user agent and signature headers.
How many big UK brand sites refuse automated visits?
About a third of the ones we tried. We requested the homepages of the 50 brands in our Google UK brand search reading three times each between 06:08 and 06:12 UTC on 18 November, once under a normal Chrome browser's name and once under a name that said it was an automated research request. 16 of the 50 never returned their homepage to either. 33 returned it every time to the browser name, and BT returned it in two of the three runs.
Nine of the 16 answered with HTTP 403 every time: Tesco, Sainsbury's, Asda, Revolut, ScottishPower, Primark, easyJet, Compare the Market and Argos. Lloyds Bank, Halifax and giffgaff sent an error or block page in place of the homepage. Waitrose, ASOS, Jet2 and British Airways dropped the connection, timed out or showed a holding page. Four of the five supermarkets and three of the five travel brands were among them.
The name made little difference. Only Next and Aviva served the browser name every time and refused the self-declared automated one every time, and BT varied. The 16 sites that refused everything did so whatever the request called itself, which points to rules about where traffic comes from, the kind Google lists first in its troubleshooting.
This does not show what Google-Agent meets on these sites. Its requests come from Google's own published addresses, and bot services can verify and allow them, which our cloud server cannot imitate. It does show how many big UK sites turn away automated traffic from cloud networks by default, which is the setting Google is asking site owners to check.
16 of 50
big UK brand homepages refused every request from a cloud server, under a browser's name and an automated one
Three runs each, 18 November 2026, 06:08 to 06:12 UTC
What should a UK brand do?
Decide on purpose. A site can choose to block Google-Agent, but many blocks come from default rules set up long ago to stop scrapers, and nobody has looked at them since. Ask your CDN or bot management provider whether it verifies Google-Agent automatically; Google says some do and others need it configured.
If you want to allow it, put the rule above any data centre, managed bot or AI agent category rules, exempt verified requests from CAPTCHAs, keep IP-based rules in step with Google's latest published ranges, and test the pages an agent would actually use, such as search, product pages, basket and booking, not only the homepage. Search your server logs for "Google-Agent" to see whether it already visits and what status codes it receives.
Robots.txt still matters for the crawlers that do read it. The free AI crawler checker reads your robots.txt and shows which AI crawlers it lets in; a firewall rule is a separate check you make with your provider. To see whether AI answers name your business at all, the free AI visibility check asks buyer questions about your business on Claude, ChatGPT and Google AI Overviews, with no login, and shows who each answer names and the pages it cites.
How we measured it
We requested https://www. followed by each of the 50 brand website addresses in our 6 October Google UK reading three times, at about 06:08, 06:11 and 06:12 UTC on 18 November 2026, from a cloud server, following redirects and waiting up to 25 seconds. Each run sent two requests per site: one with a current Chrome on Windows user agent and one with a user agent naming an automated research client. A site counted as serving its homepage when it returned HTTP 200 with a page title that was not an error, block, challenge or maintenance notice and at least 50 words of text.
A single cloud address is one vantage point. Sites treat traffic differently by network, country and time, so a home connection, or Google's own addresses, may get a different answer, and the same sites may answer differently tomorrow. We counted what the sites returned, not why. The address count comes from Google's user-triggered-agents.json file as read on 18 November, and the facts about Google's guidance come from the pages linked above.
Sources
- Google Search Central: Allowing Google-Agent traffic (opens in new tab) developers.google.com
- Google Search Central: Crawling documentation changelog (opens in new tab) developers.google.com
- Search Engine Roundtable: New Google Search Help Document On How To Allow Google-Agent Traffic (opens in new tab) seroundtable.com
- Search Engine Roundtable: Google-Agent user agent, March 2026 (opens in new tab) seroundtable.com
- Google Search Central: Google's user-triggered fetchers (opens in new tab) developers.google.com
Related
Keep exploring: Free AI visibility checkerWhat is GEO?The best GEO tools (2026)AI Visibility Index: 40 brands measuredGEO statistics 2026Google AI Overviews trackerChatGPT visibility trackerPerplexity visibility trackerGemini visibility tracker