In this article
An ecommerce team finds their Shopify store has 14,000 faceted filter URLs being crawled but never ranked, burning Google’s crawl budget on pages that will never drive revenue. That crawl budget problem lives in one file: robots.txt. In 2026, that same file also controls whether GPTBot, ClaudeBot, and PerplexityBot can crawl your product catalogue and surface your store in AI-generated answers. This guide covers how to edit Shopify’s robots.txt with the current UI, how to protect crawl budget from faceted navigation, and how to make an informed decision on AI crawler access.
Contents
What Shopify’s robots.txt does (and what it can’t)
How to edit your Shopify robots.txt in 2026
Managing crawl budget for Shopify faceted navigation
The AI crawler decision: allow or block GPTBot, ClaudeBot, PerplexityBot?
What you can’t control in Shopify robots.txt
Testing your robots.txt changes
Picture a marketing manager logging into Google Search Console for a routine check. The Page Indexing report shows 14,000 URLs listed as “Discovered: currently not indexed.” Clicking through, nearly all of them are collection filter pages: /collections/womens-shoes?sort_by=price-ascending, /collections/jeans?color=blue&size=32, thousands of combinations generated automatically by Shopify’s filtering system. Googlebot has been visiting these pages regularly, spending crawl budget on pages that will never rank, while the product and category pages that actually matter sit lower in the crawl queue.
That problem lives in robots.txt. And in 2026, the same file also controls something store owners rarely think about: whether GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot can crawl your products and cite your store when a shopper asks an AI for a recommendation. Shopify’s robots.txt was read-only until June 2021; today it’s a Liquid template you can edit directly. Here’s how to use it deliberately.
What Shopify’s robots.txt does (and what it can’t)
Shopify’s robots.txt file is a plain-text file served at yourdomain.com/robots.txt that tells web crawlers which paths on your store they are and aren’t permitted to access. Every major crawler reads it before crawling: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, and dozens of others. Shopify’s default robots.txt already blocks a sensible set of paths including /admin, /checkout, /cart, /orders, and /account. You don’t need to add those; they’re baked in.
What the default file doesn’t do is protect you from crawl budget waste on faceted navigation, or make any deliberate decision about AI crawlers. Both of those require editing the template yourself.
IMPORTANT
Shopify’s robots.txt is a Liquid template: changes go live on your production store the moment you save. There is no staging preview. Make one change at a time, then verify the live file at yourdomain.com/robots.txt before making another. A single typo that accidentally blocks Googlebot will suppress crawling of your entire store within days.
One key constraint worth knowing upfront: Shopify’s Liquid template system supports Disallow rules cleanly, but adding true Allow: overrides that override a parent Disallow is awkward and sometimes unreliable. If you need complex allow/disallow logic, canonical tags and noindex meta tags on individual page templates are more reliable tools. More on that in the limitations section below.
How to edit your Shopify robots.txt in 2026
The “Actions” button referenced in older guides no longer exists: Shopify replaced it with a three-dot (“…”) context menu in the Online Store 2.0 interface. If you’ve tried following instructions from 2021 or 2022 and couldn’t find the right button, that’s why. Here are the current steps.
Log in to your Shopify admin. In the left sidebar, select Online Store, then Themes. You’ll see your currently active theme at the top of the page.
Next to your active theme, you’ll see a three-dot button (not “Actions” as older guides show). Click it, then select Edit code from the dropdown.
In the code editor’s left sidebar, look for a Templates folder. If robots.txt.liquid already exists, click it to open. If it doesn’t, click Add new template at the top of the sidebar, select robots from the template type list, and click Create template.
The default template uses a Liquid for loop to render Shopify’s built-in rules. Add your custom Disallow rules below the existing loop. Do not delete the default loop; it preserves the paths Shopify blocks out of the box (admin, checkout, cart, etc.).
Click Save. Changes go live immediately. Open a browser tab and visit https://yourstore.com/robots.txt to confirm your new rules appear correctly in the rendered file before crawlers pick them up.
One important note from Shopify: customisations to robots.txt.liquid are unsupported, meaning Shopify’s own support team won’t troubleshoot issues introduced by custom rules. For technical SEO work at this level on a large catalogue, it’s worth having someone with ecommerce SEO experience review the changes before saving.
Managing crawl budget for Shopify faceted navigation
Faceted navigation is the single biggest crawl budget problem for Shopify stores with catalogues over a few hundred products. Every time a customer filters by size, colour, price, or brand, Shopify generates a new URL with query parameters. A clothing store with 10 colours, 8 sizes, and 5 sort orders across 200 collections can produce over 80,000 unique URLs, almost all of them variations of pages Google already has, none of which will ever rank independently.
Google allocates a finite crawl budget to each domain based on its authority and server speed. When that budget is consumed on filter URLs, the product and collection pages that actually drive revenue get crawled less frequently. New products take longer to appear in search results. Category page updates take weeks to register.
RECOMMENDED ROBOTS.TXT RULES FOR FACETED NAVIGATION
Add these rules inside a User-agent: * block in your robots.txt.liquid file, below the default Shopify rules loop. Adjust the patterns to match your specific filter parameter names.
Disallow: /collections/*?* Disallow: /collections/*+* Disallow: /search?*
The /collections/*?* pattern blocks any collection URL containing a query string (everything after the ?). This covers sort, filter, and pagination parameters. Review your specific URL patterns in GSC’s Page Indexing report first; if your store uses path-based filters rather than query strings, the pattern will differ.
Use robots.txt disallow alongside canonical tags on your collection template pointing back to the canonical collection URL. The two work together: robots.txt prevents the crawl; canonical tags tell Google’s index which URL to treat as authoritative if a filter page is ever reached from an external link. Either alone is less effective than both together. For a full technical SEO framework covering crawl budget, indexability, and site architecture, see our technical SEO workflow guide.
The AI crawler decision: allow or block GPTBot, ClaudeBot, PerplexityBot?
Shopify’s default robots.txt allows all AI crawlers full access to your store: GPTBot, ClaudeBot, PerplexityBot, and CCBot can all crawl your product pages, category pages, and blog content unless you explicitly disallow them. For most ecommerce stores in 2026, that default is actually the right choice. But it’s worth making that decision deliberately rather than discovering it after the fact.
Each AI crawler serves a different function, with different implications for whether your products appear in AI-generated search results. Here’s what each one does and what to consider before blocking or allowing it.
Bot
GPTBot is OpenAI’s crawler. It collects training data for GPT models and also powers ChatGPT Search’s real-time web results. Allowing GPTBot means your product pages can be cited when ChatGPT answers shopping queries like “best waterproof hiking boots under $200 Australia.” Blocking it removes you from both training data and ChatGPT’s live search index. For most ecommerce stores, this is worth keeping open. Block it only if you have proprietary pricing or exclusive catalogue data you don’t want OpenAI to use in training.
Bot
ClaudeBot is Anthropic’s training and search crawler, used for both building Claude’s knowledge base and for Claude’s live web search feature. Claude is increasingly used by business buyers for research and product comparisons. Allowing ClaudeBot means your store can appear in Claude’s responses. The user-agent string is ClaudeBot; note that older documentation references anthropic-ai and claude-web, both deprecated as of 2024. If you have existing robots.txt rules referencing those, they’re no longer effective.
lexity
PerplexityBot crawls your store for Perplexity’s real-time search index. Perplexity is particularly active in “best of” and “where to buy” queries, exactly the searches ecommerce stores want to appear in. Allowing PerplexityBot is strongly advisable for most Shopify merchants. Perplexity also cites sources directly in its answers, which drives referral traffic as well as brand awareness.
Bot
CCBot collects data for the Common Crawl dataset, a large public archive used as training data by many AI labs and researchers. Blocking CCBot reduces your presence in a wide range of AI training pipelines. For most ecommerce stores the content is not sensitive enough to justify blocking, and the broad coverage benefit outweighs the risk. Stores with exclusive or proprietary content may reasonably choose to block it.
To block a specific AI crawler, add a targeted user-agent rule in your robots.txt.liquid. For example, to block GPTBot while keeping Googlebot and others: add User-agent: GPTBot followed by Disallow: / in a separate block above the wildcard User-agent: * section. The same pattern applies to any other bot.
At Shout Digital, our default recommendation for ecommerce stores is to allow all four crawlers and focus energy on making sure your product and category pages are structured so AI models actually cite them. Blocking a crawler is a simple robots.txt change; building the content quality and authority that gets an AI to recommend your store takes considerably more work. Our AI search visibility service covers both the technical access side and the content optimisation side of this. For brand-specific GEO strategy, the Answer Engine Optimisation guide is a good starting point.
What you can’t control in Shopify robots.txt
robots.txt is a crawler access protocol, not an indexing control: there’s an important distinction between the two that Shopify store owners regularly confuse. Disallowing a path tells well-behaved crawlers not to visit it. It does not prevent that page from being indexed if Google finds it through another route, such as a backlink from an external site.
Three specific things the Shopify robots.txt file cannot do:
| What you want | Why robots.txt won’t do it | Better tool |
|---|---|---|
| Remove an already-indexed page from Google | Disallowing a crawled page stops future crawls but doesn’t remove the cached version from the index immediately | Add a noindex meta tag, then use GSC’s URL Removal tool |
| Hide a specific page’s prices from Google | Disallowing the page stops crawling, but Google may still index it from external links and show a cached version | noindex meta tag on the specific page template in theme.liquid |
| Allow a sub-path inside a disallowed parent | Shopify’s Liquid template makes complex Allow/Disallow logic unreliable; some crawlers interpret Allow/Disallow precedence differently | Restructure disallow rules to be path-specific, or use canonical tags |
For hiding individual pages from search engines on Shopify, the more reliable approach is adding a noindex meta tag via the theme.liquid file. This tells Google not to index the page regardless of how it was found, and is faster than waiting for robots.txt-based deindexation. For a full checklist of indexing controls and when to use each, the Google Search Console guide covers the Page Indexing report in depth.
Testing your robots.txt changes
Always verify changes to your Shopify robots.txt immediately after saving, using at least two checks: a direct browser fetch and the Google Search Console URL Inspection tool. These two tools show you different things, and both matter.
Browser fetch (immediate). Open a new browser tab and navigate to https://yourstore.com/robots.txt. You should see the rendered file as a plain-text document. Scan it for your new rules and confirm no existing rules were accidentally removed. This confirms the Liquid template saved and rendered correctly.
Google Search Console URL Inspection (within minutes). In GSC, go to URL Inspection and paste a URL from a path you’ve just disallowed. Run a live fetch. If the page returns “URL is not available to Google” or shows a robots.txt block, the rule is working as expected. If it’s still showing as accessible, check whether the user-agent block is correctly structured and that no other rule is overriding it.
GSC Page Indexing report (ongoing). After a few days, check the “Blocked by robots.txt” count in the Page Indexing report. A significant increase confirms Googlebot is picking up the new rules. For a store that previously had 14,000 filter URLs being crawled, you should see that number shift into the “Blocked by robots.txt” category rather than “Discovered: currently not indexed.” That’s the right outcome: deliberate exclusion rather than accidental neglect.
Frequently asked questions
Updated June 2026. Shout Digital is a Melbourne-based ecommerce and technical SEO agency with 15+ years of experience managing Shopify SEO for growth-stage and established brands. For related reading, see our Shopify SEO fundamentals guide and our ecommerce SEO services.
We are digital marketing experts
