Most business owners have no idea that their website might be invisible to ChatGPT, Perplexity, and Google AI Overviews. And the ones who do know often assume blocking AI crawlers is the safe, smart move.
Sometimes it is. Often it isn't. And getting this wrong is quietly costing businesses leads they'll never know they lost.
Here's how to figure out which side of the line you're on.
How AI Search Actually Works
When someone asks ChatGPT "best personal injury lawyer in Edmonton" or Perplexity "top mortgage brokers in Calgary," those tools don't pull from a paid ad database. They crawl the web, read websites, and recommend businesses based on what they find.
No payment. No bidding. No shortcut.
AI tools decide who to recommend based on two things: whether they can access your site at all, and how well your site communicates who you are and what you do.
If your site blocks AI crawlers, you don't exist. If your site is slow, poorly structured, or sends mixed signals about your services, you get buried. Either way, your competitor who figured this out gets the referral.
The gatekeeper is a file on your server called robots.txt. It tells crawlers what they're allowed to read. Most business owners have never seen it. Some hosting providers and security tools, Cloudflare being the most common, add AI blocks to it automatically. Silently. Without telling you.
Businesses That Should Welcome AI Crawlers
If your business model is "client finds you, client contacts you, transaction happens offline," AI search is pure upside. There is no reason to block it.
Law firms. A potential client asking ChatGPT for the best personal injury lawyer in their city and getting your firm recommended is a free, high-intent lead. Your website's only job is to get people to call. AI search does exactly that.
Medical and dental practices. People search for providers constantly, and AI recommendations carry real trust. If your site is open and well-structured, you show up. If it's blocked or slow, the clinic down the street does.
Accountants and financial advisors. High-value, relationship-based services where being recommended by AI before a competitor is a genuine edge.
HVAC, plumbing, and trades. These searches are urgent and high-intent. Nobody asks Google "best emergency plumber Edmonton" casually. AI search captures that urgency and routes it to whoever shows up. Most trades sites are so poorly built that a properly optimized static site dominates almost immediately.
Mortgage brokers and real estate agents. People making the biggest financial decision of their lives are doing a lot of AI-assisted research. Being visible in that research pipeline matters.
Therapists and counsellors. Demand is high, supply of good websites is low, and AI recommendations carry particular weight in a space where trust is everything.
The common thread: these businesses don't sell content. They sell access to a person. AI reading their site and recommending them is free marketing.
Businesses That Should Block AI Crawlers
If your revenue depends on people actually visiting your site and consuming your content, AI summarizing that content for free is a direct threat to your business model.
News and media publishers. If ChatGPT summarizes your article, the reader never clicks through to see your ads. This is why major publishers have been aggressive about blocking AI training crawlers, and why the New York Times took legal action against OpenAI. The math is simple: pageviews pay the bills, and AI summaries kill pageviews.
Paywalled content and subscription products. Research databases, legal research platforms, paid newsletters, online courses. If someone pays for access to your content, you don't want AI giving it away for free.
Ecommerce with proprietary pricing or product data. Competitors can use AI tools to monitor and undercut your pricing strategy if your site is fully open.
Niche content businesses built on affiliate revenue. Recipe sites, review sites, comparison tools. If your entire model is getting people to click through to a product, AI summarizing your recommendation removes the click.
Businesses That Should Do Both (And How)
This is where it gets nuanced, and where most advice falls short.
Some businesses have competing interests. They want AI to find them and recommend their services. But they don't want AI training on their creative work without compensation.
Photographers. A wedding photographer wants to show up when someone asks ChatGPT for photographers in their city. But they don't want their images scraped into an AI training dataset that generates competing images. These are two different things, and robots.txt can handle both.
Search recommendation crawlers like `GPTBot` (ChatGPT) and `ClaudeBot` (Claude) can be allowed while training crawlers like `CCBot`, `Amazonbot`, and `Bytespider` are blocked. One robots.txt file, two different outcomes.
Graphic designers and illustrators. Same situation. Your portfolio should be findable. Your original work shouldn't feed a model that generates competing artwork. Selective blocking is the right move.
Artists and creative professionals broadly. The line between "I want to be discovered" and "I don't want my work used without permission" is real, and you don't have to choose one or the other.
If you're in this category, the solution isn't to block everything or allow everything. It's to be deliberate about which crawlers get in and why.
How to Check If You're Accidentally Blocking AI Right Now
This takes about 30 seconds.
Go to your website URL and add `/robots.txt` at the end. So if your site is `example.com`, go to `example.com/robots.txt`.
Look for any of these lines:
```
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
```
If you see `Disallow: /` under any of these, that crawler is completely blocked from your site. If you see this and you're a local service business, you're invisible to AI search right now.
This is more common than you'd think. Cloudflare's bot protection settings add these blocks automatically for many users who never asked for them. WordPress security plugins like Wordfence do the same thing. It's one of the most common and most overlooked technical issues I find when auditing local business websites.
The fix is straightforward once you know it's there. But most businesses don't know to look.
The Bigger Picture
AI search isn't replacing Google. It's adding a new layer on top of it, one that's growing fast and that you can't buy your way into.
For local service businesses, this is actually good news. It levels the playing field. A well-built, properly configured website from a small firm can outperform a bloated corporate site because AI recommends based on technical quality and clarity, not ad spend.
For content businesses, it's a genuine threat that requires a deliberate response.
Knowing which category you're in, and acting accordingly, is the difference between AI search working for you or against you.
Want to know if your site is visible to AI search? I offer free audits for Edmonton businesses. Check your score at redwillowdigital.ca/ or email mandy@redwillowdigital.ca.