thegroveclermont.com

Many sites sit behind a firewall or bot-protection service that blocks automated visits. An agent stopped at the door reads nothing, so it answers from other sites. Claude's web fetch, for example, gets only what your site sends back.

AI services send crawlers, robots that read sites for them. One your robots.txt blocks cannot use your pages, so answers about your business come from other sites. OpenAI, Anthropic and Perplexity each say their crawlers obey robots.txt.

A page search engines may not list is missing from Google's and Bing's results, and from AI answers built on them, such as Google's AI Overviews. Brave's search, which some AI agents use, follows Google's rules.

Google's crawler skips any line of robots.txt it cannot read, so a rule you meant to set never applies. A page you wanted found may be left out, or one you wanted kept out may be read.

Google and Bing use a sitemap to find your pages, though neither promises to list every page in it. A page must be in Google's index to appear in its AI answers.

Search engines find most of your pages by following links. Google follows only real links with an address, so a page reached only through code may never be found.

Google builds the link to your page in its results from the page's title, and may rewrite one that is missing, vague or repeated. A clear title keeps your own words in front of searchers.

When a missing page says so, Google drops the address from search. When it answers like a normal page, Google flags a soft 404, and every mistyped address looks like a real page.

Google shows only one address for a page. When you name it, the page's standing stays in one place and searchers land on the version you chose, in their language. When you don't, Google guesses.

Each extra redirect is another trip across the network that can delay the page by hundreds of milliseconds. Every visitor waits for every hop.

Agents read the catalog before they open a page, so they know straight away which of your services they can use, such as your MCP server. Lighthouse checks it, and the grade counts standards likely to be adopted, such as this draft.

Agents read Markdown faster and more accurately than a full web page, and it costs them fewer tokens, the units AI services count text in. Cloudflare calls Markdown the common language of agents.

Content Signals let you say, in one line, whether AI may use your content for answers, for training, or both. They follow AI Preferences, a proposal at the IETF, the body that sets internet standards.

Many agents read only the page your server sends and run none of its scripts. If your text appears only after scripts run, those agents see an empty page and answer from other sites. Claude's web fetch is one of them.

Structured data saves agents and search engines from guessing your type of business and contact details from the page. Google says it gives explicit clues about what a page means. Bing says it may help and promises nothing.

Clear headings show search engines and agents how a page is laid out, so they find the part that answers a question, such as your opening hours. Google's guide to its AI features recommends them.

Agents rely on an image's text description, called alt text, to know what it shows, the same description that lets people who cannot see the image use the page. Google calls alt text the most important attribute for describing an image.

Agents read a list or a table as one only when the page's code marks it as one. Otherwise your prices or opening hours reach them as loose lines. The same code lets people who use screen readers follow them, and WCAG requires it.

llms.txt gives agents a short map of your site, so they reach your key pages without reading every one. Lighthouse checks it as an emerging convention, and the grade counts standards likely to be adopted.

Google may show your summary under the page's title in its results, and asks for a different one on every page. Without one, it picks a snippet from the page, which may be a line you would never choose.

A declared language tells software how to read your page, from the voice a screen reader uses to the language a translation starts from. WCAG, the accessibility standard most laws point to, requires it.

Your site can tell agents what it lets a customer do, for example book a tour, reserve a table, ask a question, or search what's available. An agent can then do it for them, start to finish. Without that, the agent can only read your pages and send the customer to do it themselves, or do it on another site that lets it.

Some agents never open a web page. They connect to a business directly and work from there. Giving them a way in means your site can be searched and booked by those agents too.

Agents choose a button or link by its name. One without a name, such as an icon-only button, cannot be chosen, so the agent stops or goes elsewhere. The same names let people who use screen readers find their way.

Agents find what they can click or fill in through the accessibility tree, the browser's list of a page's controls. A control missing from it does not exist for an agent, so it cannot book, buy or send an enquiry. Screen readers use the same tree.

Agents trust these labels. A wrong one tells an agent a control is something it is not, such as a button marked as a heading, so it clicks the wrong thing or nothing. Screen readers trust them too, so people with disabilities are misled the same way.

Agents move through a page from control to control, as keyboard users do. A page that skips controls or refreshes on its own keeps them from the button they need. WCAG requires every function to work from a keyboard, which people with disabilities rely on.

Some agents look at a page as a picture, as people do, and pale text is easy for both to miss. The same contrast lets people with low vision read it. WCAG, the accessibility standard most laws point to, sets how much contrast text needs.

Agents that click where a button sits, as a person does, miss small or crowded targets, and so do people on phones and people with limited movement. Google's web.dev recommends targets of about 48 pixels, and WCAG sets a minimum of 24.

Each one tells agents about a service they can use, such as an API for checking stock. Few businesses that sell to the public need them yet, so they earn bonus points and never cost any.

Visitors leave pages that are slow to show anything, and Google uses this speed in ranking. It is called Largest Contentful Paint, one of Google's three Core Web Vitals.

A page that freezes after a tap feels broken, so visitors tap again or leave. Google rates a reaction under 200 milliseconds as good and counts it among its Core Web Vitals.

When content jumps as a page loads, visitors tap the wrong thing, such as a link that moved under their finger. Google counts layout shift among its three Core Web Vitals.

A secure connection stops others reading or changing what passes between your site and its visitors. Browsers mark pages without one as not secure, and Google counts it in page experience.

Every visitor and agent waits for your server's first response before anything else can load. Chrome says a CDN, a network of servers close to your visitors, cuts that wait.

An image bigger than the space it fills wastes data and slows the page, most of all on a phone connection. Smaller images mean a faster page for every visitor.

Heavy pages take longer to load and cost visitors on mobile data real money. Chrome says large downloads are closely tied to long load times.

Downloading every file again on each visit is slow and costly. When browsers keep the files, returning visitors see your page sooner.

An error in the browser often means part of your page is broken, such as a form that will not send. Chrome says each one is a problem to resolve.

People distrust or are confused by a page that asks for their location the moment it opens. A field that blocks pasting makes a form harder to fill in.