Paste any URL and get one clear answer: can search engines index it? We check the HTTP status, robots.txt, meta robots noindex, the X-Robots-Tag header and the canonical tag together, so you don't have to. Free, no signup.
Free plan · Real fetch, five signals, one answerBeing allowed into the index is only step one. Run the free website audit to check titles, meta tags, headings, content and technical SEO.
Run Free Website Audit
A page can look perfectly healthy and still be invisible in search, because indexing can be switched
off in four different places, and they don't all show up in the page you see. A
robots.txt rule can stop crawlers fetching it. A meta robots tag can say
noindex in the HTML. An X-Robots-Tag HTTP header can say the same thing
where you'd never look. And a canonical tag can tell search engines that a different URL is
the real version, so this one gets dropped. Add a 404 or a redirect and there are five ways to quietly
lose a page.
This tool checks all five together, with real requests. It follows redirects, reads
robots.txt with a proper parser (user-agent groups, * and $ wildcards,
longest-match precedence, and how Google treats a missing or erroring file), parses the HTML with a real DOM
parser, reads the response headers including bot-specific googlebot: noindex values, and
compares the canonical to the URL actually served. Any single blocker turns the answer into a
No, and the report lists exactly which signal caused it and how to fix it.
It also flags the traps: a robots.txt block that hides a noindex from crawlers so it never
takes effect, a Noindex: line in robots.txt that Google no longer honors, and directives that
apply only to a different crawler. It can't see directives injected by JavaScript, and a "Yes" means nothing
is blocking the page, not that Google will choose to index it.
Indexability Checker: Find What Is Preventing Google from Indexing Your Website Pages
Publishing a webpage does not automatically make it visible in Google search results. Before a search engine can consider a page for indexing, it needs to discover the URL, access the content, process the page, and determine whether it should be included in its search index.
This is where website indexability becomes important.
Indexability describes whether a webpage is technically available for a search engine to include in its index. Several website settings can prevent this from happening, even when a page looks perfectly normal to human visitors.
For example, a blog post may load successfully in a browser but contain a noindex directive telling search engines not to include it in their index. Another page may be blocked by a robots.txt rule, return a 404 error, or contain a canonical tag pointing to a different URL.
These problems can be difficult to identify because they are controlled by different parts of a website's configuration. Checking only the visible HTML is not enough to understand every indexing restriction.
The Auditest Indexability Checker helps website owners, SEO professionals, developers, and digital marketers investigate these issues in one place. It performs real requests, follows redirects, examines robots.txt rules, parses page HTML, checks HTTP response headers, evaluates canonical URLs, and identifies important signals that may prevent a page from being indexed.
Instead of checking each directive separately, the tool combines five important indexability signals into a single assessment and explains which issues require attention.

A webpage's indexability can be affected by several technical instructions. The Auditest checker focuses on five major areas that website owners should understand.
The robots.txt file contains instructions that participating crawlers can use to determine which areas of a website they are allowed or disallowed from crawling.
It is normally available at the root of a domain:
https://example.com/robots.txt
For example:
User-agent: *
Disallow: /private/
This rule tells crawlers covered by the wildcard user-agent group not to crawl URLs under the /private/ path.
Robots.txt rules can be more complicated than a simple Allow or Disallow instruction. They may contain different user-agent groups, wildcard patterns, and end-of-URL matching rules.
The Auditest Indexability Checker uses a robots.txt parser that considers:
User-agent groups.
Wildcard matching with *.
End-of-URL matching with $.
Longest-match precedence when Allow and Disallow rules overlap.
How Google handles missing or unavailable robots.txt files.
This matters because a robots.txt rule can prevent Googlebot from fetching a page. When a crawler cannot access the page, it may also be unable to read a noindex directive contained in that page's HTML.
Important: Blocking a URL in robots.txt does not guarantee that the URL will disappear from search results. Google may still know about the URL through external links or other references, even when it cannot crawl the content.
A meta robots tag is an instruction placed inside the HTML document's <head> section. It tells supported search engine crawlers how to handle the page.
For example:
<meta name="robots" content="noindex, follow">
The noindex directive instructs compliant crawlers not to include the page in their search index.
A common alternative is:
<meta name="robots" content="index, follow">
However, index is generally the default behaviour, so explicitly adding it is usually unnecessary.
A page can accidentally receive a noindex directive through a CMS setting, SEO plugin, website migration, development configuration, or manually edited HTML.
For example, a developer may add noindex to a staging website and forget to remove it when launching the production site.
The Auditest checker parses the page's returned HTML using a DOM parser and examines its meta robots instructions to identify indexing restrictions.
What to check: Confirm that important public pages do not contain an unintended noindex directive.
Not every indexing instruction appears in the HTML source.
Web servers can also send indexing directives through HTTP response headers, particularly the X-Robots-Tag header.
For example:
X-Robots-Tag: noindex
This instructs compliant search engine crawlers not to index the relevant resource.
One advantage of X-Robots-Tag is that it can be applied to non-HTML resources such as PDFs, images, and other files.
It can also include crawler-specific instructions. For example:
X-Robots-Tag: googlebot: noindex
This targets Googlebot rather than applying the same instruction indiscriminately to every crawler.
These headers can be configured at the server, hosting, application, or CDN level. As a result, website owners may overlook them when inspecting only the visible webpage.
The Auditest Indexability Checker reads the actual HTTP response headers and checks for indexing directives, including bot-specific values.
What to check: Review the returned X-Robots-Tag headers and confirm that any noindex instruction is intentional and applies to the correct crawler.
A canonical tag tells search engines which URL should be treated as the preferred version among duplicate or substantially similar pages.
For example:
<link rel="canonical" href="https://example.com/preferred-page/">
This indicates that the specified URL is the preferred version of the page.
Canonical tags are useful when websites have similar content available through multiple URLs, such as:
Pages with tracking parameters.
HTTP and HTTPS versions.
URLs with and without trailing slashes.
Product pages with multiple filter or sorting parameters.
Duplicate pages created by a CMS.
However, an incorrect canonical can cause indexing problems.
Imagine that you publish an article at:
https://example.com/seo-guide/
But its canonical tag points to:
https://example.com/old-seo-guide/
Search engines may interpret the old URL as the preferred version, potentially choosing not to index the newer page separately.
Canonical tags are signals rather than absolute indexing commands. Search engines can select a different canonical based on their own evaluation.
The Auditest checker compares the canonical URL declared in the page with the URL actually served after redirects. This helps identify situations where the page points to a different address.
What to check: Confirm that important pages use an appropriate canonical URL, usually pointing to the intended preferred version of that page.
A webpage must also return an appropriate HTTP response before its indexing status can be assessed meaningfully.
Some common responses include:
| HTTP status | Meaning | Potential indexing implication |
|---|---|---|
| 200 OK | The request succeeded | The page can be retrieved and evaluated |
| 301 | Permanent redirect | Search engines are directed to another URL |
| 302 | Temporary redirect | Search engines are directed elsewhere temporarily |
| 404 | Not Found | The requested page does not exist |
| 410 | Gone | The resource has been permanently removed |
| 403 | Forbidden | Access to the page is denied |
| 500 | Internal Server Error | The server failed to process the request |
| 503 | Service Unavailable | The server is temporarily unavailable |
A page returning 404 or 410 is not a normal indexable page. A redirect also means that the original URL is not serving the final content directly.
The checker follows redirects and evaluates the final response, helping you identify whether the requested URL ultimately leads to an accessible page or encounters an error.
What to check: Make sure important pages return the expected response and that redirects lead to the correct destination.
The tool combines real HTTP requests with HTML and robots.txt analysis to investigate the main technical signals that affect indexability.
Here is how the checking process works:
Enter the webpage URL: Submit the full address of the page you want to test.
Fetch the URL: The tool sends a live request to the submitted address.
Follow redirects: If the page redirects, the checker follows the redirect chain and identifies the final destination.
Check robots.txt: It retrieves and parses the relevant robots.txt rules, considering user-agent groups, wildcard matching, and rule precedence.
Parse the HTML: It examines the returned HTML using a DOM parser to identify meta robots instructions and canonical tags.
Inspect HTTP headers: It reads response headers and checks for X-Robots-Tag directives, including crawler-specific instructions.
Evaluate the HTTP response: It considers whether the page returns a suitable response or encounters an error or redirect.
Compare the canonical URL: It checks whether the declared canonical matches the URL actually served.
Generate the indexability result: The tool combines the findings and reports whether it identified a technical blocker.
This approach provides a more complete baseline assessment than checking only the page source or robots.txt file independently.
The Auditest Indexability Checker provides a clear Yes or No assessment.
<escape>Yes – No detected technical blocker</escape>
A Yes result means the tool did not identify any of the supported blocking signals that would prevent the tested page from being indexable.
It does not mean Google has already indexed the page or will definitely include it in search results.
<escape>No – A potential indexability blocker was detected</escape>
A No result means that at least one of the checked signals indicates a technical restriction or problem.
The report identifies the relevant signal and explains what you may need to investigate.
For example:
Robots.txt prevents the relevant crawler from accessing the URL.
A meta robots tag contains noindex.
An X-Robots-Tag response header contains noindex.
The canonical points to a different URL.
The requested page returns an error or redirects rather than serving the expected content.
This makes it easier to identify the cause of an indexing problem rather than guessing which website setting is responsible.
| Problem | Why it matters | Recommended fix |
|---|---|---|
| Robots.txt blocks an important page | Search engines may be unable to crawl its content | Review the relevant Disallow rule and adjust it if access is intended |
| Meta robots contains noindex | The page is instructed not to appear in the search index | Remove the unintended noindex directive |
| X-Robots-Tag contains noindex | An HTTP header instructs crawlers not to index the resource | Review server, CDN, or application header configuration |
| Canonical points to another page | Search engines receive a signal that a different URL is preferred | Correct the canonical if the current page should be the preferred version |
| Page returns 404 or 410 | The requested resource is missing or removed | Restore the page or use an appropriate redirect where a relevant replacement exists |
| Page redirects unexpectedly | The original URL does not serve the expected content | Review redirect rules and final destinations |
| Robots.txt contains an obsolete Noindex directive | Google does not support Noindex as a robots.txt indexing directive | Use an accessible meta robots or X-Robots-Tag noindex instruction instead |
Always confirm that a detected issue is genuinely unintended before changing your configuration. Some pages, including private areas, duplicate content, campaign URLs, and temporary resources, may correctly be excluded from search indexing.
Some indexing problems are not obvious unless you understand how the different directives interact.
One common mistake is blocking a page in robots.txt while also placing a noindex instruction inside its HTML.
For example:
User-agent: *
Disallow: /private-page/
And inside the blocked page:
<meta name="robots" content="noindex">
The problem is that a crawler prevented from fetching the page cannot read its meta robots directive.
If your intention is to remove an already indexed page from Google, blocking it in robots.txt may prevent Google from seeing the noindex instruction. The page may remain eligible to appear in search results based on information Google already knows about the URL.
For a page that needs to be removed from the index through a noindex directive, allow the crawler to access the page so it can process that directive.
Some website administrators mistakenly add instructions such as:
Noindex: /private-page/
Google does not support the Noindex directive in robots.txt. It should not be relied upon to prevent indexing.
Use a supported method, such as a meta robots noindex tag or X-Robots-Tag response header, when you want to instruct Google not to index a resource.
Not every robots.txt rule or indexing directive applies to every search engine crawler.
For example, a rule targeting Googlebot may not apply to Bingbot. Similarly, a crawler-specific X-Robots-Tag instruction may affect one search engine while leaving other crawlers unaffected.
This is why checking the applicable user-agent group and directive is important when investigating differences between search engines.

One of the most important things to understand about technical SEO is the difference between a page being indexable and a page actually appearing in search results.
Indexability concerns whether technical restrictions prevent a search engine from including a page in its index.
Indexing is the process through which a search engine processes and potentially stores a page in its search index.
A page may pass every technical indexability check and still not appear in Google.
This can happen because of:
Poor or unhelpful content.
Duplicate or substantially similar content.
Weak internal linking and discovery signals.
Search engine quality assessments.
Canonical selection by Google.
Crawl prioritisation and other indexing decisions.
A Yes result from the Auditest checker means that the tested page has no detected blocker among the signals it examines. It is not a guarantee of indexing, ranking, or search visibility.
For confirmation of Google's own view of a URL, use the URL Inspection tool in Google Search Console.
The Auditest Indexability Checker is useful for:
SEO professionals: Diagnose technical issues that may prevent important pages from being indexed.
Bloggers and publishers: Check whether new articles are accessible to search engines.
WordPress website owners: Identify accidental noindex settings introduced by plugins, themes, or CMS configurations.
Web developers: Investigate robots.txt rules, response headers, redirects, and canonical configuration.
Digital marketing agencies: Include indexability checks in technical SEO audits and client website reviews.
E-commerce businesses: Review product, category, and landing pages for unexpected indexing restrictions.
Whether you manage a personal blog, business website, online store, or content platform, checking indexability is an important part of technical SEO maintenance.
An Indexability Checker is an SEO tool that examines a webpage for technical instructions that may prevent search engines from including it in their index. These include robots.txt restrictions, meta robots noindex tags, X-Robots-Tag headers, canonical URLs, and HTTP response problems.
Crawling is the process through which search engine bots discover and retrieve webpage resources. Indexing involves processing and potentially storing the retrieved content in a search engine's index. A page can be crawlable without being indexable.
Not necessarily. Robots.txt controls crawling, not indexing directly. Google may still know about a blocked URL through other signals and could show it in search results without a full content description. Use a supported noindex directive when your objective is to prevent indexing, and ensure Google can crawl the page to see it.
Noindex is an instruction telling compliant search engines not to include a page or resource in their search index. It can be delivered through an HTML meta robots tag or an X-Robots-Tag HTTP response header.
A canonical URL identifies the preferred version of a page when similar or duplicate content is available through multiple addresses. It helps search engines understand which URL a website considers the main version, although search engines may select a different canonical.
Yes. A page can return HTTP 200 while containing a meta robots noindex tag or an X-Robots-Tag noindex header. This is why checking the HTTP response alone is not enough.
No. A Yes result means the tool has not detected a blocker among the five technical signals it checks. Google makes its own indexing decisions based on additional factors, including content quality, duplication, canonical selection, and discovery signals.
The checker analyses the HTML returned by the server and does not execute JavaScript in a fully rendered browser. Therefore, it cannot reliably detect directives injected exclusively after the initial HTML response through JavaScript.
A webpage can be published, accessible, and visually perfect while remaining technically blocked from search indexing. The problem may be hidden in robots.txt, an HTML meta tag, an HTTP response header, a canonical declaration, or an unexpected server response.
The Auditest Indexability Checker brings these important checks together in one practical report. It follows redirects, examines robots.txt rules, analyses HTML meta directives, reads response headers, compares canonical URLs, and identifies common technical barriers that may prevent a page from being indexed.
Instead of guessing why a page is missing from search results, use the tool to investigate the technical signals that influence its indexability and identify what needs attention.
Test your important website URLs regularly, especially after website migrations, redesigns, SEO plugin changes, or updates to server configuration.
Visit Auditest's free website analysis tools to explore more tools for technical SEO, indexing, website performance, and search visibility.
Author: Kester Terna, Founder of Auditest
Last Reviewed: November 2026