probeo

How to move X-Robots-Tag directives from HTML meta tags to HTTP headers?

robots-directives-meta-only global

The issue

A robots directive written as <meta name="x-robots-tag"> may not be obeyed. A page meant to stay out of search can be indexed anyway, and nothing on the page shows it. Google documents robots meta tags under name="robots" or a specific crawler’s name; X-Robots-Tag is the name of an HTTP header.

The cause is mixing the two mechanisms. The directive was meant for the response header, where it can also cover PDFs and images, but ended up in the HTML under the header’s name.

Scope

Probeo checks each page’s HTML for a meta tag named x-robots-tag. Where the directive should live depends on its reach: a site-wide policy belongs in server configuration, a page’s own directive in that page’s response or its robots meta tag.

How to check you have it

  • In probeo: the page reports Found <meta name="x-robots-tag"> in HTML; this directive is typically sent as an HTTP header.
  • By hand: view the page source and search for x-robots-tag. Run curl -sI <url> to see whether an X-Robots-Tag header is actually sent.
  • With a tool: the URL Inspection tool in Google Search Console shows whether Google found a noindex on the page.

The fix

Send the directive as an X-Robots-Tag HTTP header, or as <meta name="robots">, and remove the x-robots-tag meta tag.

Use the header when the rule should cover files that have no HTML, such as PDFs and images, or when it is set at the server or CDN. Use the robots meta tag when a page’s own template sets it. Combine directives with commas, as in X-Robots-Tag: noindex, nofollow.

Done when: each directive is sent as a header or a robots meta tag, no x-robots-tag meta tag remains, and the check above comes back clean.

Do not

  • Do not name a meta tag x-robots-tag.
  • Do not send directives in the header and the meta tag that disagree.
  • Do not forget the file types that can only take the header.
  • Do not put page-specific directives in global server configuration.

FAQ

Should HTML pages use HTTP headers, meta tags, or both for robots directives?

Either works for HTML. The header suits rules set at the server or CDN, and is the only option for non-HTML files. The robots meta tag suits rules set per page in a CMS. Using both is unnecessary, and a source of disagreement.

What syntax should be used for multiple directives in X-Robots-Tag headers?

Comma-separated in one header, as in X-Robots-Tag: noindex, nofollow, or one directive per header. Google documents both forms.

Can X-Robots-Tag headers target specific search engines?

Yes. Prefix the directive with the crawler’s name, as in X-Robots-Tag: googlebot: noindex. Unprefixed directives apply to all crawlers.

How do X-Robots-Tag headers interact with robots.txt?

robots.txt controls whether a URL is fetched; X-Robots-Tag controls what happens after it is. A URL blocked in robots.txt is never fetched, so its header is never seen. To keep a page out of the index, allow the crawl and send noindex.

Where should X-Robots-Tag headers be configured for maximum efficiency?

At the server, reverse proxy or CDN for rules that cover many URLs or whole file types. In the application for rules that depend on the page.

What happens when X-Robots-Tag conflicts with meta robots tags?

Google applies the more restrictive rule. A header saying noindex and a meta tag saying index produce noindex. Keep one source per rule to avoid the question.

Which product detects this

Probeo checks this on every page of your site as part of Site Clarity. This page explains one instance; the product tells you every page that has it.