Meta Robots Tag Generator
Tick the directives you need and get the meta robots tag, plus the equivalent X-Robots-Tag HTTP header for PDFs and images. A plain-English summary tells you exactly what the combination does.
Everything is at its default (index, follow). This tag is harmless but redundant, you can omit it entirely.
X-Robots-Tag HTTP header (for PDFs, images, and other non-HTML files)
Every robots directive, in plain English
- index / noindex
- Allow or forbid the page from appearing in search results.
- follow / nofollow
- Allow or forbid crawlers from following the links on the page.
- noarchive
- Do not show a cached copy of the page in search results.
- nosnippet
- Show no text snippet or video preview at all for this page.
- noimageindex
- Do not index images on this page (they can still be indexed if linked elsewhere).
- notranslate
- Do not offer machine translation of this page in results.
- max-snippet:[n]
- Limit the text snippet to n characters. -1 means no limit, 0 means no snippet.
- max-image-preview:[size]
- Largest image preview allowed: none, standard, or large. Large is needed for Discover.
- max-video-preview:[n]
- Limit video previews to n seconds. -1 means no limit, 0 means a still image only.
- unavailable_after:[date]
- Stop showing the page in results after this date. Useful for expiring offers and events.
- none
- Shorthand for noindex, nofollow.
- all
- Shorthand for index, follow, the default. Redundant in practice.
Meta tag or HTTP header?
The meta tag only works in HTML, because that is where a meta element can live. For PDFs, images, video files, or any other non-HTML resource, send the same directives as an X-Robots-Tag response header from your server or CDN. The header also wins when you need to apply a rule across thousands of files with one server config line instead of editing pages.
The one combination to avoid
Never pair noindex with a robots.txt disallow on the same URL. The disallow stops Google from fetching the page, so it never sees the noindex, and the URL can linger in the index as a bare link. Let Google crawl the page, let it read the noindex, and the page drops out.
Frequently asked questions
Do noindex pages waste crawl budget?
Google must crawl a page to see its noindex, so a noindexed page still gets crawled, though less often over time. The important rule: never block a noindex page in robots.txt at the same time. If crawling is blocked, Google cannot read the noindex, and the URL can stay indexed from external links alone.
Should I use nofollow on internal links?
Almost never. Nofollow on internal links throws away link equity rather than redirecting it, and Google treats nofollow as a hint anyway. If you do not want a page indexed, noindex it. If you do not want it crawled, use robots.txt. Internal nofollow solves neither problem cleanly.
What is the difference between meta robots and robots.txt?
Robots.txt controls crawling: it tells bots which URLs they may fetch. Meta robots controls indexing and presentation: what Google may do with a page it has already fetched. They are not interchangeable, and combining disallow with noindex on the same URL is the classic mistake, because the noindex never gets seen.
What happens if a page has no robots tag at all?
Index, follow is assumed. The defaults are permissive, so you only need the tag when you want to restrict something. A tag reading index, follow is harmless but redundant, and you can omit it entirely.