robots.txt

Oct 2, 2021 / Updated on Oct 2, 2021

Allow All

User-agent: *
Disallow:

Disallow All

User-agent: *
Disallow: /

Disallow specific files

User-agent: *
Disallow: /my-path/
Disallow: /some-path/some-file.html

Disallow: /my-dir/ prevents search engine indexing any files under the path /my-path/* Disallow: /some-path/some-file.html prevents just one file under /some-path/

Disallow specific bots

User-agent: Googlebot
Disallow: /

User-agent: *
Disallow:

This disallows Google search engine bot alone and allows all other search engines

Page meta data to prevent indexing the page

<meta name='robots' content='noindex,nofollow,noarchive,nosnippet' />

specific bot

<meta name="googlebot" content="noindex" />

Response header to prevent indexing the page

X-Robots-Tag: noindex

References

comments powered by Disqus
Categories:
Tags: