robots.txt

Tags:
Categories:

  less than a minute  

Allow All

User-agent: *
Disallow:

Disallow All

User-agent: *
Disallow: /

Disallow specific files

User-agent: *
Disallow: /my-path/
Disallow: /some-path/some-file.html

Disallow: /my-dir/ prevents search engine indexing any files under the path /my-path/* Disallow: /some-path/some-file.html prevents just one file under /some-path/

Disallow specific bots

User-agent: Googlebot
Disallow: /

User-agent: *
Disallow:

This disallows Google search engine bot alone and allows all other search engines

Page meta data to prevent indexing the page

<meta name='robots' content='noindex,nofollow,noarchive,nosnippet' />

specific bot

<meta name="googlebot" content="noindex" />

Response header to prevent indexing the page

X-Robots-Tag: noindex

References