Tags: robots.txt

robots.txt

Oct 2, 2021 1 min.

Allow All User-agent: * Disallow: Disallow All User-agent: * Disallow: / Disallow specific files User-agent: * Disallow: /my-path/ Disallow: /some-path/some-file.html Disallow: /my-dir/ prevents search engine indexing any files under the path /my-path/* Disallow: /some-path/some-file.html prevents just one file under /some-path/ Disallow specific bots User-agent: Googlebot Disallow: / User-agent: * Disallow: This disallows Google search engine bot alone and allows all other search engines Page meta data to prevent indexing the page <meta name='robots' content='noindex,nofollow,noarchive,nosnippet' /> specific bot