Posts in 2021
-
robots.txt
Saturday, October 02, 2021 in Web
Categories:
less than a minute
Allow All User-agent: * Disallow: Disallow All User-agent: * Disallow: / Disallow specific files User-agent: * Disallow: /my-path/ Disallow: /some-path/some-file.html Disallow: /my-dir/ prevents search engine indexing any files under the path …