htaccess Archive

Block Bad Bots and Spiders With a robots.txt File

In an earlier post, I talked about how to block bad bots and spiders from searching your website using a .htaccess file. In addition to that method, I am also using a similar technique with the robots.txt

Block Bad Bots and Spiders Using .htaccess

In setting up this site, I discovered that unscrupulous people set up bots to harvest email accounts, as well as my content. Yes, I have found exact copies of my stuff on other sites. This is by