Announcement

Collapse
No announcement yet.

how do I block ***ALL*** spiders/bots/non-human users?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • how do I block ***ALL*** spiders/bots/non-human users?

    I had it set up previously and somewhere along the lines a server update has deleted my old .htaccess file that blocked all index spiders and everything, I've tried searching on google for how to do this to no avail, it seems I can find "how to block bad spiders" but, I don't care what they are, I want them all blocked.....

    any assistance would be appreciated....

    thanks

  • #2
    The common way to block spiders is to save a robots.txt in your sites root directory e.g. website.com/robots.txt and define what robots can and can not crawl.

    See here for more info: http://www.robotstxt.org/robotstxt.html

    Bare in mind that this file can be publicly accessed by typing in the web address, so any locations you want hidden should not be placed into the file.

    Reece

    Comment


    • #3
      Unique Value Proposition

      awesome work.

      Comment

      Working...
      X