Register a free account to unlock additional features at BleepingComputer.com
Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.
Spiders will index your site regardless of whether you want them to or not. It is only by polite convention that some developers create spiders that will abide by the contents of robots.txt; it is after all merely a text file.
If your web host is using Apache web server, then you can place an .htaccess file inside the folder your do not want anyone to access. Inside this put a line Options -Indexes. Also do not put any link to the pages inside that folder, anywhere on your site (or any other site) so a crawler cannot get them.
If your web host is using anything else than Apache, then I am sorry I do not know.