Jump to content


Register a free account to unlock additional features at BleepingComputer.com
Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.

Click here to Register a free account now! or read our Welcome Guide to learn how to use this site.


Spiders and boots

  • Please log in to reply
2 replies to this topic

#1 JUICYboy


  • Members
  • 537 posts
  • Gender:Male
  • Location:Anaheim, Ca
  • Local time:10:29 AM

Posted 10 May 2011 - 08:54 PM

The question came up:

Do spiders update every website or just websites that have been edited with content?
And if so how does it know? Does the servers send a link to the search boots? :scratchhead:

BC AdBot (Login to Remove)


#2 shenandoah2


  • Members
  • 4 posts
  • Gender:Male
  • Local time:10:29 AM

Posted 11 May 2011 - 12:22 AM


I think that the spiders and bot's are constantly scanning all websites. There are good bot's and bad bot's out there......

The good bot's will follow instructions and of course the bad bot's will scan what they can get to.

<meta name="robots" content="index,follow" />
<meta name="revisit-after" content="1 days" />
<meta name="revised" content="webmaster of this site, 07/10/10" />

The above are only a very small example of what I have on my pages in an attempt to talk to the bot's. There can be some good debate as to if/what the bot's look at. Though when I check out my server logs, I can see that the bot's seem to pick and chose what they want to see. I believe that the bot's are now tending to ignore meta tag's and probably just look for a "last modified" date. I'm doing a little guessing, but I figure that the bot's can just check out your files and compare the dates and go from there.

#3 Romeo29


    Learning To Bleep

  • Members
  • 3,194 posts
  • Gender:Not Telling
  • Location:
  • Local time:09:29 AM

Posted 13 May 2011 - 12:53 PM

I think Google Bot updates everyday many times. I published an article on my site and 2 hours later I googled it, it was indexed!
Some blogging software like Wordpress also ping the search engines to inform that content has been updated, come visit me.
You can also manually ping these servers, so they know your content is updated and now they can check.

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users