Posted 11 May 2011 - 12:22 AM
I think that the spiders and bot's are constantly scanning all websites. There are good bot's and bad bot's out there......
The good bot's will follow instructions and of course the bad bot's will scan what they can get to.
<meta name="robots" content="index,follow" />
<meta name="revisit-after" content="1 days" />
<meta name="revised" content="webmaster of this site, 07/10/10" />
The above are only a very small example of what I have on my pages in an attempt to talk to the bot's. There can be some good debate as to if/what the bot's look at. Though when I check out my server logs, I can see that the bot's seem to pick and chose what they want to see. I believe that the bot's are now tending to ignore meta tag's and probably just look for a "last modified" date. I'm doing a little guessing, but I figure that the bot's can just check out your files and compare the dates and go from there.