Why is Bing / MSN bot picking on me?

[clivejo]

I recently received an email warning that the average CPU load on one of my accounts was too high, and if it exceeded my limit, my account would be suspended. This account normally averages at 1-5% CPU but was over my 15% limit for 2 days running. I opened a support request and the very helpful technician pointed out that five 'bots' were continuously hammering my site!!

Now Ive had to ban the bots via my robots.txt which seems to be working, but I've noticed Bings referrals are sliding off. Does anyone know why Bing have decided all of a sudden to hammer my site and how can I manage it without a total ban?

I've emailed abuse@microsoft.com but no answer

[hockey97]

I recently received an email warning that the average CPU load on one of my accounts was too high, and if it exceeded my limit, my account would be suspended. This account normally averages at 1-5% CPU but was over my 15% limit for 2 days running. I opened a support request and the very helpful technician pointed out that five 'bots' were continuously hammering my site!!

Now Ive had to ban the bots via my robots.txt which seems to be working, but I've noticed Bings referrals are sliding off. Does anyone know why Bing have decided all of a sudden to hammer my site and how can I manage it without a total ban?

I've emailed abuse@microsoft.com but no answer

It's spider bots nature. Spider bots crawl the internet. They first require you to sign up with there service. You give them your website address. They then have their spider bot crawl your website.

What this means is that the spider views your website and then opens up all links on that website and views those other websites. I can't say exactly for sure but do know that sometimes spiders if they find a link or button or content clickable but they try opening to read it at times it gets stuck in a infiniti loop. Which means the spider gets stuck their and keeps hammering your server with requests.

They call these events a spider trap because it keeps pounding your server with requests but never getting correct results so it keeps going in a cycle.

you can read more about it on here:

http://en.wikipedia.org/wiki/Spider_trap

They claim that if you have a dynamic website then these may happen quiet often.

For my website I only allow the First page of my website since it's static to be crawled. I then ban bots from the rest of my website. So, I never had this issue yet.

Hope this helps any. My suggestion is to ban the spiders from the rest of the page but only allow the spider to crawl the first page of your website.

Hope this helps any.

[clivejo]

The content is very unique and records valuable local history, stories, maps and photographs. Therefore, I do want the bots to index it. But the content is stored in a wiki and is generated dynamically, so puts huge stress on the server.

Google and others have no problems with it, why is it just MSN bots?

is there any way to tell Bing bots to go easy/throttle back?

[cjadahy]

It is very possible that it is actually not Bing. In other words fake Bing bots which is common. Block the IP to your server through your firewall.

These are some recent IPs that appear to imitate Bing.

61.47.43.196
108.175.148.207
86.110.75.224
209.236.71.8
176.31.224.27
74.50.27.165
205.134.249.85
83.64.121.55
112.78.205.201
213.136.8.98
74.220.207.169
184.173.227.209
67.23.248.28
66.84.0.151
216.226.157.129
85.13.141.102
188.138.107.147
178.248.39.43
89.111.177.98
89.145.160.18
198.170.250.183