MSNPTC robot attacks? (question for server admins more than programmers)

[Victor]

Hi,

As of yesterday, this forum's being scanned periodically by the bot MSNPTC. This appears to be some (new?) microsoft search engine bot.
The problem here is that this bot executes literally hundreds of simultaneous requests, that actually completely flood the forum!

I've now disallowed the bot in robots.txt hoping it will leave us alone now. If it doesn't I guess I'll have to go for heavier measures like firewalling the entire IP range it uses. This shouldn't be needed though - bots simply shouldn't act like DoS attacks :scratchch

Does anyone know about this or has experienced the same? Google didn't tell me much about it after a first search.
I'm just curious if this is a known problem or if anyone heard anything about it.

[ORION]

hm I havent had problems with that one yet, but another bot downloaded my whole website like 4000 times in a row, causing about 33GB traffic in a couple hours...
However, its not new - this thread is from 2004:
http://www.webmasterworld.com/forum11/2451.htm

[Victor]

can't view that link :

Forbidden

You don't have permission to access /forum11/2451.htm on this server.

[AndroidXP]

:confused: I can look at it without problems...

Maybe try a proxy (http://hiddenip.org/index.php?q=aHR0cDovL3d3dy53ZWJtYXN0ZXJ3b3JsZC5jb2 0vZm9ydW0xMS8yNDUxLmh0bQ%3D%3D&hl=3ed)?

[LRB_Aly]

Works for me too

[AndroidXP]

Guess that site has UK IPs blocked, as I got an "cannot access the site" when using an UK based proxy...

[Victor]

i'm in holland remember? ;)

Anyway, i could read that thread with that proxy link you showed. No real answers still. Nothing official anywhere. So I guess if i spot it flooding again, I'll just disallow access for msnptc altogether.

[St4Lk3R]

Short version of the thread for you, Victor:

the discussion is about MSNPTC/1.0
it has been seen under the IP-Adresses 131.107.3.74, 131.107.3.84, 207.46.238.143
it is confirmed that it does (or did) NOT request the robots.txt
wild guesses are going that its Partner-Bot msnbot relays the robots.txt to MSNPTC


Our team recently had the same problem with one of Yahoo!'s bots. We had no choice except actually Firewalling the bot's ip.

[Victor]

so this bot exists for 3 years now and still it's behaving in the same rediculous manner? Heh, goooooooooo MS!

[St4Lk3R]

c'mon, it's MS. What did you expect? :D

[EDIT: tags: microsoft bashing]

[ORION]

so this bot exists for 3 years now and still it's behaving in the same rediculous manner? Heh, goooooooooo MS!

well, intercrap exploder exists since 12 years and its still bugridden rubbish :D

saved the page from above, even though I just read you already read it...

[the_angry_angel]

For what it's worth, the inference appears to be that it's an ad-quality/adsense-like bot from Microsoft :(

[freddybob]

Not that I want to leap to MSN's defence, but they do state the range of IP addresses their bot crawls from.

http://advertising.microsoft.com/Home/Article.aspx?pageid=710&Adv_Articleid=3128

If it doesn't begin 65.54. it is a fake.

[Victor]

hmm ok thanks. The ip's i have logged were not in that range. So I may just firewall them then (the attacking range - not the range MS states there)

[Ian.H]

Seems it's been hitting my server for some time too, although it would appear (at least at this time), not viscously.. but the UA name appears to have changed too since it started:

131.107.0.77 - - [04/Dec/2006:19:39:25 +0000] "GET / HTTP/1.1" 301 239 "-" "MSNPTC/1.0 (compatible; MSIE 6.0; Windows NT 5.2; MyIE2; .NET CLR 1.1.4322; .NET CLR 1.0.3705)"
131.107.0.77 - - [04/Dec/2006:19:39:30 +0000] "GET / HTTP/1.1" 200 1331 "-" "MSNPTC/1.0 (compatible; MSIE 6.0; Windows NT 5.2; MyIE2; .NET CLR 1.1.4322; .NET CLR 1.0.3705)"
131.107.0.77 - - [08/Jan/2007:20:58:29 +0000] "GET / HTTP/1.1" 200 11441 "-" "MSNPTC/1.0 (compatible; MSIE 6.0; Windows NT 5.2; MyIE2; .NET CLR 1.1.4322; .NET CLR 1.0.3705)"
131.107.0.77 - - [09/Jan/2007:01:06:11 +0000] "GET / HTTP/1.1" 200 5647 "-" "MSNPTC/1.0 (compatible; MSIE 6.0; Windows NT 5.2; MyIE2; .NET CLR 1.1.4322; .NET CLR 1.0.3705)"
131.107.0.77 - - [16/Jan/2007:15:40:22 +0000] "GET / HTTP/1.1" 200 11797 "-" "MSNPTC/1.0 (compatible; MSIE 6.0; Windows NT 5.2; MyIE2; .NET CLR 1.1.4322; .NET CLR 1.0.3705)"
131.107.0.85 - - [20/Jul/2007:15:45:27 +0100] "GET / HTTP/1.1" 200 12160 "-" "MSNPTC/1.0"
131.107.0.85 - - [22/Jul/2007:20:10:41 +0100] "GET / HTTP/1.1" 200 12160 "-" "MSNPTC/1.0"


Now it appears to hit almost daily. Recent hits are coming from the 207.46.92.0/24 range.


OrgName: Microsoft Corp
OrgID: MSFT
Address: One Microsoft Way
City: Redmond
StateProv: WA
PostalCode: 98052
Country: US

NetRange: 207.46.0.0 - 207.46.255.255
CIDR: 207.46.0.0/16
NetName: MICROSOFT-GLOBAL-NET
NetHandle: NET-207-46-0-0-1
Parent: NET-207-0-0-0-0
NetType: Direct Assignment
NameServer: NS1.MSFT.NET
NameServer: NS5.MSFT.NET
NameServer: NS2.MSFT.NET
NameServer: NS3.MSFT.NET
NameServer: NS4.MSFT.NET
Comment:
RegDate: 1997-03-31
Updated: 2004-12-09

RTechHandle: ZM39-ARIN
RTechName: Microsoft
RTechPhone: +1-425-882-8080
RTechEmail: noc@microsoft.com

OrgAbuseHandle: ABUSE231-ARIN
OrgAbuseName: Abuse
OrgAbusePhone: +1-425-882-8080
OrgAbuseEmail: abuse@microsoft.com

OrgAbuseHandle: HOTMA-ARIN
OrgAbuseName: Hotmail Abuse
OrgAbusePhone: +1-425-882-8080
OrgAbuseEmail: abuse@hotmail.com

OrgAbuseHandle: MSNAB-ARIN
OrgAbuseName: MSN ABUSE
OrgAbusePhone: +1-425-882-8080
OrgAbuseEmail: abuse@msn.com

OrgNOCHandle: ZM23-ARIN
OrgNOCName: Microsoft Corporation
OrgNOCPhone: +1-425-882-8080
OrgNOCEmail: noc@microsoft.com

OrgTechHandle: MSFTP-ARIN
OrgTechName: MSFT-POC
OrgTechPhone: +1-425-882-8080
OrgTechEmail: iprrms@microsoft.com

# ARIN WHOIS database, last updated 2007-09-02 19:10
# Enter ? for additional hints on searching ARIN's WHOIS database.



Regards,

Ian

[freddybob]

Can anyone good with .htaccess files tell me how to block this user_agent only when it does not come from the IP range listed on microsoft.com?

[JamesF1]

Just use robots.txt and Disallow the MSNPTC/1.0 user agent.

[the_angry_angel]

order allow,deny
deny from IPHERE
deny from ANOTHERIPHERE
allow from allIf you want to deny an entire subnet, you're better off doing that at the firewall level.

[Dygear]

Just use robots.txt and Disallow the MSNPTC/1.0 user agent.

But that only works if the robot honors the deal a htaccess file on the other hand would stop it on the server level.

SetEnvIf User-Agent ^MSNPTC/1\.0 block
<Directory /docroot>
Order Allow,Deny
Allow from all
Deny from env=block
</Directory>


Something like that should work, if it does not you can find more help here:
http://httpd.apache.org/docs/1.3/mod/mod_access.html#allow

Damn you angel, you are quick

[the_angry_angel]

Damn you angel, you are quickI use apache too much :o

[GruntOfAction]

This reminds me of Lemon Deamon's flash "When robots attack".

"ROBOTS! ROBOTS! No1 is safe when Robots attack!" lol.

[freddybob]

Sorry, what I meant was how do I allow this user_agent when it comes from the 65.54.xxx.yyy IP range but disallow it when it comes from any other IP?

[Kada_CZ]

Sorry, what I meant was how do I allow this user_agent when it comes from the 65.54.xxx.yyy IP range but disallow it when it comes from any other IP?

SetEnvIf User-Agent ^MSNPTC/1\.0 BAD_MS_ROBOT
<Directory /docroot>
Order Deny,Allow
Allow from 65.54.0.0/255.255.0.0
Deny from env=BAD_MS_ROBOT
</Directory>I did not tested it, just a guess, no warranty :-). Notice also Ian.H's post about a new IP range.....

[JamesF1]

But that only works if the robot honors the deal a htaccess file on the other hand would stop it on the server level.
You are entirely right, sir. I shouldn't stray near the forums when I'm as shattered as I was when I posted that last night :nod:

[Dygear]

I'm just saying, your right, but so am I.