A view under the hood
by Volker Weber
I have an axe to grind. Before I start my rant, let me give you some numbers.
The page you are viewing is #4842 on vowe.net. This comment is #6996. This week and as of 4pm today local time the site has been referred to from 2651 distinct pages, one of them being macsurfer.com which sent 2354 referrers this way. Currently the site is being viewed by 29 people, and it has been viewed by 9778 distinct users this week, ringing up 59536 page views and 212436 hits.
At least one of those users has been abusing the site. That is business as usual. To give you an idea of how that looks from the inside, I have provided you with a typical excerpt of the log. Since the shields have not been completely up, I just had to remove a dozen spam messages.
I could close the comments altogether, and this would have an end. But I enjoy the conversation with you and so I am keeping them open. I would just like to take this opportunity to tell you that fighting the abuse is a lot more work than writing the normal content.
So please keep those comments coming in. I enjoy reading them. And I try as long as I can to keep the comments completely open, without registration, moderation or a Captcha controller.
But you also know now why I have zero tolerance against smartass people, who cannot stand by their comments with their real name and an email address that I can reply to. If you cannot trust me with your email address please take your business somewhere else. This has nothing to do with what you say. I left one comment live although all of the information provided was bogus, just because it was coming in from left field and tried to fire a cheap shot. (Don't ask, I took the conversation that developed from there down later).
You have seen that I am busy as hell fighting the abuse. While I am wielding that axe, I am cutting of any smartass comments as well. Sorry about that, but now you heard the rules again.
On a slightly related topic:
Those spammers try something that does not work, at least not with Google. They want to raise their pagerank by dumping links to their own sites all over the place. Google however does not like link farms. So you have to be careful not to overdo it. If one of those idiots dumps thousands of links into my site, then Google punishes me. That alone is reason enough to weed them out. If they would dump only one link at a time and then return days later with another one, that would work much better. (Actually some do that).
What can you take away from this? Look at your own site. Does it have hundreds of links in your "blog roll"? And do all the people you link to have the same links pointing to all of the other (same) sites? That sure looks like a link farm to me. And maybe to Google as well.
Comments
Interesting - I will remove my own blogroll section and see what happens...
You don't have to remove it. There are lots of other options. You can include it via Javascript, put smaller chunks on separate pages, or list only the sites that you read. :-)
I don't have anything like the same traffic as you, of course, but my comment spam has dropped to ZERO since migrating to WordPress.
The issue is IMHO twofold:
a) MT is widely used and therefore attractive to spammers; MT-Blacklist needs continuous maintenance to keep it "fresh"
b) WP provides better spam protection than MT, including the ability to block comments having more than "x" links embedded in the comments - which is usually a good indication of spam.
I'm not saying you should migrate to WP, there are other platforms that you might think about, but I do think the MT 2.x platform will remain a problem area...
I presume this is a case in point.
John, that is certainly the case. However I have no plans to migrate right now.
I don't get your traffic either, but Rocky gets almost the same number of hits I do and he says he gets much more spam. I have gotten fewer than a dozen spam comments thus far, total. Unfortunately, I have no idea why not, so I won't know how to stop them when they eventually come.
Three lines of defense:
1. Block IP from accessing the site. Don't know how to do this in Domino. Pretty easy in Apache.
2. Block IPs from commenting. Before you upgrade them to #1.
3. Filter content. This is a good import for your filter. Update once a day.
4. Throttle commens. No more than once a minute.
5. Block excessive posters. More than 10 comments in 15 minutes blocks the IP.
Actually, you don't have to get a lot of traffic to attract comment spammers. Just a few days ago I cleaned up a site that gets even less hits than my own site and that one only makes a fraction of vowe.net. Still, on that site I was presented with roughly 15.000 comments of which 14.700 were spam. All you have to do to get this is to run a standard installation of MT. It's like a honey pot for content spammers. Once they find a vulnerable system, they'll flood it. As Volker said, they're mostly after the search engines, so the number of visits by real users don't really matter.
For Domino, the server document, under 'Internet Protocols', has an ip address 'deny address' list field. Set the blacklisted ip address here, either manually or programmatically. Be certain to give priority to 'deny list'.
Does it accept ranges there?
Should accept * as wildcard for subnets, not sure about ranges.
I believe ranges are accepted, but I'm working from memory. The analogous fields for SMTP relaying have a bit more information about configuration. I'm presuming the HTTP settings work the same way.
Shelley Powers reports some ISP's MT users are getting hit hard enough with comment spam to bring down their servers:
http://weblog.burningbird.net/archives/2004/08/19/unscheduled-downtime/
Its a constant battle Volker - the people who create this spam traffic are the sc*m of the earth in my books.
I've implemented 3 lines of defense in dominoblog to battle against comment spam which seems to have stopped the flow into my own site (but I wasn't getting the volume you were).
One line of defense I found very simple but effective (I dont know whether you can do this with your blog) is on the onsubmit event add some extra text to the comment post - then server side check whether that text exists and if not reject it.
This works because the agents scan real time the post html on the site and use it - but have no way of detecting if you change that post html after the submit.
Next stop the battle against referrer spam - I've not had an issue with this so far - but only a matter of time!
hmm - dominoblog - although www.dominobog.com is funnier! Maybe we could put some analyst reports there!
I'm not sure how reliable Domino's IP blocking is. I've been getting quite a bit of referrer spam from a small number of IP addresses. I've added maybe a dozen IP addresses to the deny list in an effort to cut it down. Over the past week, my referrer spam suddenly spiked up again. I tracked down the sources, and found it was all coming from two IPs that are already in the list. I've restarted HTTP, and they're still getting through. I haven't tried restarting the entire server yet. Perhaps that will do the trick.
-rich
