Their Akismet spam comment blocking service is a godsend – without it we would quite simply be overrun with spam. It catches 15,000 or more spam comments per day and auto-deletes them.

I agree that Akismet has been a great help keeping the number of spams down here. It might not be 15,000 comment spams a day, but since I first deployed Akismet plugin on this blog, it has already blocked 220,000+ spams for me.

So you’ll know it when Akismet is offline — lots of comment spams came through waiting for manual moderation. I can’t seem to get onto their website, and their blog returns system error 500 every now and then. Anyone else experiencing the same? I won’t be surprised if they get DDoS’ed, which seems to be the norm these days with all the spam-fighting companies.

As you can see the Netural line has been burnt. They came and replaced the fitting, and the power is good again. Everything is up and running.

I have also been experimenting with greylisting using Gld + Postfix. It is kind of effective in reducing spams by asking the sender’s email server to try again. I am still having Amavis + SpamAssassin to do content filtering after the greylisting, but I am getting significantly less spams slipping through.

So, how does Akismet catch spams? How does it reduce false positives? What sort of algorithm does it use? Well, hmmm. We don’t know. Akismet is a centralised spam classifying service. For every comment received by your blog, it gets delivered to a centralised server, using a REST-based API. If the big brain on that server doesn’t like, it yells back “Spam!!!” and so that comment will be marked.

So, how does this centralised server determine whether a comment is a ham or a spam? According to the FAQ,

When a new comment, trackback, or pingback comes to your blog it is submitted to the Akismet web service which runs hundreds of tests on the comment and returns a thumbs up or thumbs down.

Hmm. Probably something like SpamAssassin but for blog comments. According to Michael Hampton, it “entirely replace plugins such as wp-hashcash, Spam Karma 2, AuthImage, etc” so I guess they must have sampled some of those implementations. Further on, he mentioned that he has “integrating CJD’s Spam Nuker”. So we probably get some idea what kind of backend does it have.

It also allows the users to manually classify comments as spam or ham. In the sense it might have some kind of Bayesian classifier that can be trained. Useful to report all the false positive. and false negatives.

So, what’s good about Akismet?

• A large sample of comment spams allows its Bayesian classifier to be thoroughly trained.
• Centralised service so Matt and co can do all the fine tuning without touching your site. No more updates for algorithm changes.
• Nice API that can be easily integrated into other blog tools. There might even command-line tools that can submit spam/ham in bulk.

But why I probably would not use it?

• A centralised server. I hate latency, especially my blogs are hosted somewhere half way around the world to Akismet’s central server.
• A centralised service. Just imagine millions of WordPress blogs download this plugin and deploy it today, and send millions of comments to this potentially CPU intensive classifying job…
• A centralised user-trained classification service. Although FAQ said that it is unlikely to poison the classifier (probably some kind of jail on a per-API key level), I just don’t feel right when someone anonymous blogger is moderating my comments.
• I don’t earn USD$500 a month blogging, but I hope one day I will. (Currently projecting when un*x time(2) wraps around…)
• But most importantly, I don’t get spams. Well. Rarely — to a point that it has never bothered me, when I require all first-time commenters to be moderated, which should be a default option for WordPress.

Still, I applaud for this great product. Not perfect, but probably still the next best thing than that red button labelled “Kill All Spammers”.

Update: Since I moved the site to DreamHost, I have actually started to use Akismet, and was surprised by the result — it is quite good. Centralised issue still concerns me. Things like Ping-o-matic outage can really stall blog posting, but fortunately Akismet plugin has good built-in timeout so that it will give up if the server is not responsive.

Not that I clicked on every referrer URL to see where it leads to. I love my statistics, and anything that skews the result annoys me. I do not want to open up my faviourite log analyzer, and see all the top referrers are pr0n or p0ker sites.

At the same time, it is also one of the most difficult method of spam to combat with, without causing too much inconvenience. You can put all the incoming comments and trackbacks through a moderation queue without really irritating your readers. Try to moderate every page view when it comes from an unknown referrer!

There has been quite a few attempts to resolve this issue in the past.

Mod_Rewrite

My initial attempt is by putting some mod_rewrite rules into .htaccess file, to return 403 access denied to referrer spammers/spambots. For example,

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_REFERER} pr0n.example.com [OR]
RewriteCond %{HTTP_REFERER} p0ker.example.com [OR]
RewriteCond %{HTTP_REFERER} r0ulette.example.com [OR]
RewriteCond %{HTTP_REFERER} l0an.example.com
RewriteRule .* - [F,L]

The problem is, my .htaccess file soon become a gigantic unmanageable piece of mess, as I started adding spammy host names one after another. Moreover, referrers spammers seem to have unlimited supply of domain names that there is just no way you can catch them all (also caused by free .info domains a while ago). And you normally only find new ones to add when the spambots have already visited your sites and tainted your logs.

Combating against referrer spams with mod_rewrite rules is a battle that can never be won.

Referrer Bouncer

Then I found Referrer Bouncer (via Blogging Pro News), a WordPress plugin that blocks referrer spams by matching against a plain text file. It gives some interesting responses — instead of denying spambot the access, it actually sends back a 302 “Found” to tell the spammers to go back its own website.

It is designed this way to punish the spammers to consume its own resource. However, the effect cannot be verified, as we do not know whether referrer spammers have actually implemented following the URL.

Does it work? Only slightly better than the old mod_rewrite approach. At least you will not render your blogsite inaccessible when you stuffed up your regular expressions in RewriteCond. However the old issues persists. It still does not address the ever-changing host names of referrer spams.

Referrer Karma

Last week I installed Referrer Karma (2.3b). The design philosophy is:

1. If the referrer exists in the white-list — in.
2. If the referrer exists in the black-list — out.
3. Fetch the content of the referrer, and if my site name can be found in the content — add to white-list and in.
4. Otherwise, add to black-list and out.

It is actually a bit more complicated than that, but you get the idea.

What is good about this approach is, it places all page views via referrers through a moderation queue, and it automates the moderation by checking whether the referrer is genuine. So there is no more weekly-hunt in my access_log files searching for fishy referrers, as those illegitimate ones would have been filtered out.

I am now running Referrer Karma on some of my sites, and it worked reasonable well. I can see logs of baddies getting rejected at the door, and the good guys getting welcomed in — most of the time, and at a price…

You see. The method of auto-moderation is by fetching the source content and try to find traces of links. However, a legitimate referrer might not have links inside the HTML content. Links might be loaded and generated via Javascript. It might be clicked from inside an <iframe/>. So in order to reduce false negatives, Spam Karma tried to retrieve up to 8 levels of recursion on Javascript and IFrames — breadth-first style traversal, all at once, regardless whether the main HTML content has already matched the links or not.

That is a lot of bandwidth usage, especially over some complicated sites where it might import 1-20 Javascripts or IFrames. Over the last couple of days my out-bound link was actually full-satuated for a few hours a day, and initially I could not figure out what caused it except for the fear of getting hacked or getting DDoS’ed. Well, you can blame my relative narrow pipe (only 512kbps out-bound), but after debugging and analysing Referrer Karma it turned out to the cause.

Bad bad coding.

A quick hack to change its behaviour to stop traversal as soon as a match is found, as well as reducing the depth to 2, solved my bandwidth problem. Referrer Karma is again useful now.

Conclusion

There are still issues that make Referrer Karma generating a lot of false negatives. For example,

• Source pages cannot be reached (side Intranet, behind HTTP auth, etc)
• Legitimate pages does not contain links (links not available to anonymous users, time-sensitive links)
• Obfuscated Javascript pages
• etc…

There is no perfect solution, but Referrer Karma (+ a few hacks) does make sites more spam-free.

Click on “Full Headers” reveals what Yahoo SpamGuard has done:

Sounds like my IP address has just been black listed, and it affects every single email originated from this IP – including personal emails sent by myself! In fact, I don’t recall myself sending any unsolicited commercial email lately, nor have I infected by virus – the mail server is guarded well with firewalls and anti-virii software. Who knows how my IP address ended up over there?! Anyway, if you are using Yahoo, and you are expecting an email from me (or from one of the FOCUS mailing lists), then please check your bulk mail folder. You might find surprises there…

What steps should I take? I might write to Yahoo complaining about the situation. If you have my emails (or other FOCUS related emails) in the bulk folder, please select them and mark them explicitly as “Not Spams” – maybe this can somehow re-train the filter.

Hopefully it is only temporary.

Updated 11 Feb 2005 @ 9:35am: Instead of asking Postfix to send emails directly to the destination, I am having it relaying to Exetel’s SMTP box. It seems to be working now, and all mails appear in Inbox!

So I fell back to the mighty MT BlackList, and tried to manually add offending domains to the black list. But the effort was in vain – all of them have different domains! One way to stop spamming is to reduce the commercial viability of the spammers, based on the principle that if they can’t make profit, they will stop. Black listing domains has been an implementation of that principle, assuming that the spammers cannot have unlimited supply of domains, as registration new domains on TLD costs money…

Until some registrars started to offer free .info domains for one year. When the news hit the street over 2 months ago, I initially thought that the idea was interesting – you can register up to 20 .info domains for free. Who would want to register that many second-level domains with only 1 year expiry, other than the evil domain squatters? But apparently spammers, who represent another form of evil on the Internet, have found a new application on these free domains – to defeat the black list.

That is exactly what has been happening. Many spam-bot generated comment spams over the last couple of days were pointing to many weird but unique .info domains. There is no way that I can block every single one of them with MT BlackList, especially when they can quickly register new ones at no cost!

At the end, I am very tempted to just block the entire .info domain from leaving comments on FOCUSer.net, since over the last 2 years no one has ever commented on these blogsites with a .info domain, other than the comment spammers. We’ll see.

SetEnvIf User-Agent libwww-perl NOPOST

<Limit POST>
    Order Allow,Deny
    Allow from all
    Deny from env=NOPOST
</Limit>

It seems that many spam bots are written in Perl, so here I simply reject their POST request if the user-agent matches. It is also easy to extend if there are variants. From the Apache log it rejects about 10 requests over the last 24 hours. Not bad I guess.

It does not work against the trolls though. I got 6 comments today on this site, with nothing but a swearing word starts with ‘F’. Should have block it from WordPress…

And yes, I am still here! Too many things to do over the last couple of days. Actually it was worse – got this mental block that I can’t find any interesting thing to blog. Expect it to continue for another few days…

Become a legally ordained minister within 48 hours

As a minister, you will be authorized to perform the rites and ceremonies of the church!

Perform Weddings, Funerals, Perform Baptisms, Forgiveness of Sins, Visit Correctional Facilities

Want to start your own church?

Then there is a link at the bottom of the email that points you to the website that sells you this certificate for USD$29.95. It has an interesting emphasis on the power of “forgiveness of sins”:

FORGIVENESS OF SINS

The Catholic Church has practiced the forgiveness of sins for centuries.

^** Forgiveness of Sins is granted to all who ask in sincerity and willingness to change for the better!!

Yeah, right! Why go to theological college and study hard for 3-4 years learning Hebrews, Greek, church history and the actual word of God, when you can be ordained on-line at fraction of money and time with free delivery! Moreover, at theological college they teach you that only God can forgive the sins that you have committed against Him. But hey! This certificate grants you the power to order God that He should forgive and forget other people’s sin as well!! How awesome is that! With USD$29.95, that sounds like a bargain to me…

I wonder how many people actually went and bought themselves a copy.

The installation is quite straight forward, except maybe the Perl modules as I have a broken CPAN configuration. Everything seems to be working fine so far. Last night it has successfully quarantined 20+ incoming Worm.SomeFool.Gen-1, one Win32.Yaha.P and a few different spams. Still trying to work out who is that MTS guy on the Optus network that has caught the virus…

The only complain I have is probably the amount of resources it requires to run Clam AV + AMaVis. When the daemons bootstrapped, clamd uses around 12Mb RSS and 3 instances of amavisd are running at 20Mb each! Scanning each incoming email would take at least 3 seconds on my aging dual 400Mhz Celeron. I wonder how would a large deployment would cope.