Well, yesterday I encountered a situation I’d not seen before. I started to receive mails with a subject line of “Subject: {Spam not delivered} {Spam not delivered} {Spam not delivered}” and included some occurrences of “{Spam not deli! vered}”. This was odd as it appeared that SpamAssassin and MailScanner had got into a nasty loop for some reason. I found it odd that the spam notification e-mails where themselves getting flagged as spam.

Upon further investigation, I found that there were some MBL rules getting triggered for every e-mail (excerpt from /var/log/mail.log):

Message XXXXXXXXXXXXXX from 123.123.123.123(user@domain.mail) to other.mail.domain is spam, SpamAssassin (not cached, score=35.688, required 5.1, BAYES_00 -1.90, MBL_330105 3.50, MBL_331475 3.50, MBL_337470 3.50, MBL_338477 3.50, MBL_338785 3.50, MBL_339415 3.50, MBL_339871 3.50, MBL_340040 3.50, MBL_345076 3.50, MBL_346112 3.50, MBL_349876 3.50, RP_MATCHES_RCVD -0.81, SPF_PASS -0.10)

Now, those SpamAssassin rules get downloaded every couple of hours from http://www.malware.com.br/ and stored in /var/lib/spamassassin/3.003002/10_MalwareBlockList.cf . It seems that the ruleset which was downloaded at 15h32 had bad entries and this mail thread seems to corroborate this: http://comments.gmane.org/gmane.comp.security.virus.clamav.user/38926. The mails stated bouncing at 17h56 after a MailScanner restart due to old age not long before then.

Now that I knew the cause of the problem – a bad set of signatures – I could go about fixing it. So, first thing was to download an updated set of signatures. The update luckily appeared to have the problematic signatures removed and I restarted MailScanner to activate them. I next started to trawl through the /var/log/mail.log log file to see what e-mail messages had been affected and hence blocked. A few greps later and I had a list of e-mails to sort out. A few more greps and awks and I had the messages and their recipients. So, I set about forwarding the quarantined messages on. The first one was one destined to myself. Curiously, this was flagged as spam again. Ahh – the MailScanner SpamAssassin cache. So, I stopped MailScanner, deleted /var/spool/MailScanner/incoming/SpamAssassin.cache.db and restarted MailScanner. I then resent the message in question which now arrived as expected. I then set about forwarding on the remaining messages.

So – what can we do about updates like this which cause false positives? Well not much it seems since we need regular automated updates to keep systems safe and secure from “bad things”. This is much like the situation with McAfee anti-virus updates which have caused systems to become unusable due to false positives on system files in the past. Unless we manually vet each update with a sample set of emails/systems before releasing them into production we are bound to have false positives every so often. Therein lies the question – which is less risk: “Having regular automated updates with some false positives or delayed/out of date updates with fewer false positives and lots of testing yet not being protected against the latest threats” – your choice…