Spam Countermeasures
This article describes ways in which Mail Transfer Agents (MTA’s) can be used to detect, deflect and help protect users from the various types of spam commonly encountered. This is not a general tutorial on spam as it is assumed the reader has this background. For a primer on spam please check out our What Is Spam article on this site.
How to Block?
Effective blocking of spam is unfortunately not as easy as it might seem. The problem is not so much how to block the bad stuff from coming in, but how to do it while AT THE SAME TIME not affecting legitimate messages flowing in and out. The blocking of good messages by mistake (also known as false positives) is considered much worse than letting bad messages through. Many techniques have been tried with varying degrees of success with no single approach being completely effective by themselves. A few of these are discussed below.
User Filters
Bayesian spam filtering techniques were used in many early email server solutions attempting to block spam. These and other similar types of filtering techniques are available in Email User Agents (programs that users use to send and receive email) such as Thunderbird and some webmail systems. Through continued use and the classification of messages as spam the mail program “learns” or gets a better idea of what patterns in a mail message represent spam for a given user. The system can then automatically take action on these messages on behalf of the user. The accuracy of these systems vary considerably so usage is not as common as the designers would have liked. Only the more technically sophisticated users tend to use these features which also limits their usefulness. Bayesian filters are not used on any Jade mail servers.
Content Filters (Server Side)
Email messages can be analyzed and acted upon as they are received by the inbound Message Transfer Agent (MTA). One of the most common server-side content analysis engines is SpamAssassin. This open source solution takes a message and optionally runs several different filters against it to determine a spam score. Some of the many available plug-ins to SpamAssassin include hooks for the Distributed Checksum Clearinghouse (DCC), Pyzor, and Razor. There are many more filters (including server-side bayesian and phishing filters) that can be plugged into SpamAssassin as documented on the project website.
We have found that there is a large variance in accuracy of the different network based content filters with many returning unacceptably high false positive rates. While we use SpamAssassin internally we are very cautious about integrating additional content filters as in some situations they have proven to block too many good messages along with the bad. The Razor 2 content analysis engine as been incorporated into Jade MTA’s since August, 2015. From the Razor home page:
Razor is a distributed, collaborative, spam detection and filtering network. Through user contribution, Razor establishes a distributed and constantly updating catalogue of spam in propagation that is consulted by email clients to filter out known spam. Detection is done with statistical and randomized signatures that efficiently spot mutating spam content. User input is validated through reputation assignments based on consensus on report and revoke assertions which in turn is used for computing confidence values associated with individual signatures.
Virus / Malware Scanning
Many viruses and other malware (worms, trojan horses, etc) initially make their way onto a victim’s computer or network as an email attachment. The mail system together with improved end user education on basic security form the front lines of defense in this war. Jade MTA’s scan all incoming messages for malware before accepting mail for delivery. The scanning does not catch all malware but is a good first line of defense. It is highly recommended that users also scan and take extreme care when opening or running programs sent as attachments.
Several well known file extensions have a high correlation with malware when found as message attachments. Messages with attachments of these types are refused when submitted to our systems. The current list of banned file extensions are: exe, vbs, pif, scr, bat, cmd, com, cpl, and exe-ms. If legit senders need to send these kinds of files it is recommended that they be encapsulated in a common archive format such as tar, zip or rar.
Greylisting
Standard email systems queue received messages and if temporary problems are encountered will retry sending at a later time. Most spammers use a one-shot approach to sending spam. The queuing and retry overhead, both in terms of processing and local storage, is more than most spammers can justify (or want to pay for). They usually attempt to send messages just once and then discard regardless if it was received for not.
With this in mind many Internet mail servers (MTA’s) keep track of the message senders and recipients of messages that flow through their systems. When a new message arrives if there is a record of the external sender and local recipient address pair the message is immediately accepted (pending other security checks). If not (which is usually the case for spam messages) the MTA issues a temporary unavailable return code to the sending system. This signals the foreign system that the email cannot be immediately accepted due to a local problem but to try again later.
Normal mail systems will keep the message and retry later during their next queue run (during which time the message is usually accepted). The Jade configuration uses a one minute delay during which time new message delivery attempts will continue to be deferred. After this delay any new attempt to send email using this sender/recipient pair will succeed with the new sender/recipient pair information stored in a local database. Further communication using these addresses will pass through the system with no delays.
The first time a new sender tries to communicate normally results in a short initial delay. Most all non-spam messages do get retried and delivered within a short period. Our experience with greylisting is that this simple mechanism alone accounts for an over 80% spam reduction through our MTA’s with no message loss from legitimate senders.
Recipient Address Validation
Many older email systems blindly accept all messages that are submitted and try to deliver them the best they can after accepting them. This has proven to be a bad model as spammers usually forge the message envelope sender address to something that either cannot be replied to or that points to an innocent recipient address. When the later happens a non-delivery notification is sent to the forged sender address. This is known as backscatter (see the Wikipedia backscatter article for an in-depth discussion of this). Backscatter messages are considered by many as another form of spam and can get the receiving MTA blacklisted as well.
Backed up non-delivery notification messages can cause many additional problems for receiving MTAs. If too many messages get backed up mail system performance is affected. The solution these problems is to verify all recipient addresses during the initial SMTP transaction (SMTP RCPT TO command). Any submission attempts to improper recipient addresses then get shut down at the SMTP level before they are ever accepted in the first place. Most modern email systems take this approach as do the Jade Networks MTA’s.
Another tactic that spammers use is dictionary lookups to discover valid addresses. These discovered addresses are later used when sending spam and in the generation of spam address lists. It is not uncommon for spammers to simply walk through names using the SMTP RCPT TO until they find a valid account name. Older MTA’s will blindly reply with a 200 OK return for all and worry about the consequences later. MTA’s that perform address validation during the SMTP transaction will let the remote MTA know right away if an address is valid or not.
There is a serious security problem with providing criminals immediate feedback in their search for valid users. Unless throttled the criminal can figure out quickly who most of your users are using this technique, even if they never send email. One defence against this type of an attack is to keep track of the failed attempts and to disconnect the remote MTA after an error threshold has been breached. Jade Networks is configured with a threshold of 2 so a single failed attempt is accepted but not multiple. Technically this is a violation of the SMTP specifications however in the many years we have used this across many sites we have yet to encounter any message loss as a result. This approach effectively shuts down dictionary attacks at the SMTP level.
Sender Policy Framework (SPF)
The Simple Mail Transfer Protocol (SMTP) is used to transfer email messages between systems. SMTP is in fact so “simple” that without proper safeguards in place it is trivial to forge sender addresses. Criminals use this technique to make email appear to come from someone else – most often an innocent unsuspecting person with no relationship to the fraudulent message being sent. The criminal uses this masquerade to try to hide their criminal actions behind innocent people.
One solution to this problem in widespread use across the Internet is the Sender Policy Framework (SPF). SPF specifies an extension to the Domain Name System (DNS) where additional Resource Records (RR‘s) are defined. These new RR’s associate authorized message originating MTA’s with each domain. When email is received, the receiving MTA looks at the SMTP MAIL FROM: value to determine the sender domain. A DNS check is then made for this domain to see if SPF records exist. If they do this information is compared against the value of the connecting IP address of the sending MTA. If the connecting IP address is associated with a valid SPF record the message is accepted. If not the message is rejected during the SMTP session with a SPF violation error.
SPF checks are very effective in fighting the reception of email with forged envelope sender addresses. Jade configures SPF in all Jade DNS nameservers for all internal and hosted domains. SPF is also used for all inbound messages to its MTA’s. Jade SPF DNS records for customer domains are initially configured to list the appropriate Jade MTA’s. For most customers this works fine as long as email is sent through the Jade network. If there is a need for a customer to do message submission through other MTA’s please let us know so we can adjust the SPF records accordingly. Use of public MTA’s is highly discouraged however as anyone else with access to these submission points will also have the ability to send masqueraded messages appearing to come from the customer domain.
For additional information on SPF there is an excellent article with extensive references on the WikiPedia site.
DNS Blacklists (RBL)
Spammers rarely just send out a few messages but rather try to spew as many as they can to get their message out. It’s a numbers game – the more rubbish they spew the higher the chance that they will get a bite somewhere even if the percentages are low. Network operators around the world try to keep track of where spam messages originate. While many are propagated through BOTNET’s using hijacked computing resources, many are also sent through conventional channels. DNS Blacklists, also known as DNS Blackhole Lists (DNSBL’s) and Real-time Blackhole Lists (RBL’s) are databases of known IP addresses associated with spam senders. These databases are stored using the DNS and can be queried by MTA’s (using the sender’s IP address) to determine if the remote sender is listed or not.
Many RBL’s exist but there are large differences between them. The listing (and de-listing) criteria between different list providers can be significant. The more aggressive providers will easily add addresses to their lists with false-positive rates typically being high. Overall accuracy rates can be problematic if too many good messages get rejected along with the bad.
At Jade we’ve run tests with various RBL’s configured with our mail servers. Most have, for our customers at least, resulted in unacceptably high false-positive rates. The loss of messages critical to business can cost companies far more than a few extra obvious spam messages to delete. At the time of this writing the only outside RBL we use is zen.spamhaus.org. This RBL has very low false-positive rates while still catching most spam getting through our other countermeasures (see above). We also use the Jade RBL (bl.jade.net) which we maintain based on spam that makes it through our other defenses. The Jade RBL is public however it is not setup like the the mainstream RBL’s in that no formal mechanisms exist for checking for listings or for de-listing. The contents of this RBL relate to confirmed spam received locally.
Non-SMTP Access Controls
Spam and other fraudulent messages that get through the above controls and are detected are manually transferred to a special spam handling account. An automated system then retrieves them and extracts the information about the connecting host. An abuse report message is constructed with this information and other diagnostic information and sent to the Jade Abuse Report account and to the abuse contact for the site which submitted the message to the Jade MTA (if one was found in the remote whois data). The abuse report is sent using the Internet standard ARF (Abuse Report Format) format (see Abuse Report Reporting Standards for links to the technical standards). The text portion of the abuse report contains a brief description of the report, the original message headers and the ARF (message header summary section) and WHOIS data. The original message is attached to this administrative message intact without any modification. Once received by the Jade Abuse Report account the message is reviewed for a second time. If it is determined that the original message was not spam the block is removed and remote admin notified. Domains who do not have abuse contact information published get silently blocked with no notification. The double human verification of spam messages was designed to reduce the possibility of improperly blocking innnocent sites. So far there have been no false positives with this approach.
Messages processed by our internal spam handler result in the spammer connecting IP address automatically being entered into our DNSBL (bl.jade.net). Should the secondary review determine that the message was inappropriately submitted (not spam) the block is later manually removed and the remote admin notified.
Jade Networks Email Configuration
Jade Networks uses all of the above mechanisms in the fight against spam. Of the above countermeasures greylisting and RBLs catche over an estimated 90% of the received spam. SPF checks catch some as well as protecting against forged or masqueraded messages. As long as we value legit email over spam some just will always get through. As we continue to tune our systems and add additional protections the percentage of just that gets past our defenses continues to decline.
Current Development Work
We are always looking for ways to improve the Jade systems. Work is ongoing on a new Jade security backend (partially completed and in operation) where security threats can be dealt with faster and integrated into new automation. The Jade SpamTools were upgraded to be fully ARF compliant in September 2015. The system was changed to allow automatic submission to upstream MTA’s when valid abuse reporting addresses were found in the submitting MTA’s whois record. This combined with enhanced reporting and ARF standard compliance has so far resulted in better and faster responses from some of the larger providers while at the same time drastically reducing the overhead of manually dealing with individual spam messages.
Other work we are currently doing is in the configuration and deployment of DNSSEC and the secure versions of SMTP, IMAP and HTTP. After this is complete we plan to test and deploy a Domain Keys (DKIM) solution (see DKIM Standards for the technical details). This will compliment our current SPF authentication tools. After DKIM integration we will be evaluating the newer DMARC recommendations for possible integration into the Jade mail infrastructure.
See Also