In the mid-1990s, two software engineers came up with an innovative solution to deal with unsolicited and unwanted productivity-draining emails and developed the first rudimentary form of the anti-spam software that is so prevalent today.
They created a list of IP addresses that were being used to send huge quantities of unsolicited emails and started blocking messages sent from those IPs. The system was called the Mail Abuse Prevention System (MAPS) – SPAM backwards. That system was the basis for the Domain Name Server Blackhole List or Real-Rime Blackhole List (RBL) which is still in use today.
A blacklist of IP addresses forms the basis of most modern day anti-spam software solutions. When IP addresses are determined to be used for sending bulk unsolicited emails or distributing malware they are added to the RBL and messages are blocked.
While an RBL is useful for blocking bulk email campaigns, spamming has become far more sophisticated. Botnets now extensively used to send spam emails from a myriad of IP addresses in much smaller volumes which are harder to detect as spam. New IPs used faster than it is possible to add them to an RBL.
Today, spam filters need to include a much wider range of mechanisms to detect spam messages and malicious emails. By incorporating a range of spam detection techniques it is possible to increase the block rate from 60% to more than 99.9%.
Why Anti-Spam Software Now Essential
There has been considerable research into the negative effects from email spam over the years. Some recent estimates suggest an average employee receives around 100 messages a day and spends around 2.8 minutes a day just dealing with spam messages. That doesn’t sound like a major loss, but for a business with 1,000 employees, that adds up to more than 11,000 hours lost each year – Spam is a major drain on productivity. While it is unlikely that all those hours will be recovered, blocking spam will certainly result in productivity gains.
A much more important reason for using anti-spam software is to block malicious emails, such as those used to ‘phish’ for sensitive information or deliver ransomware and other forms of malware. A malware infection or phishing attack can prove incredibly costly. Data breaches often cost millions of dollars to resolve, with the Ponemon Institute’s Annual Cost of a Data Breach study suggesting the average cost of resolving a breach has now reached $3.6 million.
Email is the primary vector used to conduct phishing campaigns and spread malware and the volume of malicious messages now being sent has increased considerably in recent years. Without an effective spam filtering solution, businesses will be extremely vulnerable to attack.
Anti-spam software comes at a cost, but nowhere near as much as the cost of a data breach. In fact, the productivity gains alone would more than cover the cost of anti-spam software.
Methods Used by Anti-Spam Software to Detect Spam and Malicious Emails
Each anti-spam solution functions slightly differently, although at the heart of each solution you will usually find the following four controls:
Real-Time Blackhole Lists
When an IP address or domain is discovered to be sending large volumes of unsolicited emails it is added to a real-time blackhole list. When a message is received from an IP address on the list, instead of being delivered to an inbox, the message is directed to a junk or spam folder. There have been significant advances which have improved the effectiveness of RBLs. Instead of a black and white approach – spam or not spam – an IP reputation score is applied to IP addresses. Factors that affect the IP reputation score include open rates, click-through rates, hard bounces – when messages are rejected as they no longer exist, and spam complaints. Businesses can choose the level of aggression of their spam filters by setting a threshold for IP reputation score.
Sender Policy Framework
The Sender Policy Framework (SPF) is used to detect and reject emails that claim to come from one domain but are sent from another. SPF checks whether inbound messages have been sent from a host authorized by the administrators of a domain, which helps to prevent email spoofing – A common tactic used in phishing campaigns.
Recipient Verification
Recipient verification is a method used to block messages that have been sent to valid domain, but an incorrect email address. Rather than simply rejecting these messages – which could include emails from prospective customers – the emails are directed to a spam or quarantine folder.
Content Analysis
Since spam campaigns have become much more sophisticated and IP addresses are now being used to send much smaller quantities of emails – well under the threshold for triggering inclusion into an RBL – content analysis is required. Most anti-spam software solutions analyze message headers and email content looking for common signatures of spam. Machine learning improves the system over time based on user actions – A technique known as Bayesian analysis.
Spam Detection Methods Used by Advanced Anti-Spam Software
Spam filtering solutions combine the above techniques and usually allow users to set their level of tolerance by setting an appropriate threshold. If a spam score – a combination of content analysis and IP reputation – exceeds a certain level, messages will be quarantined. Companies can choose a permissive level, which will see some spam emails delivered, or a more aggressive level that will see more spam messages blocked but some genuine emails misclassified and sent to the spam folder.
The above techniques will typically block between 97% and 99% of spam and malicious emails, without having unacceptably high false positive rates. However, if a user receives 100 emails a day, and a company has 1,000 employees, even a 99% block rate will see 1,000 spam or malicious messages delivered each day.
Advanced spam solutions incorporate additional controls to improve detection rates and will typically block more than 99.9% of spam and malicious messages. Three of these advanced techniques, typically not included in standard anti-spam software, are:
Greylisting
One problem with anti-spam software that relies on blacklists is difficulty blocking spam messages from new email servers and IP addresses that have not previously been used for spamming. Spammers constantly change IP addresses to fool RBL-based filters. Greylisting is a useful technique for assessing whether a message is genuine when it has been sent from an IP address not included in an RBL. Greylisting is used to check the validity of messages sent from IP addresses with questionable reputation scores. Messages are rejected and a request is made for the message to be resent. Since spammers’ servers are busy with large-scale mailing campaigns, these requests are often ignored. The delay in response is a good indication of the genuineness of the message.
SURBL/URIBL filtering
Many phishing emails include a URL which, if clicked, will direct the recipient to a phishing site or webpage where malware is downloaded. SURBLs are lists of webpages that have been previously included in unsolicited messages and are a good indicator that a message is spam. By using SURBLs it is possible to block around 75% of spam messages that would not be blocked by RBL-based controls. URIBL filtering checks the domain of a hyperlink against a list of blacklisted URLs in real-time – domains and webpages that have been reported as malicious.
SMTP controls
The purpose of SMTP controls is to authenticate the source of emails and their configurations. These controls block emails sent from unqualified domains such as those without a DNS A or MX record and those with non-fully qualified MAIL FROM commands.
The Importance of Outbound Email Scanning
Most companies are primarily concerned with preventing spam and malicious emails from being delivered to their employees’ inboxes, although it is also important to scan outbound messages. This is an area often overlooked by businesses.
Anti-spam software blocks IP addresses that are used for spamming. If a business has its own IP addresses blocked, important outbound business emails will not be delivered. Outbound email scanning allows a business to take control of its own email accounts and prevent employees from deliberately or inadvertently sending messages that could affect an organization’s IP reputation score. It also alerts a company when email accounts have been compromised and are being used for spamming or malware distribution – due to botnet infections for example.
How to Choose Anti-Spam Software
Companies that want the greatest level of protection from email-based threats may look to the most expensive anti-spam software solutions, although price is not the best gauge of effectiveness and suitability. There are often cheaper solutions that are better suited to the needs of a business, while providing excellent protection from email-based threats and productivity-draining spam.
When considering anti-spam software, be sure to take the following into account:
- There are often hidden costs with spam filtering solutions, such as the time required to manage the solution – While anti-spam software may be effective, a high management overhead will add significant costs over time. Use a trial to assess the time it will take to maintain your spam filter.
- Many solutions appear ideal at the start but are inflexible – When the needs of a business change, the solution should change too. You do not want to be locked into an inflexible contract that will not reduce in price if you cut the number of email users.
- Some solutions, especially appliance-based filtering solutions, lack scalability. Expansion of the business may require the purchase of additional – and often expensive – hardware.
- Many anti-spam solutions charge extra for outbound filtering and other advanced controls – These can add significantly to the cost so make sure these costs are factored in when considering a spam filtering solution.
- Advanced spam filtering solutions incorporate greylisting, SURBL/URIBL filtering, SMTP controls, more than one AV engine, and outbound email filtering – These controls will block a higher percentage of spam and malicious messages.
- Look for a solution that has a high detection rate and low false positive rate – When choosing a solution, try to independently verify the detection/false positive rate.
- Consider a cloud-based spam filtering solution – cloud-based anti-spam software requires no hardware purchases, is more flexible, and has greater scalability.
- Take advantage of a free trial to find out how simple a solution is to use and how effective it is at blocking spam before committing to a lengthy contract or purchasing an expensive appliance.