I am not a Spammer. I am a legitimate business who wants to market my services via email. I am not someone claiming to be a prince and offering millions if you transfer your money out of the country. I’m just trying to do well for my business and communicate effectively with my customers, so why do my emails to them keep getting caught in an ISP spam-filter blackhole?
Understandably, good-intentioned email marketers around the world are often confused and angered by ISP (Internet Service Provider) spam filters for dumping their legitimate messages before they can arrive at their customer’s inbox.
To understand what's going on, we need to first examine how ISP spam filters work. ISPs, like Search Engines, aren't willing to publically divulge their strategies. It’s understandable – it’s a bit like the police giving away their radar detection technology secrets to speeders. Sure, the speeders will probably figure out a work-around eventually, but the police don't really want to help them along. But as understandable as this is, we are still frustrated by our inability to deliver legitimate email messages to waiting customers.
ISP Spam Filters
ISP email filters work on a number of levels, on both the sender’s and recipients’ end. To determine message validity and ultimately whether or not the message will be delivered, ISPs assess a variety of information from within the message itself (text in the subject line and body of the email), the domain that the message is sent from (return addresses etc…), and the list of recipients.
Blacklists: ISPs use public and private blacklists (Commonly called Realtime blacklists, DNSBL or RBL) to determine a sender's authenticity. When an ISP filter sees a blacklisted domain crossing its bridge, it puts up a wall and blocks the email from crossing. It is important to note that the “from” email address, IP addresses and even ISPs that allow spam can be blacklisted. To check to see if your organization appears on a blacklist, you can use a service such as MXToolbox.
Sender Policy Framework (SPF): SPF isn’t just a good suncare regime, it is also a framework for publishing information through DNS describing a list of IP addresses allowed to send emails from a specific domain. When an email passes through an ISP filter, the “From” field is analyzed to confirm that the SPF and the domain match, ensuring that the message is authorized from that domain. A project overview and can be found at the open SPF Project repository at openspf.org.
Recipient List: Another common method is the use of special software known as "harvesting bots" or "harvesters", which spider Web pages, postings on news groups, mailing list archives, and other online sources to obtain e-mail addresses from public data. Spammers may also use a form of dictionary attack in order to harvest e-mail addresses where valid addresses at a specific domain are added to a list by guessing them using common combinations of usernames at that domain.
For example, trying alan@example.domain, alana@example.domain, alanb@example.domain, etc, - any that are accepted for delivery by the recipient email server, instead of rejected, are added to the list of theoretically valid e-mail addresses for that domain.
Reporting Spam: In most webmail clients (Hotmail, Gmail etc.) there is an opportunity to mark a message appearing in the user's inbox as SPAM. If a message is sent to a large list of recipients, and enough recipients from that list report the message as spam, the remaining portion of the list will be blocked.
Content Filters/ Bayesian Filtering: Particular words have a higher probability of occurring in spam email rather than in legitimate email. A good example - in spam email the word Viagra will appear at a much higher frequency than in legitimate email. The filter does not know the probability of a word appearing but must first be trained so it can create relevancy. To train the filter, the user must manually indicate whether a new email is spam or not. For all words in each training email, the filter will adjust the probabilities that each word will appear in spam or legitimate email in its database. After training, the word probabilities (also known as likelihood functions) are used to compute the probability that an email with a particular set of words within it belongs to either category. More in depth information on Bayesian Filtering can be found here.
HTML Filters: ISPs keep on the lookout for poorly formatted HTML code. A HTML filter searches the body of messages for HTML tags. Spammers often use HTML tags to hide text from spam filters. Poorly formatted HTML will be flagged as Spam.
High Graphic to text ratio: Because the ISP’s use content filtering to check the validity of the message, Spammers have taken to converting all of the text to an image. The content check would not pick anything up in this case but the message would be flagged as Spam for containing a low HTML to Image ratio.
If you are in the business of sending bulk / blast emails to customers, clients or prospects; you can expect problems with deliverability; it's just the nature of email communications today. Email marketers often face Can-Spam law and ISP filters that prevent them from legitimately communicating with their waiting customers. Aside from complaining about the ways things work, it makes sense to take the necessary steps to optimize the process.
9 Ways to Improve Deliverability
-
Check blacklists: If you are experiencing frequent non-deliveries that seem unexplained because you use permission-based opt-in lists, be sure to check blacklists to make sure your domain, IP address or ISP isn't blocked.
-
Keep Opt-in records: A spam monitor, like Spamcop or Spamhaus might contact you with an alert, giving you an opportunity to show that the recipient actually did opt-in. To successfully prove your case and remove your address from that list, you must maintain and have access to your opt-in data records. It is also recommended that you provide an opt-in reminder in your email messages so that the recipient of your message sees their subscription details in the message and will be less apt to report the message as spam. A solution such as Empower can help manage both the opt-in process as well as list management!
-
Manage your lists: Reduce the amount of hard bounces by keeping your lists scrubbed clean. Remove invalid addresses quickly and ensure that list members who have opted-out are also removed promptly.
-
Check your content: There is a variety of content checking software available today that will check your message, give it a spam score and provide recommendations.
-
Design for images being turned off: any of your recipients who use AOL, Outlook 2003, Outlook Express and Gmail will never see the images in any emails you send them by default.
-
Check your HTML:
Include the<style>element in your HTML file, don’t reference it externally. Some providers (like Gmail) will strip it altogether, while others won’t use it if images are disabled. For Hotmail recipients, make sure the<style>element appears in the<body>and not the<head>.
Avoid using CSS for positioning. The support is very limited and will more than likely result in a broken layout for most of your recipients.
Don’t use Flash or JavaScript. Not for emails anyway.
Make sure you properly format your HTML and CSS – don't use WYSIWYG tools that leave extra HTML code. Leaving a tag open or overly messy HTML may result in your email being filtered as spam.
If you’re targeting Gmail, make sure you do your CSS inline, the<style>element will be removed completely. -
Provide opt-out functionality: One sure-fire way of getting your email messages reported by recipients as spam is if you don't provide an opt-out facility. But don't only provide an opt-out link; make sure that you actually remove those addresses that have chosen to opt-out.
-
Get a Dedicated IP Address: If sending bulk email is a regular part of your business, consider getting a dedicated IP address. If you currently share an IP address with other users, there is a higher chance that the IP address will be flagged by an ISP spam-filter. Make certain that your dedicated IP address comes with a reverse DNS entry setup, otherwise it may not resolve to the name that you think it will.
-
Test, Test, Test: Because ISPs are unique in their spam filtering techniques, it is important to test every message across different providers before deploying to your entire list.
Read & Learn for Yourself
Check to see if you are on a blacklist
- http://www.spamhaus.org/xbl/
- http://www.mxtoolbox.com/blacklists.aspx
- http://www.spamcop.net/bl.shtml
Subject line testers / Determine Spamminess of Message
HTML Standards
About Blake
Blake Medulan joined WebFeat as Director of Technology, bringing
16 years of experience with enterprise applications and advertising.
Blake has a unique ability of melding business requirements
with innovative technical practices – and has developed Web,
Windows and key delivery systems for clients like Microsoft,
Intel, Bell, Nokia, US Senate, the FBI and Disney. In addition
to prototyping new technologies and standards, Blake is an
effective systems trainer, industry speaker and strategist.
