MRPostfixBounce
Email bounce processing sucks. I realized that after I had to deal with it, but after talking to two different companies in the same week that were having gripes, I realized that everyone hated it.
There a few popular ways to handle bounced emails:
- Using IMAP to retrieve all emails that have gone to no-reply@example.com. This is the easiest way and is robust in the sense that any email that returns to no-reply is one that will need to be bounced. The drawbacks are that you have to deal with the actual inbox and making sure it stays small enough, not to mention that you’re limited by your IMAP API.
- Using commercial software. A quick Google turns up that many of these options exist, however I can’t say I ever investigated any of them. There’s probably a lot of good off the shelf solutions, but the drawbacks are going to be cost.
- Processing your own Postfix logs to look for permanent failures (hard bounces). The drawbacks are that mail server response messages are not standardized (there is an RFC but it gives a decent bit of leeway with how the message ought to be formed)
Clearly the latter options seems like the most interesting. Let’s look at three bounce messages from Major SMTP Players:
Jan 12 00:00:05 smtp4 postfix-lowpriority/smtp[20885]: A235210002159: to=<example@gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.155.27]:25, conn_use=17, delay=9.2, delays=3.7/0.04/5.1/0.35, dsn=5.1.1, status=bounced (host gmail-smtp-in.l.google.com[74.125.155.27] said: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient’s email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 http://mail.google.com/support/bin/answer.py?answer=6596 y17si965722icb.154 (in reply to RCPT TO command))
Jan 12 00:00:07 smtp5 postfix-lowpriority/smtp[18609]: D6B5A80015CC: to=<example@earthlink.net>, relay=mx4.earthlink.net[209.86.93.229]:25, delay=1, delays=0.49/0/0.38/0.18, dsn=5.0.0, status=bounced (host mx4.earthlink.net[209.86.93.229] said: 550 example@earthlink.net…User unknown (in reply to RCPT TO command))
Jan 12 00:00:46 smtp5 postfix-lowpriority/smtp[18663]: 3B5AC800054C: to=<example@hotmail.com>, relay=mx3.hotmail.com[65.55.92.184]:25, conn_use=27, delay=0.35, delays=0.23/0/0.04/0.08, dsn=5.0.0, status=bounced (host mx3.hotmail.com[65.55.92.184] said: 550 Requested action not taken: mailbox unavailable (in reply to RCPT TO command))
As you can imagine, most SMTP servers have their own subtle variation on these messages. There’s two broad cases to take care of: our Postfix server is unable to even connect to the remote server (common in the typo cases of hotmai.com and gmaill.com) and then per-domain specific rules for handling.
I’ve recently committed mr _postfix_bounce to Yelp’s mrjob for processing Postfix log files. It’s pretty straightforward but has some nice features:
- Custom processing rules are loaded in as a JSON file. This means that hopefully this file can become more robust as cases are added for specific domain providers. The file is simply a list of (list of domains, match criteria) as shown:
[[“yahoo.”, “ymail.”],
{“remote_smtp_string”: [“This account has been disabled or discontinued [#102]”, “554 delivery error: dd This user doesn’t have a yahoo.com account”]}]
- Because it’s a mrjob it can run on EC2. This is a huge win if you have gigabytes of mail log processing to do every day. Even if you don’t, as long as you have mrjob installed, you can run it locally over your log files.
The rules list at this point is no where near comprehensive, but focused on the domains that we had the bulk of our bounces from. I encourage everyone to fork and create pull requests as they add new domains to the JSON rules file.
