Last Updated: 980419

"Sometimes I'm amazed Internet mail works at all." -djb

Qmail Message Bouncing

Introduction

Handling bounce messages can seem like a black art of mail handling. A large cause of this frustration is the large numbers of incorrectly written mail systems out there. Many of them don't follow the simple rules of message bouncing. This can cause evil things such as message loops which can eat up resources and disk space.

Standard Message Parts

There are several parts of e-mail transferring that should concern you if you are trying to detect a bounce. Let's start with the low-level components:

Envelope Sender
The envelope sender address is the address received from the MAIL FROM command during SMTP negotiation. It may be nothing or a single e-mail address. A blank sender normally indicates message coming from a mailer daemon (generally a bounce or information message).

The envelope sender is recorded into the message headers as "Return-Path:".

Envelope Recipient
The envelope recipient is one or more addresses received from the RCPT TO command during SMTP negotiation. This indicates who the final recipient(s) of the message are.

The envelope recipients are recorded into the message headers as "Delivered-To:".

Next are the headers of the message that may contain useful information:

Return-Path
This is a copy of the envelope sender. It is added to a message by qmail-local or preline and removed by qmail-inject.
Delivered-To
This is a copy of the envelope recipients. It is added to a message by qmail-local or preline.
Errors-To
This is an outdated hack. Don't use it. If you don't know what it is, don't worry. ;)

Finally, some useful information may be contained in the body of the message. We will get to this later in the document.

Point of Bounce

Where exactly a message bounces will determine what kind of bounce you can expect. Let's start at the beginning of the transfer process.

1) Injecting. If you use a separate program such as qmail-queue or qmail-inject to send a message (or /usr/lib/sendmail), then the error code returned from these programs indicate a local configuration error. This isn't really a bounce, but it can fail. If you are manually sending the mail via a socket, then you probably know what you are doing.

2) qmail-send. This is the next step responsible for sending the mail from the queue to its destination. It uses other programs for the responsibility of actually sending a message. Based on the return of these programs, it determines whether or not the message failed. If it failed, then a bounce is returned in QSBMF to the envelope sender. The envelope sender of the bounce message will be set to <>. Note that the mail may have successfully been sent but still may fail later. If the original message's envelope sender is an invalid address, then this will create a double bounce (which is discussed later in this document).

3) Remote mail system. Remote mail systems may respond with bounce messages in a wide variety of ways. They may set the envelope sender of the bounce to something like "postmaster" or "mailer-daemon" or <> like qmail does. Under normal circumstances, the remote system will respond with a bounce message to the envelope sender of the original message. However, some systems will (incorrectly) respond to one of the addresses listed in the headers of the document. This could be "From:", "Reply-To:", "Sender:", or one of the many other ways of indicating the sender of the message.

QSBMF

qmail-send Bounce Message Format is a format of a bounce message used by the qmail package. It is simple to parse and straight forward. More information about it is available in the file RFCQSBMF in the qmail package. There are perhaps hundreds of other bounce message formats out there. Trying to parse every kind is virtually impossible. The best bet is to check for some common "Subject:" headers, "From:" headers, and body contents. Here are some examples from the ezmlm package:

Subject: success notice
Subject: deferral notice
Subject: Warning: could not send message
Subject: Returned mail: warning: cannot send message
From: Mail Delivery Subsystem <MAILER-DAEMON@
Auto-Submitted: auto-generated (warning

Looking at my collection of thousands of bounce messages that I have collected, I see several hundred different "Subject:" headings and body contents that can be parsed if you try. However, this path is not recommended. The small number that ezmlm checks for will only catch a small percentage of bounces. It's best to stay in the realm of envelope addresses and check their values. The content of the message is unreliable and not worth parsing.

Bounce Error Codes

Qmail uses hash code error numbers defined in RFC 1893. These are a convenient method of determining the actual cause of an error. Unfortunately, it can not be easily parsed out of a message, and is only useful for determining a problem by a human.

VERPs

A mechanism introduced in qmail is Variable Envelope Return Paths. This allows one to easily control the envelope sender mainly for a mailing list. This magic is done by re-writing the envelope sender in the following fashion:

pre@host-@[]

Will be rewritten to:

prerecip=domain@host

For a delivery to recip@domain.

Thus, if a message bounces, it will be returned to prerecip=domain@host in which case it is simple to extract the recip=domain part.

Note that this is not the silver-bullet solution it may seem to be. Several mail systems will truncate the local part of the address at some arbitrary length, which can effectively nullify all your efforts to track bounce messages. Another problem is that some systems may not like the equals sign in the local part of the address (although a perfectly legal character per RFC 822, it appears that not everyone read that document).

A final note is that if the bounce occurs in qmail-send, then the bounce will be sent to pre@host without the extended information. You will need to parse the QSBMF part of the file to extract any useful information.

AutoResponders

An auto-responder is similar to a mailing list, but it simply responds with a pre-defined message to the person who sent the mail. The most common program for this purpose is vacation. However, this program may not be sufficient for most purposes. I will begin with an annoying trait of system e-mail configuration. Many people in their attempts to block unwanted mail have implemented routines that analyze the envelope sender of a message. For instance, if the sender's domain has no entry, then it will not accept the message. Another thing they will try to do is block all mail from <>. This is a foolhardy thing to do. Not only will user's not be able to get bounce messages, it makes it more difficult to send auto-responses. The next step would be to create a bit-bucket address for the envelope sender and do some trickery with the "From:" header. In my situation, I would like "From:" to point to the autoresponder that sent the response. If you rely on this situation, you can potentially create a mail loop because some (albeit ignorant) mail systems will respond to the "From:" header and not the envelope sender. The only sure-fire solution is to set the "From:" header and the envelope sender the same as the autoresponder, and then check for mail received from the same address over n times over some period of time. You can check for certain addresses, such as <>, #@[], postmaster@, mailer-daemon@, etc. to crack down on some of the more worthless mail you don't want to autorespond to.

Double Bounces

A double bounce is created when a bounce bounces on its way back. This normally indicates a configuration error (you should always have your envelope sender set up as a valid address, or <> or #@[]). If a bounce bounces and the envelope sender is <> or #@[] then it is ignored. But if it is an address, then the bounce will be sent to that address. If the address fails, then the double bounce situation occurs. Qmail will then send a QSBMF message to the doublebounce address (by default this is postmaster). Note that SMTP prevents envelope senders to be #@[], and thus it is only used in the internals of a qmail system.

Bounce Handling in Qmail Configuration

Qmail has several configuration items that can help you control bounce messages. Here are several for the control directory.

Some further things you can do to control the bounce behavior:

To have an address automatically bounce a message, put the following into a .qmail file:

|exit 100

Read the qmail-command man page for other exit codes.

If a message is sitting in the queue, and you would like it to bounce right now, touch the appropriate info file with a time sufficiently in the past (beyond queuelifetime) and give qmail-send an ALRM signal. Note that qmail-send will try to send it one more time before it will bounce it. Is there a better way to do this?



Written by Eric Huss

Comments welcome.