Subject: [ietf-dkim] NEW ISSUE: NAKED CR & LF issues with body canonicalization

From: Michael Thomas <[email-masked]>
Date: Wed, 12 Jul 2006 08:31:08 -0700

In our testing at Cisco, we are seeing a small but
significant number of failure mainly due to various system
bots that send naked CR's in a message. What I have found
is that at the very least, sendmail and Ironport handle
these two cases differently. Thus for the input:

testing 1\n
\r
\r
2 3\r
\r\n
.\r\n

sendmail yields:

testing 1\r\n
\r
\r
2 3\r
\r\n
.\r\n

Ironport yields:

testing 1\r\n
\r\n
\r\n
2 3\r\n
\r\n
.\r\n

Ie, both Ironport and Sendmail are doing CR stuffing for
naked LF's. However, for Ironport it's always doing LF
stuffing for naked CR's in the input stream. I suspect
that sendmail's behavior is right, but that's not really
what is at issue here.

First my suspicion is that this kind of confusion is not
uncommon in the wild. We've already found out that it's
pretty common, especially with mailbot's in our own
organization. Nor do I doubt that Ironport is an outlyer
here -- I'd guess that this is a common transformation.

Here's my suggestion: we fix this by modifying the relaxed
body canonicalization such that any lone CR or LF is
*always* canonicalized as CRLF.

Please, please -- if people know about other wrinkles to
this let's get them out in the open now. As it stands,
simple body canonicalization fails some pretty common cases.

	    Mike
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: John Levine <[email-masked]>
Date: 17 Jul 2006 00:33:33 -0000

>In our testing at Cisco, we are seeing a small but significant number
>of failure mainly due to various system bots that send naked CR's in
>a message.

Yeah, there are a lot of badly written MUAs.

> What I have found is that at the very least, sendmail and Ironport
>handle these two cases differently.

Right.  Different MTAs do different, fairly random things to mail that
isn't 2822 compliant.  (They do random things to mail that is
compliant, but their mutations to non-compliant mail are more
extreme.)  Trying to guess what's going to happen and to try to
program around it is a guaranteed exercise in frustration.

My strong suggestion is to say that if you want your DKIM signatures
to interoperate, you should only sign compliant mail.  If someone or
something injects a non-compliant message to your MTA, fix it before
signing it.  It's hard for me to envision a situation where this isn't
the right thing to do.

This is what my qmail setup does now -- locally injected mail has
formatting errors corrected and missing headers added before it's
passed along, while incoming MX mail is just passed through except if
it has bare linefeeds in which case it's rejected.

There are a whole lot of ways to construct a not-quite-2822 message
other than bare carriage returns, and I see no reason to try to revisit
this well-trodden territory.

R's,
John

---

From: Michael Thomas <[email-masked]>
Date: Sun, 16 Jul 2006 21:16:31 -0700

>program around it is a guaranteed exercise in frustration.
In the months since we went live with probably hundreds of millions
of messages passing through our signers/verifiers, this is the only thing
that I've seen with any consistency that breaks the body with simple.
You keep raising the spectre of slippery slopes, but in fact the millions
and zillions of ways that things could be broken doesn't seem to be borne
out in reality, at least in our experience.

>to interoperate, you should only sign compliant mail. 
>
That's completely unhelpful.

> If someone or
I guess you could take that up with Eric, and the half of the email 
infrastructure
 that runs his software.

       Mike

---

From: John L <[email-masked]>
Date: Mon, 17 Jul 2006 00:26:47 -0400 (EDT)

> that I've seen with any consistency that breaks the body with simple.

Right.  So fix the message, then sign it, thereby making the message and 
signature robust regardless of what MTAs it may later pass through.  Is 
there some reason this wouldn't work?  Does something downstream depend on 
getting naked CRs in the message?

> That's completely unhelpful.

Just in case you missed it the last three times I said this: make the 
message compliant, then sign it.

I suppose I can sort of imagine hypothetical situations where two hosts 
would be passing messages back and forth that require naked CR or LF, but 
that's a private network, not SMTP.  I am utterly unable to imagine why an 
IETF standard should require DKIM to handle such messages when we all know 
that the only reason they happen is software bugs, and it's already common 
practice to fix them up at a relay.

R's,
John

---

From: Michael Thomas <[email-masked]>
Date: Sun, 16 Jul 2006 21:35:22 -0700

> message compliant, then sign it.

I guess you missed where I said that you'd be better taking that up with 
Eric.

>
> I suppose I can sort of imagine hypothetical situations where two 
> hosts would be passing messages back and forth that require naked CR 
> or LF, but that's a private network, not SMTP.  I am utterly unable to 
> imagine why an IETF standard should require DKIM to handle such 
> messages when we all know that the only reason they happen is software 
> bugs, and it's already common practice to fix them up at a relay.

Incorrect. They could be applications that were built well before the 
issuing of RFC 2822,
were following STD11 and thus completely in spec. It's hardly asking 
very much to
expect IETF specs to be liberal in what they receive when it comes to 
previous full standards
with 25 years of legacy.

       Mike

---

From: John L <[email-masked]>
Date: Mon, 17 Jul 2006 00:54:22 -0400 (EDT)

> issuing of RFC 2822, were following STD11 and thus completely in spec. 
> It's hardly asking very much to expect IETF specs to be liberal in what 
> they receive when it comes to previous full standards with 25 years of 
> legacy.

I have never come across an application that depended on naked CR or LF 
being passed through SMTP.  I've seen lots and lots of of naked CR and LF 
(particularly since I've been using an MTA that spits them back since 
about 1999) and without exception they are due to bugs.  If anyone is 
aware of deliberate naked CR or LF, I would sure like to hear about it, 
and how they deal with the large fraction of mail relays that silently 
discard them or stuff the missing LF or CR.

What's the benefit of being backward compatible with buggy behavior that 
nobody depends on?

R's,
John

---

From: Mark Delany <[email-masked]>
Date: Mon, 17 Jul 2006 06:21:49 +0000

> I guess you missed where I said that you'd be better taking that up with 

Strange. You must have missed where I suggested you take it up with
Eric. That suggestion being fix the problem at the layer creating the
problem. If your SMTP listener is presenting you with mal-formed mail,
then that's a contractual matter between you and your SMTP listener to
resolve.

You and it should agree on what to do with malformed mail prior to
bothering with verifying or whatever else you do.

Ditto your submitter. If it presents you with non-compliant mail, make
it compliant prior to signing. Conflating SMTP or SUBMIT bugs into
another protocol is entirely unnecessary when you can fix it much more
cleanly at the boundary.

All I hear you saying is that such boundary fixes should be uniform as
far as is possible. That sounds fair.

Mark.

---

From: Mark Delany <[email-masked]>
Date: Mon, 17 Jul 2006 06:23:53 +0000

> that I've seen with any consistency that breaks the body with simple.

How many of those were signed though? And what was the intent of the
signer wrt these mail-formed mails? And hundreds of millions over many
months sounds like a tiny sample.

Mark.

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 17 Jul 2006 02:40:44 -0400

----- Original Message -----
From: "John L" <[email-masked]>
To: "Michael Thomas" <[email-masked]>

> being passed through SMTP.

Is QMAIL strickly a SMTP to SMTP transport system?

Does QMAIL only accept feeds from SMTP or does it support the eyeballing of
queued folders and feeds from integrated or external packages or processors?

If so, what formats can it accept?

Did QMAIL evolved from UUCP/SLIP processing with support for unix flavor LF
eol storage formats? or MAC based feeds with its CR eol storage formats?

I saw you wrote QMAIL will reject LF eol SMTP feeds.  I was surprise to read
this.  Sounds odd.  Is that an follow strict 2822 option or something in
qmail?

Our Wildcat! List Server has the option to create its distribution and
tranport it via SMTP or UUCP.  Many of our operators will choose the UUCP
method because they want to isolate their LS machine to non-SMTP duties.

In SMTP mode, it sends directly to some defined SMTP host(s). In UUCP mode,
it creates and queues standard *.DAT/*.XQT/*.CMD files with the expectation
that some other processor will pick up this unix flavor storaged files and
do something with them.

In the old days, that would been some UUCP/SLIP gate.

Today, it could be a SMTP router/server eyeballing some outgoing queue feed.
An SMTP server with such long term support, will handle all types of feeds,
wither it was comeing directly from a SMTP client or queues for routing from
another source.

Similar, our gate software works with many formats and also centralizes
itself with standard UUCP style files between the online HOST and the SMTP
server.

However, our SMTP server will ALWAY send out conpliant CRLF lines.  On the
SMTP receive side, it will follow RFC 2821 requirements for:

   - Transformation of initial double dots to single dots,
   - Check for the DATA delimiter <CRLF>.<CRLF>

and everything else is received as is and typically left alone because of a
historical and common mail designer ethical tradition to avoid the design
taboo of "thou shall not screw around with mail content."

Anyway,

-1 for any idea that touches base with the score + plus years and old
tradition of not tampering with mail content - not even for DKIM.

I believe the better solution is Michael's suggestion of just dealing with
the DKIM feed.  No need to alter content permanently unless the WORLD
accepts and adopts this fundamental change in mail design mindset.

PS: No need to tell me that software already change content. To me, they are
the ones that partly responsible for the mess and disrespect in mail
intrigity issues.   The problem isn't really SMTP or the mail system, it is
DKIM, it is the one that needs to fit with the system, not the other way
around.  Thats the pragmatic and conservative approach to maximimize its
adoption and minimize acceptance barriers.

---
HLS

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 17 Jul 2006 04:12:28 -0400

----- Original Message -----

> If you're wondering what qmail does, feel free to drop by www.qmail.org
> and find out for yourself.

Sure thing!  I see it is very oddly and uncommonly very strict in its
non-support for SMTP LF feeds.  Odd, but I stand corrected by your original
statement.

This pages clearly describes its myth breaking excuse for STRICT RFC 2822
compliance:

http://homepages.tesco.net/J.deBoynePollard/FGA/qmail-myths-dispelled.html

Yet, ironically, I see it isn't so anal by allowing for the incorrect space
in the MAIL FROM command:

           MAIL FROM:sp<address>  <---- invalid format

I take it iecc.com is running qmail software?

helo metro
250-tom.iecc.com (netqmail 1.05+tls+ssl) NO UCE, NO UBE, C=US L=NY
250-This server is operated by I.E.C.C., Trumansburg NY USA.
250-We do not accept unsolicited bulk mail.
250- ------- snip: stuff removed ---
250 tom.iecc.com (netqmail 1.05)
mail from: <[email-masked]>
250 ok

Such is life with hypocrisy. :-)

I don't blame it. Most systems support the incorrect format as well.  But
ther are some systems that reject this type of transactions.

--
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

---

From: Michael Thomas <[email-masked]>
Date: Mon, 17 Jul 2006 08:08:40 -0700

Mark Delany wrote:

>How many of those were signed though? 
>
All.

>And what was the intent of the
>signer wrt these mail-formed mails? 
>

That they be signed and that they be verifyable.

>And hundreds of millions over many
Sorry, we're just a tiny enterprise.

       Mike

---

From: [email-masked]
Date: Mon, 17 Jul 2006 07:56:46 -0700 (PDT)

> being passed through SMTP.

I know of only one counterexample. It is possible to write PostScript that
depends on line termination characters not changing from one character to two
or vice versa. (The specific underlying technical issue is that there's an
operator in the language that takes a counted sequence of characters, possibly
containing line breaks, as an argument and if you change the length around
you'll desynchronize the reading process. I forget the specific operator but I
can look it up if anyone cares.)

Now, if such PostScript is sent unencoded (as is sometimes the case -
PostScript looks textual so people figure they can get away with handling it as
text), you  now have a situation where changing those bare CRs and LFs to CRLFs
effectively destroys the content. Of course anyone who sends unencoded
PostScript around is asking for trouble, since there are plenty of other things
you can do with PostScript content that don't conform to either 7bit or 8bit
text restrictions (overly long lines, binary content, etc.).

> I've seen lots and lots of of naked CR and LF

I haven't seen this problem in well over five years so I suspect that the
systems that did it are long gone. (The sorts of PostScript that get generated
have also changed dramatically over time.) Our solution to the problem at the
time was to write a PostScript fixer-upper utility that dealt with this and
several other corruption issues. (There are a bunch that are far more common
than this one.)

> What's the benefit of being backward compatible with buggy behavior that

Very little, I believe. I will also point out that while RFC 821/822 don't
prohibit the use of bare CR and LF in messages, they are somewhat
self-contradictory in specifying their handling.  For one thing, section 3.1.2
of RFC 822 in effect bans the use of bare CR and LF in header field bodies, but
the ABNF for some header fields allows them (via the "text" production). And
while bare CR and bare LF are allowed in messages bodies and presumably are
distinct from CRLF line breaks, section 3.4.10 of RFC 822 makes it clear that
converting messages to and from local line termination conventions is OK even
when those conventions will have the effect of causing those bare LFs, CRs, or
both to be converted to CRLFs.

It is therefore a stretch to believe that there was this happy world of systems
merrily exchanging messages containing mixtures of line terminators with nary a
problem, foot loose and fancy free, up until big nasty old RFC 2821/2822 came
along and told them they had to stop. The fact is that bare CRs and LFs have
never interoperated worth spit and that's the reason why RFC 2821/2822 banned
their use. it's one thing to remove something from the allowed set when things
have worked well, quite another when things worked poorly.

				Ned

---

From: [email-masked]
Date: Mon, 17 Jul 2006 08:32:01 -0700 (PDT)

> months sounds like a tiny sample.

And more to the point, do you have any way of tracking what happened to these
messages between the time they were composed to the time they were read? Most
submission agents find it necessary to perform a huge array of fixups because
the stuff that clients emit is often egregiouly bad. Just because the tiny
fraction that underwent signing at some point after initial submission and
verification at some point shortly thereafter did OK doesn't mean that nothing
interesting happened at other points along the path.

I have to say I see a lot of "blind man and the elephant" thinking going on
here. We have have our own set of experiences with email, but the Internet is a
huge and surprisingly disjoint place, and none of us can claim to have a good
understanding of everything out there. In my particular corner I deal with
problems caused by malformed messages all the time. (Last week it was some
voicemail thingie that was submitting what amounted to raw binary material in
plain old SMTP. And the system in question is apparently widely deployed - the
people who developed it are trotting out the time honored "this is the first
time there has been a problem so it must be someone else's fault" canard.) But
I have no problem believing that someone else's experiences in a similar
position with a different development group would be quite different.

				Ned