Subject: [ietf-dkim] More on naked CR canonicalization

From: Michael Thomas <[email-masked]>
Date: Fri, 14 Jul 2006 16:55:23 -0700

In a word, it's very, very real, and I believe that simple 
canonicalization will need
to be changed to accommodate this. I just read through RFC 2822 and it 
has this
to say:

2.3. Body

   The body of a message is simply lines of US-ASCII characters.  The
   only two limitations on the body are as follows:

   - CR and LF MUST only occur together as CRLF; they MUST NOT appear
     independently in the body.

   - Lines of characters in the body MUST be limited to 998 characters,
     and SHOULD be limited to 78 characters, excluding the CRLF.

   Note: As was stated earlier, there are other standards documents,
   specifically the MIME documents [RFC2045, RFC2046, RFC2048, RFC2049]
   that extend this standard to allow for different sorts of message
   bodies.  Again, these mechanisms are beyond the scope of this
   document.

and Section 4.0:

   Finally, certain characters that were formerly allowed in messages
   appear in this section.  The NUL character (ASCII value 0) was once
   allowed, but is no longer for compatibility reasons.  CR and LF were
   allowed to appear in messages other than as CRLF; this use is also
   shown here.

This says that free (naked) CR or LF's are specifically *disallowed* in 
RFC2822
compliant implementations. However, section 4 mentions the uncomfortable 
truth
that they were allowed in previous (ie, 822) revisions. In fact, those 
implementations
are as common sendmail since even sendmail 8.13.6 still allows naked 
CR's to be
transmitted instead of CR stuffed as it does with LF stuffing.

As I said, what we've been seeing is that naked CR's are a not uncommon 
occurrance,
and my suspicion is that they are often due to windows file system's use 
of CRLF as
its line terminator, though I'm sure there are lots of other reasons 
that this is common.

To my mind, the canonicalization problem is actually a problem of simple 
too as we
really need to accommodate the entire[2]822 universe, and regardless of 
whether
2822 outlaws it, there's lots of perfectly useful applications that are 
unaware or
unconcerned that they are not in spec.

I believe that the *only* change needed to simple is to say that free CR 
or LF
as defined in RFC2822 MUST be canonicalized to CRLF, and this problem
goes away.

       Mike

_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Mark Delany <[email-masked]>
Date: Sat, 15 Jul 2006 00:40:53 +0000

> to be changed to accommodate this. I just read through RFC 2822 and it 

To work around broken mailers? I think not.

A good number of sites reject bare LFs. Also at least the standard
qmail inbound smtp rejects such email. I hope that other well
implemented receivers do the same.

Hopefully the trend to accept any goop is declining rather than
increasing. I expect my inbound SMTP receiver to only present me with
a valid SMTP stream.

Mark.

---

From: Michael Thomas <[email-masked]>
Date: Fri, 14 Jul 2006 18:08:27 -0700

>implemented receivers do the same.
Sendmail is broken? RFC 822 is "broken"?

I'm sorry, RFC2822 was published in 2001 --  a blink of the eye. To ignore
legacy is foolhardy, and is as foolhardy as relying on simple for headers
as an "aspiration".

>increasing. I expect my inbound SMTP receiver to only present me with
Goop is timeless.  We are here to patch a broken mail system, not to 
design it
as if we could go back to first principles. Our experience is that the 
kind of
applications that generate this kind of mail are as common as dirt, and
they don't have any aspirations for email at all.

       Mike

---

From: Michael Thomas <[email-masked]>
Date: Fri, 14 Jul 2006 18:09:32 -0700

Mark Delany wrote:

>A good number of sites reject bare LFs.
>

Also: note I'm talking about bare CR's. Do any do that?

       Mike

---

From: [email-masked]
Date: Fri, 14 Jul 2006 18:34:41 -0700 (PDT)

> Also: note I'm talking about bare CR's. Do any do that?

Yes. And some convert bare CRs to CRLFs. And some silently drop the CRs.
And some leave them unchanged. And some sites even make use of more than
one of these modes of operation depending on message source or destination.

Like it or not, you are not going to be able to define a canonicalization that
can tolerate all the cases and possible transformations. My best advice is not
to try. Signing messages that are syntactically invalid is going to be
unreliable and the only solution is to stop creating such messages.

				Ned

---

From: Mark Delany <[email-masked]>
Date: Sat, 15 Jul 2006 02:06:24 +0000

> Sendmail is broken? RFC 822 is "broken"?

Actually, oftenso, yes.

My main point is that all of these issues are out of scope. Let your
MTA best decide how to deal with non-standard SMTP and presume that it
will present standard rfc2822 to a verifier.

The same applies for signing. I can cite any amount of broken submit
code - largely because most of it is passthru rather than verify - but
as signers and verifiers our mechanism should assume legitimate input.

In short, this is a layering issue and DKIM does not deal at the
transport layer.

Apropos your concern, I would suggest that at best we *might* want to
give some BCP on how non-standard streams should be dealt with by MTAs
and SUBMITs prior to presentation for verification and signing,
respectively. But that's about it. We definitely do not want to get
into compensating for transport layer bugs. After all, what happens
when we we use this for SIP or HTTPS or ... do we want to embed support
for their peccadilloes as well?

Mark.

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 14 Jul 2006 23:20:56 -0400

----- Original Message ----- 
From: "Michael Thomas" <[email-masked]>
To: "IETF-DKIM" <[email-masked]>
Sent: Friday, July 14, 2006 7:55 PM
Subject: [ietf-dkim] More on naked CR canonicalization

> or LF as defined in RFC2822 MUST be canonicalized to CRLF, and this 
> problem goes away.

+1

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 14 Jul 2006 23:20:03 -0400

----- Original Message -----
From: <[email-masked]>
To: "Michael Thomas" <[email-masked]>
Cc: "Mark Delany" <[email-masked]>; <[email-masked]>
Sent: Friday, July 14, 2006 9:34 PM
Subject: Re: [ietf-dkim] More on naked CR canonicalization

> unreliable and the only solution is to stop creating such messages.

That might be more easily said than done.  I think the issue is more about
"easy of use" vs. "required processing."   For legacy reasons, the "simple"
method is not going to be as easy to support.

Thomas is theoretically correct,  for canonicalization purposes, all bare CR
and LF eol delimiters should be corrected to CRLF pairs.  This would
theoretically resolve the problem and it will reduce integrated framework
implementation issues.

I can attest that would be the case for us.   DKIM is not as easy to
implement as Crocker believe it is.   Software Change is required and the
more integrated your system is, the more changes it needs.

All signers and verification used a single transformation or a consistent
method this would solve the problem and then Crocker would be more correct -
Less software changes is requires because now the DKIM Function Generator
will be independent of feed type.

---

From: [email-masked]
Date: Fri, 14 Jul 2006 20:42:39 -0700 (PDT)

> theoretically resolve the problem and it will reduce integrated framework

Actually, it solves the problem neither in theory nor in practice: What happens
when one of these messages hits a machine that simply drops the offending bare
CRs? Such systems do exist - I've encountered them. (This is apparently done
because there also machines that emit bare CRs in SMTP responses which, if
treated as CRLFs, cause the SMTP command-response alignment to go awry.  I
personally advocate using different rules for the handling of bare CRs and LFs
in commands, responses, and data, but that's not how some folks implement
things.)

In any case, I could live with a rule that says to canonicalize bare CRs and
LFs to CRLFS as part of the canonicalization process, but let's please not
pretend that this will solve all the problems with how bare CRs and LFs
interact with DKIM.

				Ned

---

From: "Hector Santos" <[email-masked]>
Date: Sat, 15 Jul 2006 04:55:30 -0400

----- Original Message -----
From: "Ned Freed" <[email-masked]>

> Actually, it solves the problem neither in theory nor in practice: What
> happens when one of these messages hits a machine that simply drops
> the offending bare CRs? Such systems do exist - I've encountered them.
> (This is apparently done because there also machines that emit bare
> CRs in SMTP responses which, if treated as CRLFs, cause
> the SMTP command-response alignment to go awry.

That sounds a rare edge case. It is certainly not the norm.  But even then,
for this case, does it remove the EOL (regardless of type) of the line so
that two or more subsequent data lines become one?  Because if it doesn't  I
think.Thomas's suggestion still applies

> I personally
> advocate using different rules for the handling of bare CRs and LFs

Most system comply with RFC x822/x821.  Thank God. :-)   Most systems will
send out compliant CRLF delimited DATA.   But then again, I had no reason to
sit down to look for broken mail senders because our Receiver and post mail
processors will handle any format.  This is basically Postel's principal:
Conservative with what you send out, liberal on what you receive.   I think
most systems behave in this manner.

> pretend that this will solve all the problems with how bare CRs and LFs

I'm speaking in terms of a black box DKIM function generator for the signer
and verifier, that independent of any varying EOL formatted feed, it would
be able to produce the same result.   I believe that is it possible.

Of course, there will be rare exception to the rule with the extremely rare
highly broken edge cases, which is probably problematic for other things as
well, especially if they are joining DATA lines.  I'm sure they will be see
the problems once they attempt to implement DKIM.

---

From: Michael Thomas <[email-masked]>
Date: Sat, 15 Jul 2006 02:31:14 -0700

Ned Freed wrote:

>>>Like it or not, you are not going to be able to define a canonicalization
>>>      
>>>
>>that
>>    
>>
>>>can tolerate all the cases and possible transformations. My best advice is
>>>      
>>>
>>not
>>    
>>

The problem here is that up until a few years ago, the apps that did this
were compliant with STD11. This is quite a different situation than apps
that are just flagerant protocol scofflaws. RFC 2822's proscription is
probably a good idea since it clarifies something that wasn't entirely clear
in the previous standard, but I think it is quite another thing to design a
protocol which is trying to accommodate the mail universe as
it's currently constituted, not as we wish it were to be. These apps exist
whether we like it or not, and as far as the app owners are concerned they
aren't broken since they manifestly do the job their owners expect.

I also think that whether there are MTA's that reject 2822 non-compliant
messages is a bit of a red herring: they'd do that with or without the help
of DKIM. The fact still remains that they are outliers, and while that is
the case, it seems perfectly reasonable for DKIM to be robust.

>pretend that this will solve all the problems with how bare CRs and LFs
Fine, but how else do they interact? That's the only thing in my code 
that cares
about this subject.

       Mike

---

From: Barry Leiba <[email-masked]>
Date: Thu, 20 Jul 2006 10:54:47 -0400

(I'm just picking a random message from this thread to respond to; this 
isn't a specific response to this specific message.)

I see a clear consensus in this discussion, and I think the issue's 
actually already handled in the spec.  In section 5.3, we say this:

    Should the message be submitted to the signer with any local encoding
    that will be modified before transmission, such conversion to
    canonical form MUST be done before signing.  In particular, some
    systems use local line separator conventions (such as the Unix
    newline character) internally rather than the SMTP-standard CRLF
    sequence.  All such local conventions MUST be converted to canonical
    format before signing.

I think it might be helpful to mention that some messages are formed 
with bare-CR, as well as with bare-LF, to clarify this particular 
situation, but beyond that we're OK.

In other words, rather than making this a canonicalization issue, I 
suggest that we make it an issue in the normalization step of the 
message.  (For those who aren't sure, the distinction is that the 
normalization step is actually permanently changing the message by 
putting it into normalized form, which the canonicalization is only 
changing the bytes that get signed, but not changing the message itself. 
  Therefore, the normalization is only ever done once, before doing the 
rest of the steps for signing, and the verifier never has to know about it.)

Unless we hear objections to this other than from Mike, we'll call this 
finished.  So:
Mike: Can you live with this answer, even if it's not what you'd prefer?
Eric: Will you tweak that paragraph in 5.3 to mention that some messages 
that are considered compliant with RFC 82x fail complicance with RFC 
282x in this regard and MUST be normalized?

Barry

--
Barry Leiba, DKIM working group chair  ([email-masked])

---

From: Michael Thomas <[email-masked]>
Date: Thu, 20 Jul 2006 09:47:45 -0700

Barry Leiba wrote:

 > (I'm just picking a random message from this thread to respond to; 
this isn't a specific response to this specific message.)

This paragraph is rather misleading because it implies to me that you must
convert to the canonical form for the *hash* function, not that you must 
convert
the message before forwarding. I think that the problem is that the word 
"canonical"
really wants to be "2822 format", and in particular it might be nice to 
give a
cross reference to that section in 2822.

 > Mike: Can you live with this answer, even if it's not what you'd prefer?

Towel->Ring->Throw.

I'd still like to hear from Eric about this though.

       Mik

---

From: Barry Leiba <[email-masked]>
Date: Thu, 20 Jul 2006 14:29:26 -0400

> convert the message before forwarding.

OK; if you can propose a rewrite to Eric that'll make the intended 
meaning clearer, I'm sure he'll appreciate that.

> I'd still like to hear from Eric about this though.

Yep; me too.  He's away (and away from computers), and will be back some 
time next week.

Barry

--
Barry Leiba, Internet Messaging Technology  ([email-masked])
http://www.research.ibm.com/people/l/leiba
http://www.research.ibm.com/spam

---

From: Mark Delany <[email-masked]>
Date: Thu, 20 Jul 2006 18:52:50 +0000

> meaning clearer, I'm sure he'll appreciate that.

What Mike was saying sounded pretty good. Along the lines of "must
produce rfc2822 conformant output as a consequence of signing" sounds
about right.

Mark.