Subject: [ietf-dkim] IETF-SMTP signed mail DKIM BODY HASH Failures

From: Hector Santos <[email-masked]>
Date: Fri, 13 May 2011 18:23:37 -0400

I am wondering if anyone else can confirm BODY HASH errors with the 
originating author domain DKIM signature mail submitted to the 
IETF-SMTP fora.

The IETF-SMTP MLM does not resign and in addition, it does not change 
the subject (adding [IETF-SMTP]) and does not appear to change the 
body by adding any list footer.

Here is a few A-R headers by my verifier all showing the same error.

Authentication-Results: dkim.winserver.com;
   dkim=fail (DKIM_BODY_HASH_MISMATCH)
        header.i=santronics.com
        header.d=santronics.com
        header.s=tms1;

Authentication-Results: dkim.winserver.com;
   dkim=fail (DKIM_BODY_HASH_MISMATCH)
        header.i=mrochek.com
        header.d=mrochek.com
        header.s=mauve;
	
Authentication-Results: dkim.winserver.com;
   dkim=fail (DKIM_BODY_HASH_MISMATCH)
        header.i=messagingengine.com
        header.d=messagingengine.com
        header.s=smtpout;

I was thinking it was an C14N issue, but the first two are 
simple/simple and the last one is relaxed/relaxed.

What could be hidden in this body that is different?

-- 
Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: SM <[email-masked]>
Date: Sat, 14 May 2011 13:03:56 -0700

Hi Hector,
>originating author domain DKIM signature mail submitted to the

Yes.  It may be an extra line between the message headers and the body.

Regards,
-sm

---

From: Hector Santos <[email-masked]>
Date: Sat, 14 May 2011 16:48:17 -0400

> Yes.  It may be an extra line between the message headers and the body.

Visually comparing the sent message versus the one echoed back by the 
list, that seems to be the case.   Checking into this, I see that I 
discovered this issue back in 2006 and wrote this I-D proposing a new 
C14N method called STRIP.

     http://tools.ietf.org/html/draft-santos-dkim-strip-00

Abstract

    The DKIM base protocol has offers two digital signature
    canonicalization (cl4n) methods called "relaxed" and "simple" with
    low reliability and survivability during in-transient operations.
    This proposal describes a new STRIP canonicalization algorithm and
    method to increase the reliability and survivability of the digital
    signature.  In additional, the proposal describe new original body
    hashing requirements to help secure STRIP c14n security concerns
    found in a similar but deprecated NOFWS c14n method.

 From the 1.0 introduction:

    ....

    This documents introduces the new STRIP c14n which is similar to
    RELAXED but with the added logic to remove all CR and LF characters
    from the hashing engine.  The STRIP c14n is very similar to the NOFWS
    c14n method used by Yahoo's experimental DomainKeys protocol and was
    once considered for usage for the DKIM protocol.  However, since it
    was determined the NOFWS c14n exhibited some replay security threats,
    it is expected for STRIP c14n to also inherent the same security
    concerns.

The security concern stated in the final sentence were addressed in 
this proposal.

---

From: Hector Santos <[email-masked]>
Date: Sat, 14 May 2011 18:20:31 -0400

SM,

Shouldn't the MLM I-D say something regarding C14N and CR/LF related 
mutations?

For example, it can say something in:

   - Section 3.2 for the Aliasing MLM type
   - Section 3.3 for the Minor Body Changes possibility.

Maybe something in one of the handling sections:

     Verifications of list messages resulting with an invalid body 
hash MAY
     check to see if there is an extra line between the message 
headers and
     the body and retry the body hash verification with the line stripped.

I hate kludges but the insight for interested DKIM verifiers may help 
increase valid signatures coming from Aliasing list streams with 
slight CR/LR mutations.

-- 
HLS

---

From: SM <[email-masked]>
Date: Sat, 14 May 2011 20:08:01 -0700

Hi Hector,
>Shouldn't the MLM I-D say something regarding C14N and CR/LF related

No.

>increase valid signatures coming from Aliasing list streams with

I doubt that the example you are referring to is an "Aliasing" list stream.

Regards,
-sm

---

From: Hector Santos <[email-masked]>
Date: Sun, 15 May 2011 00:40:04 -0400

>> Shouldn't the MLM I-D say something regarding C14N and CR/LF related

Can't share the wisdom why?

> I doubt that the example you are referring to is an "Aliasing" list stream.

Just going by what the MLM I-D used for a "no change" MLM 
classification, i.e. a basic address expander.

---

From: "John R. Levine" <[email-masked]>
Date: 15 May 2011 09:44:05 -0400

>> Shouldn't the MLM I-D say something regarding C14N and CR/LF related

+1 to the No.

I have my reservations about the draft, but this is not one of them.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Hector Santos <[email-masked]>
Date: Sun, 15 May 2011 11:15:29 -0400

> I have my reservations about the draft, but this is not one of them.

In general, I would say NO too because I don't like kludges.

My point is that the draft is already peppered with scenarios about 
how MLM can break things and it properly classifies the known simple 
"list-like" type of "alias" address expanders that in general, the 
messages is not altered.

Of course, we all know its not the only kind; a real List Server 
always provides list admin options to not alter things like the 
IETF-SMTP list seems to be been setup (with mailman?).

But here, it appears it does add a extra <CRLF> after the headers.

So my question is about whether we should provide an "informative 
implementator" insight that there may be an extra <CRLF> generated by 
non-DKIM aware list.  It can make all the difference in a valid versus 
invalid DKIM signed submission to a non-DKIM aware MLM, which BTW, I 
did add logic to my verifier to check for this extra <CRLF> when a 
BODY_HASH error first occurs and redo the hash without it.  It works! 
  But I am probably going to add a condition based on LIST-ID to 
enable this check.

I fail to see why we would not be interested in giving verifiers some 
insight into this real live scenario.

Is it because its a more general DKIM issue and ideally belongs in 
RFC4671bis (too late)?

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 15 May 2011 08:52:32 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of John R. Levine
> Sent: Sunday, May 15, 2011 6:44 AM
> To: SM
> Cc: DKIM List
> Subject: Re: [ietf-dkim] MLM and C14N
> 
> > Hi Hector,

+1 to the No.

This is a software problem, not something that needs to be solved by creating a protocol extension.

---

From: Hector Santos <[email-masked]>
Date: Sun, 15 May 2011 12:24:48 -0400

Murray S. Kucherawy wrote:
> This is a software problem, 

You mean the MLM who is ignorant of DKIM meta data for the past "40 
years?"

> not something that needs to be solved by creating a protocol extension.

Isn't that what you are doing in this MLM I-D, creating a DKIM 
"protocol extension" for MLMs?

IMV, this MLM I-D is 100% about a MLM and VERIFIER mail integration 
protocol inconsistency, i.e. software problems, in dictating whats MLM 
and Verifier Software need to be considered and change to retrofit 
DKIM and ADSP into a MLM and MLM related issues with verifiers.

IMV, the issue equally falls in the same MLM chaotic environment in 
how there long legacy behavior and different ways it can break 
transparent meta data we call DKIM.  The extra <CRLF> preexisted DKIM 
- we can tell these software to not do this or that for the sake of 
DKIM, but is that realistic?  I don't think so - the burden is on DKIM.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 15 May 2011 09:29:09 -0700

> From: Hector Santos [mailto:[email-masked]]
> Sent: Sunday, May 15, 2011 9:25 AM
> To: Murray S. Kucherawy
> Cc: DKIM List

Yes, that's what I mean.

> "protocol extension" for MLMs?

No, that's not what that document does.

Creating a new canonicalization is a protocol extension.  It creates a slippery slope wherein we are constantly creating or adjusting canonicalization algorithm(s) to deal with new cases we find and want to accommodate.  It's a bad idea.

---

From: SM <[email-masked]>
Date: Sun, 15 May 2011 10:44:11 -0700

Hi Hector,
>Can't share the wisdom why?

It's merely an opinion.  Please see Murray's comment about a slippery slope.

Regards,
-sm

---

From: "J.D. Falk" <[email-masked]>
Date: Sun, 15 May 2011 12:56:49 -0500

+1 to the No.

--
J.D. Falk
the leading purveyor of industry counter-rhetoric solutions

---

From: Hector Santos <[email-masked]>
Date: Sun, 15 May 2011 15:01:27 -0400

> It's merely an opinion.  Please see Murray's comment about a slippery 

I understand the point. But we are doing all this already like section 
3.3 Current MLM Effects On Signatures; Minor body changes.  It says:

    Minor body changes:  Some lists prepend or append a few lines to each
       message to remind subscribers of an administrative URL for
       subscription issues, or of list policy, etc.

The sentence can include or be extended:

       In addition, some list changes are not obvious and may be
       just an extra line between the header and start of the body.

This a MLM mail integrity deviation just like all the other obscure 
MLM mail integrity issues we can have.   I personally would like to 
extend it to say:

       This is an C14N issue that is out of scope but its possible
       invalid signatures can be valid when the extra line is removed
       from the hashing.

Again, this is a real live MLM stream scenario. Why doesn't anyone 
care about addressing these types of MLM streams?

This is suppose to be an informative guide about the issues 
retrofitting an MLM with DKIM.  Not the other way around.

Why not provide the insights, all the issues and let the verifier 
decide? Why not let the MLM developer learn what his software might be 
doing so he might fix it?

---

From: Hector Santos <[email-masked]>
Date: Sun, 15 May 2011 15:30:28 -0400

> No, that's not what that document does.

At the end of the day, this document is describing incompatibility 
issues and provide "suggestions" (remember it is suggestions not a 
mandate) in how (via protocol methods changes, administrative protocol 
setups, etc) with an idealized goal to minimize the impact of the 
incompatibility issues.

If that is not what it does, what does it do for its potential 
audience of?

    - DKIM signer developer
    - MLM developer,
    - DKIM verifier developer

What does one get out of this document?

---

From: Dave Crocker <[email-masked]>
Date: Sun, 15 May 2011 09:29:34 -0700

+1.  No.

---

From: Hector Santos <[email-masked]>
Date: Sun, 15 May 2011 18:54:09 -0400

Do you know what is being asked?

A real live LIST organism (stream) is adding an extra line (two bytes, 
<CR><LF>) after the header and before the body probably all its life 
and this new "thing," this new document called:

          DKIM And Mailing Lists

has no interest in adding another note in section 3.3 titled

          "Current MLM Effects On Signatures"

regarding a real living, walking and running MLM behavior?

This is surreal. Don't shoot the messenger, listen to the message!

---

From: Barry Leiba <[email-masked]>
Date: Sun, 15 May 2011 22:55:47 -0400

> Do you know what is being asked?

I think everyone understands what you're asking.  They just disagree,
and so do I.

When all this started, before DKIM came to the IETF, and then again
afterward, we spent a *lot* of time looking at the canonicalization
algorithms -- and they were changed a bit after the work came into the
IETF.  We didn't come by them casually.  Throughout it, there were a
few goals in choosing canonicalization algorithms:

1. We didn't want more than one or two.  This obviously never did nor
never should take precedence over a true need for another one, but the
idea is that the bar needs to be very high for a new one.  The
combinatorics get messy.

2. We wanted to cover the vast majority of the cases, though we knew
there'd always be outlying situations where some mail would get broken
because what we had didn't *quite* cover some other case.  We decided
to accept that.

3. We absolutely did *not* want to go adding new algorithms for this
or that piece of software that was getting something wrong.
Canonicalization algorithms were *not* designed to work around broken
software.  They're meant to deal primarily with legitimate, if
sometimes unfortunate, changes that get made to the mail along the
way, and secondarily with *very common* situations that creep into the
questionable area.

> <CR><LF>) after the header and before the body probably all its life

It's an outlier, off in the weeds.  This is not a common situation all
around the Internet.  And in any case, the MLM document isn't the
place to define a new algorithm.

> This is surreal. Don't shoot the messenger, listen to the message!

There's nothing surreal here, and, again, I ask you to stop being
extreme and inflammatory.  Calm discussion, please.  Anyway, we *are*
listening to the message.  It seems that at least five people have
listened, and responded.  Their response is simply one of
disagreement.  Listening is not the same as agreeing.

Six.  I also disagree.

Barry

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 15 May 2011 21:15:03 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Barry Leiba
> Sent: Sunday, May 15, 2011 7:56 PM
> To: Hector Santos
> Cc: DKIM List

I would add that adding a new canonicalization now has an extremely high cost to the people that want to use it, because signatures using it will go unverified for a very long time until software supporting the new canonicalization is widely deployed.  We shouldn't underestimate what all of that means just to handle one or two corner cases that are actually more likely broken software than an actual universal problem mandating a protocol extension.

The same goes for new signing algorithms.  ECC, whenever DKIM is extended to cover it, will be an interesting experiment.

---

From: Hector Santos <[email-masked]>
Date: Mon, 16 May 2011 00:17:19 -0400

> place to define a new algorithm.

I winged out two things; mentioning the MLM behavior which is real, 
live, current and if you like, a recommendation a "kludge" for 
verifiers to consider.

I already said I don't like kludges, but I fail to see why it can't be 
mentioned that MLM exists that can add the extra line.

    Minor body changes:  Some lists prepend or append a few lines to each
       message to remind subscribers of an administrative URL for
       subscription issues, or of list policy, etc, including some lists
       adding an extra line after the header and before the body. ....

> Six.  I also disagree.

Fine, but did you ever see the movie "12 Angry Men?" :)

No, I have no interesting continuing this but for the record, I do 
find it very inconsistent to ignore a real scenario for an MLM I-D 
who's main purpose is to provide information about the real MLM/DKIM 
incompatibility issues and intentionally wants to hide this insight 
that could promote changes.

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 16 May 2011 12:22:07 +0200

On 16/May/11 06:15, Murray S. Kucherawy wrote:
>> From: On Behalf Of Barry Leiba

However, relaxed header canonicalization is not MIME compatible.  In
particular, RFC 2045 says

 Note that the value of a quoted string parameter does not include the
 quotes.  That is, the quotation marks in a quoted-string are not a
 part of the value of the parameter, but are merely used to delimit
 that parameter value.  In addition, comments are allowed in
 accordance with RFC 822 rules for structured header fields.  Thus the
 following two forms

  Content-type: text/plain; charset=us-ascii (Plain text)

  Content-type: text/plain; charset="us-ascii"

 are completely equivalent

but they are not DKIM-wise equivalent.  Fixing this and a couple more
nits, we could claim to cover text/plain, which is still not quite the
vast majority of the cases.

> I would add that adding a new canonicalization now has an extremely
> high cost to the people that want to use it, because signatures
> using it will go unverified for a very long time until software
> supporting the new canonicalization is widely deployed.

Many MTAs still add a Domainkey-Signature.  They could stop that, and
add a legacy DKIM-Signature along with a one that features the newer
canonicalization instead.  Setting the pace in this manner can keep
DKIM development alive indefinitely.

---

From: "John R. Levine" <[email-masked]>
Date: 16 May 2011 09:00:52 -0400

> Thus the following two forms
>
> but they are not DKIM-wise equivalent.

I'm sorry, but this is just so wrong.

Helpful software can modify mail in a million ways that don't affect the 
way that a message renders.  If the contents of a message are in fact 
ASCII, these are also equivalent to the headers above:

  Content-type: text/plain; charset=UTF-8 (a superset of ASCII)
  Content-type: text/plain; charset="ISO-8859-2" (another superset of ASCII)
  Content-type: text/enriched; charset="windows-1252" (if there are no enriched codes)

The point of relaxed canonicalization was to deal with the kind of small 
changes that dusty copies of sendmail make, not to handle every possible 
message mutation that more or less renders the same.  In retrospect, it 
probably would have been better only to provide simple and tell people 
more firmly to do the signing after and the checking before any local 
modification.  The idea that an MUA can sign if an MTA doesn't is clever, 
but if anyone's doing that, it's news to me.

Perhaps Murray has data that says whether relaxed verifies much more 
often than simple does.

Regards,

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 16 May 2011 09:09:55 -0400

On 5/16/2011 9:00 AM, John R. Levine wrote:
> message mutation that more or less renders the same.

The underlying concern here actually is pretty reasonable: Variations that do 
not affect the appearance or semantics of a message could reasonably still 
permit a signature to verify.

The problem is that the working group was not able to develop a... workable... 
canonicalization algorithm to achieve this complete robustness.  In the extreme, 
this is a research topic.  Certainly it is a delicate engineering tasks, since 
too much robustness against change can easily introduce security holes.

But, then, that's why the working group debate the issue so extensively and the 
result did gain working group consensus.

Since the list of algorithms is defined to be extensible, anyone feeling that an 
additional algorithm is warranted is free to define it and seek community 
consensus for it.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 16 May 2011 15:26:02 +0200

On 16/May/11 15:00, John R. Levine wrote:
> In retrospect, it probably would have been better only to provide
> simple and tell people more firmly to do the signing after and the
> checking before any local modification.

That implies hop to hop rather than end to end.  What would the
advantage over SPF be then?

> Perhaps Murray has data that says whether relaxed verifies much more 

Yes, http://www.opendkim.org/stats/report.html#hdr_canon says

Header canonicalization use:
canonicalization	count	domains	passed
simple                  653688	6786	591938
relaxed                 3940377	56621	3640854

Although they only differ by 2% (90% simple vs 92% relaxed), such
percentages would be superb for tools like Spamassassin.  I'd expect
at least 99% from a cryptographic tool.

---

From: Hector Santos <[email-masked]>
Date: Mon, 16 May 2011 09:29:14 -0400

This was done in in 2006. I took up Stephan's suggestion to write an I-D

It addressed the concerns related to NOFWS and that of which is still 
present with RELAXED.

---

From: "John R. Levine" <[email-masked]>
Date: 16 May 2011 09:36:55 -0400

> permit a signature to verify.

Oh, sure, but we also traded off the cost of handling changes and how 
common they are.  For example, old copies of sendmail often add an extra 
blank line at the bottom of a message.  That's common (or at least, was 
common), and easy to deal with, and is the kind of thing that relaxed 
handles.  The variety of MIME rewrites is so vast that I don't see any 
hope of handling a usefully large set of them, so I'm not inclined to try.

If you really really really want your signature to verify, after signing 
the message, turn it info a base64 encoded message/rfc822 mime part, wrap 
another message around it, and unwrap it before verifying.  That works 
with S/MIME, too.

Regards,

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 16 May 2011 09:38:18 -0400

On 5/16/2011 9:36 AM, John R. Levine wrote:
> If you really really really want your signature to verify, after signing the
> message, turn it info a base64 encoded message/rfc822 mime part, wrap another
> message around it, and unwrap it before verifying. That works with S/MIME, too.

absent a standards-track specification for it being adopted, it's not interoperable.

d/

-- 

   Dave Crocker

---

From: "John R. Levine" <[email-masked]>
Date: 16 May 2011 09:41:16 -0400

> advantage over SPF be then?

The fact that most hops don't break even simple signatures.  We went 
through all this in 2006 (RFC 4686) and I don't see any reason to revisit 
it now.

> at least 99% from a cryptographic tool.

This tells me that the benefit from relaxed is at most pretty small.

Regards,

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 16 May 2011 16:28:06 +0200

On 16/May/11 15:41, John R. Levine wrote:
>> http://www.opendkim.org/stats/report.html#hdr_canon says
>>

OTOH, comparing the "count" fields of those two lines, 86% relaxed vs
14% simple, says that such kind of benefit is really really wanted.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: "Mark Delany" <[email-masked]>
Date: 16 May 2011 07:40:12 -0700

> 14% simple, says that such kind of benefit is really really wanted.

But that's a perceived benefit, not an actual one.

Folk think they need "relaxed" to significantly increase survivability
but that's not the case given the stats above. So yo may be right that
folk really really want it, but they don't really really need it.

Mark.

---

From: "Mark Delany" <[email-masked]>
Date: 16 May 2011 06:48:43 -0700

>  are completely equivalent

DKIM has never tried to understand the semantics or syntax of each
header. First off, doing this properly is very hard as there are a
large number of "equivalent" header variations, but second off, it
would make DKIM brittle against future headers that might have the
same characteristics you describe.

Mark.

---

From: SM <[email-masked]>
Date: Mon, 16 May 2011 06:40:34 -0700

Hi Alessandro,
>particular, RFC 2045 says

I don't think that it was ever meant to be MIME compatible.

>DKIM development alive indefinitely.

The MTAs that are still adding a Domainkey-Signature will keep doing 
that irrespective of what goes on in this working group.  I prefer 
not to keep DKIM development alive indefinitely if the sole purpose 
is not to produce results.

Regards,
-sm

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 16 May 2011 12:39:51 -0400

> But that's a perceived benefit, not an actual one.

I agree that the above does not give us insight into actual benefit.  Rather, it 
tells us something about beliefs and goals of signers; they chose one or the 
other because they /believed/ it would be helpful. As to whether it really was, 
we can't see here.

> folk really really want it, but they don't really really need it.

Sorry, but I believe the above also does /not/ help us to understand actual 
survivability differences.

To assess that difference, the experiment needs to send the same set of message 
twice, one with each type of canonicalization, and then see what the survival 
differences are.

The problem with the above is the biasing factor of signers' choosing to use one 
or the other, based on criteria we can't know about.  Their criteria might have 
greatly affected actual survival rates.  Or might not have...

d/

-- 

   Dave Crocker

---

From: Michael Thomas <[email-masked]>
Date: Mon, 16 May 2011 08:42:44 -0700

> folk really really want it, but they don't really really need it.

It was our experience that relaxed body didn't make much
difference. Relaxed headers was a different story.

I don't have access to all of the ways signatures broke any
more, but if you factor out "real" mailing lists it was pretty
low and sort of random, iirc.

Mike

---

From: Michael Thomas <[email-masked]>
Date: Mon, 16 May 2011 10:00:10 -0700

> greatly affected actual survival rates.  Or might not have...

My guess is that admins just don't understand any of the subtleties,
have heard lore that "relaxed" is "better" and just click "relaxed"
wherever they find it. It may also be the case that some implementations
don't even have separate nerd knobs for headers and body canonicalization.

Mike

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 16 May 2011 10:45:26 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Thomas
> Sent: Monday, May 16, 2011 10:00 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [ietf-dkim] New canonicalizations
> 

Some of the data (especially message-specific stuff) are certainly skewed by the huge senders that use one or the other.  It's more interesting to look at percentage by domains.

For the most part what I hear is "relaxed/simple" is the BCP, but Gmail (for example) uses "relaxed/relaxed" which really pulls things quite a bit in one direction.

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 16 May 2011 19:50:47 +0200

> don't even have separate nerd knobs for headers and body canonicalization.

However, Murray's stats show some difference in the choice of relaxed:

Body canonicalization use:
canonicalization	count	domains	passed
simple	                1187858	11526	1096204
relaxed	                3406207	51818	3136588

For the body count, we have 74% relaxed vs 26% simple, while it is 86%
relaxed vs 14% simple for the header.  There is a 12% difference
toward relaxing the header, which implies some thought or testing.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Ian Eiloart <[email-masked]>
Date: Tue, 17 May 2011 14:45:20 +0000

> relaxed                 3940377	56621	3640854

It does, but how does one interpret that? Certainly the weight of relaxed versus simple passes implies a user desire for relaxed canonicalisation.

However, the 90% versus 92% is meaningless without making certain assumptions. If all these messages were originally properly signed, then the 2% represents a 20% reduction in false negatives, but only if we assume that canonicalisation method was selected at random or that choice of canonicalisation method was statistically independent of the likelihood of breakage - the latter might be plausible.

However if some of the messages were never properly signed (whether failed attempts to spoof, or administrative or technical failure), then that 20% must be higher. It could even represent 100% reduction in false negatives due to (otherwise benign) in-flight modifications.

-- 
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 May 2011 12:39:15 -0400

> don't even have separate nerd knobs for headers and body canonicalization.

Based on what I see, one SWAG is that the "good" intention people are 
using the defaults or relaxed/simple,  and spammers tend to use 
relaxed/relaxed as the reduced restraint.   By far, in my samplings, 
the largest group are spammers using relaxed/relaxed.

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 17 May 2011 18:44:41 +0200

On 17/May/11 16:45, Ian Eiloart wrote:
> However if some of the messages were never properly signed (whether
> failed attempts to spoof, or administrative or technical failure),
> then that 20% must be higher. It could even represent 100%
> reduction in false negatives due to (otherwise benign) in-flight
> modifications.

Actually, those figures don't even distinguish between failures due
signature comparison and earlier errors, such as body-hash mismatch or
invalid key.  To run the test properly we'd need to put two
DKIM-Signatures with different canonicalizations, on each message.

I don't know what is going to happen with EAI and YAM, but one day
we'll have utf-8 in the header as well as in the body.  As it would be
very clumsy to insist for 7-bit normalizations at that point, I think
there will be a new revision; presumably, the next one after 4871bis.
 If we'll have some test results of a new canonicalization at that
time, showing, say, 95%~98% "pass", the new canonicalization can be
included in such future DKIM revision.  That would be a significant
improvement, won't it?
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 May 2011 13:43:23 -0400

> However if some of the messages were never properly signed (whether failed attempts to spoof, or administrative or technical failure), then that 20% must be higher. It could even represent 100% red…

What it seems to indicate to me is that its the common trait when 
isolated to the domain or domain organization.  IOW, the similar 
passage rate is a reflection of a domain implementation and 
verification pattern.

For example, a single person subscribed to two list and submitting the 
same message to two different list (i.e. a CC to a second list):

     LIST1 - DKIM AWARE, RESIGN
     LIST2 - DKIM NON-AWARE

can expect to see 100% pass with LIST1 but 100% fail with LIST2.

This tangent thread started with LIST2 did no body changes except for 
what it normally did of adding an extra line after the header and 
before the body.

The document editor and others believe this is a MLM BUG.  It could 
be, but we don't know if its really an normal attempt to add HEADER 
text that was empty:

Create List Message for Distribution:

     Body = EMPTY;
     Body +=  AppendText(GetHeaderNoticeForList()) + CRLF;
     Body +=  AppendText(GetMessageBody()) + CRLF;
     Body +=  AppendText(GetFooterNoticeForList()) + CRLF;

We just don't know. Of course, for programmers, one can easily see 
that there is a "mite" there where extra CRLF will be added.  We 
recognized it with the ending CRLF but "forgot" that list header text 
was also possible.  The key point is for "40" years, it wasn't a 
problem until a new kid in the block came and now demands MLMs adjust 
to work with it

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 May 2011 14:07:44 -0400

> problem until a new kid in the block came and now demands MLMs adjust 

As a follow up, the above is very conceivable when you also consider 
the addition of List-* which is normally done with templates:

      Head = EMPTY;
      Head = GetMessageHeaders();
      Head += GetListHeaders();

      Body = EMPTY;
      Body +=  TrimCRLF(GetHeaderNoticeForList()+CRLF;
      Body +=  GetMessageBody() + CRLF;
      Body +=  GetFooterNoticeForList() + CRLF;

      Mail = Head + CRLF + Body;

Of course, regardless of DKIM, to be aesthetically "clean" in 
displays,  it should perform trailing CRLF checks to avoid extra lines:

      Head = EMPTY;
      Head = GetMessageHeaders();
      Head += TrimCRLF(GetListHeaders());  // no extra CRLF

      Body = EMPTY;
      TextHeader = TrimCRLF(GetHeaderNoticeForList()); // no extra CRLF
      if (TextHeader != EMPTY) Body += TextHeader +CRLF;
      Body +=  GetMessageBody();
      TextFooter = TrimCRLF(GetFooterNoticeForList()); // no extra CRLF
      if (TextFooter) Body += CRLF+ TextFooter + CRLF;

      Mail = Head + CRLF + Body;

So its easy to see that a MLM not being 100% in its "clean" header and 
footer text additions to the header and body is possible without being 
label as a "BUG" or bad until something that deals with the purity of 
integrity and C14N comes into play.

The MLM I-D *SHOULD* at least acknowledge that an extra line is 
conceivable.

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 May 2011 14:33:35 -0400

Hector Santos wrote:
> the addition of List-* [headers] which is normally done with templates:
> 
>       Head = EMPTY;
>       Body +=  GetHeaderNoticeForList()+CRLF;

Base on possible code analysis , here is what I recommend:

current text in section 3.3:

proposed changed (starter text):

    Minor body changes:  Some lists prepend or append a few lines to each
       message to remind subscribers of an administrative URL for
       subscription issues, or of list policy, etc.  These few lines can
       have extra blank lines or empty text with only CRLF characters
       which under normal circumstances would be trimmed or pruned.

The goal is to give the MLM audience reading this document insights 
into their long running current code which may need some modern CRLF 
trimming considerations, if any, even if the MLM is non-DKIM aware or 
a list is setup by the admin not to DKIM resign mail submissions.   No 
need to mention C14N kludges or new C14N (i.e. STRIP) proposals.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 17 May 2011 12:02:11 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Hector Santos
> Sent: Tuesday, May 17, 2011 9:39 AM
> To: Michael Thomas
> Cc: [email-masked]; [email-masked]

According to what we have, the biggest users of "relaxed/relaxed" are the large mailbox providers like Gmail and Yahoo and other legitimate senders, not spammers.  The top 20, for example:

+----------------------------------+----------+
| name                             | count(*) |
+----------------------------------+----------+
| gmail.com                        |   421745 |
| yahoo.com                        |   313109 |
| facebookmail.com                 |   233441 |
| yahoogroups.com                  |   104523 |
| auth.ccsend.com                  |    90195 |
| linkedin.com                     |    74710 |
| google.com                       |    59049 |
| reply.newsmax.com                |    53286 |
| ATT.NET                          |    43602 |
| sbcglobal.net                    |    36534 |
| googlegroups.com                 |    34359 |
| e.groupon.com                    |    30350 |
| paypal.com                       |    24568 |
| f74d39fa044aa309eaea14b9f57fe79c |    21019 |
| emailinfo.bestbuy.com            |    17067 |
| ebay.com                         |    16192 |
| 636ae4d78ec2b46248fc59ac1ad737df |    14580 |
| expediamail.com                  |    13058 |
| bellsouth.net                    |    12431 |
| googlemail.com                   |    12426 |
+----------------------------------+----------+

Total relaxed/relaxed signatures received = 3444978; total above = 1626244 (47%)

In fact, the first domain name that (statistically) looked likely to be a spammer is way down on the list, around #106 (out of 63314), and everything before that accounted for 58% of total signatures.  So, our data don't agree with the claim, and certainly not with "by far".

But I don't understand why this is a useful line of analysis.  If spammers are using relaxed/relaxed, they merely have the same concern as a legitimate sender, namely signature survivability.  This shouldn't be a surprise.  I hope we're not talking about the idea of filtering based on which canonicalization is in use, which is almost certainly a bad idea.

-MSK

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 May 2011 16:29:46 -0400

> According to what we have, the biggest users of "relaxed/relaxed" are the large mailbox providers like Gmail and Yahoo and other legitimate senders, not spammers.  The top 20, for example:

Some good info Murray.

It is all reflective of whats called Peer or Personal Network 
Community (PCN).

The collection you have is an aggregate of many sites.  However, in 
reality each site will have a different PCN.

I agree, even for my small site collection, the majority volume are 
DKIM signed mail are from:

     Gmail
     Facebook

and for my PCN, the third is:

     mipassoc.org

But when you normalized it, there are a small part (3 to 4) of the 
total domains which are, by far, good/bad spammers.

When we started our SMTP daily stats collection in 2003, it started as 
a per site basis and the PCN patterns were obvious.  At some point, we 
automated the collection with the attempt to show an aggregation of 
the total sites.  Almost immediately, the various measurements were 
skewed in one direction or another simply because one or more sites 
had a higher measurement for thing or another.

For my PCN,  by the time mail is finally accepted, the RFC5322 payload 
is indeterminate (i.e. everything that could be done was done), and 
the analysis of the DKIM signed mail is that most of it are spamming 
domains.

While you may be eager to publicly state this input is insignificant 
and doesn't matter, my 35+ years of producing software for thousands 
of customers and inter-operating with my industry peers says it is 
very significant.  One can not always lump a total aggregation summary 
to reflect what is true or false at the site level. The fact that DKIM 
analysis is in a limbo state is reflective of whats I am stating.

Why not try redoing your stats for your PCN only and see what it shows?

Keep in mind we got of the ESP business in 1998, but we still got a 
lot of dirty, inactive addresses coming our way.  So our PCN will be 
very different.

Finally, in my opinion you have two motivations for C14N and it could 
be based simply on the degrees of separation of what one deems important:

    Private Communications:

        Desired to have the most secured integrity

    Bulk, Public Communications:

        More relaxed, less secured, with a wide degree of
        receivers, minimized C14N related issues with an
        relaxed algorithm.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 17 May 2011 16:02:49 -0700

> Sent: Tuesday, May 17, 2011 1:30 PM
> To: [email-masked]

Naturally.  And we can select for the data for a specific site if that's useful.  But in terms of input for developing a standard, even you must agree that an aggregate view is at least as important as a local one.

But since you asked, selecting only for my own server, there are 11 domains in the top relaxed/relaxed senders list before we hit the first obvious spammer.  Those are responsible for 7301 relaxed/relaxed messages out of the 16581 total that server has seen, or 44%.  And there are lots of obviously legitimate senders below that, but I wanted to keep this report simple.  Even so, it still doesn't concur with the apparent extremeness of your data, nor do I understand why this is an interesting statistic.

---

From: Ian Eiloart <[email-masked]>
Date: Wed, 18 May 2011 09:38:41 +0000

> According to what we have, the biggest users of "relaxed/relaxed" are the large mailbox providers like Gmail and Yahoo and other legitimate senders, not spammers.  The top 20, for example:

These aren't contradictory. If spammers are spreading their load across large numbers of domains, then it's entirely feasible that top 1000 signing domains are not spammers, but that spammers collectively are (or will one day be) responsible for over 50% of signatures. I don't know whether that's the case though.

---

From: Hector Santos <[email-masked]>
Date: Wed, 18 May 2011 09:31:10 -0400

Murray S. Kucherawy wrote:

>> Hector wrote:

> Naturally.  And we can select for the data for a specific site if
> that's useful.  But in terms of input for developing a standard, 
> even you must agree that an aggregate view is at least as important 
> as a local one.

Of course.

> But since you asked, selecting only for my own server, there are 
> 11 domains in the top relaxed/relaxed senders list before we hit 
> the first obvious spammer.  Those are responsible for 7301 
> relaxed/relaxed messages out of the 16581 total that server has seen, 
> or 44%.  And there are lots of obviously legitimate senders below that, 
> but I wanted to keep this report simple.  Even so, it still doesn't 
> concur with the apparent extremeness of your data, nor do I understand 
> why this is an interesting statistic.

I agree with the OP, the data implies the selection of relaxed was 
done with forethought. After all, it is not the default.

One poster stated it might be lack of understanding or just the 
appearance it is better or that the knob is too easy to set.

I stated one SWAG (Scientific Wide Ass Guess), based on what my PCN 
(Personal Community Network) samplings are showing - private domains 
tend to use the stronger C14N, spammers tend to use relaxed C14N.

That is not a generalization for everyone, although I do personally 
believe its logical and probably representative of most systems with 
the idea spammers are broadcasting their messages to more than one 
receiver. i.e. I would expect all receivers to see the same data 
results from mail sent by Long Horn Steak House, Red Lobster, Olive 
Garden food coupon spammers (same organization, different domains).

Finally, it isn't really that extreme from your 44%; in my PCN, 64% of 
the domains use relaxed/relaxed.   Thats two PCN's - average 54%.  But 
your aggregate data shown in an early post, moves it higher; 81% 
domains using body relaxed), 89% domains using header relaxed. Could 
not see data for the relaxed/relaxed set but I will venture that group 
is lower.

Interesting?

I can only see two things worth noting:

   - People are very conscious of deviating from the default,
   - Observation that most spammers use the relaxed C14N
     integrity signing.

also from your aggregate data:

   - failure/passage rate is nearly the same for simple vs relaxed.

Does it mean anything?

Well, it depends on one's perspective.

Does it change anything?

Probably not. The OP was probably making an indirect suggestion to 
consider other C14Ns.  But I don't known if that was the intent with 
the subject "New canonicalizations."

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 18 May 2011 10:42:26 -0700

> From: Ian Eiloart [mailto:[email-masked]]
> Sent: Wednesday, May 18, 2011 2:39 AM
> To: Murray S. Kucherawy
> > According to what we have, the biggest users of "relaxed/relaxed" are
> > the large mailbox providers like Gmail and Yahoo and other legitimate
> > senders, not spammers.  The top 20, for example:
> 
> These aren't contradictory. If spammers are spreading their load across
> large numbers of domains, then it's entirely feasible that top 1000
> signing domains are not spammers, but that spammers collectively are
> (or will one day be) responsible for over 50% of signatures. I don't
> know whether that's the case though.

The contradiction is in the assertion that primarily spammers are using relaxed/relaxed, not the quantity of domains involved.

---

From: Hector Santos <[email-masked]>
Date: Wed, 18 May 2011 16:48:43 -0400

Murray S. Kucherawy wrote:

>> Ian wrote:

> The contradiction is in the assertion that primarily spammers 
> are using relaxed/relaxed, not the quantity of domains involved.

I don't believe that was the correct assertion stated.

More correctly, in my PCN, for the quantity of domains involved, the 
majority were unknown spammers and by far, most of these used 
relaxed/relaxed.  It is a PCN empirical conclusion based on data 
reduction which I assert is not entirely random (when it comes to DKIM).

Whatever the actual reason, since its not the default and the reality 
the option exist and serves a purpose, there is an reasonable 
practical explanation there is a certain population of domains seeking 
the path of least resistance with reduced accidental <cr><lf> 
injections and mutations along the path as its very possible to occur 
in our heterogeneous networks of Unix (LF), MAC (CR) or DOS (CRLF) 
transport, gateways and storage I/O differences.

I think the main point in all this is whether any formulas can be 
developed or as you may state, use for developing standards.

I don't think there is anything reliable there from I can see, but its 
not unreasonable for one to hypothesize that there might be a direct 
correlation between the number of hops and the tendency to use 
relaxed/relaxed. It might be interesting to see if that may be a 
motivation for using relaxed/relaxed:

      c-param vs ave # of hops (received lines)

If this tibit turns out to be statistically true,  then I think the 
only thing we can say is that "we did our job" to provide a relaxed 
C14N method to lower the transport mutations issues for those domains 
who need it.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 18 May 2011 15:26:07 -0700

> Sent: Wednesday, May 18, 2011 1:49 PM
> To: IETF-DKIM

I think you're asking for a count of domains using various canonicalizations that produce spam.  Here's what we have:

+------------------------+-----------+------------+
| count(distinct domain) | hdr_canon | body_canon |
+------------------------+-----------+------------+
|                    214 |         0 |          0 |
|                      1 |         0 |          1 |
|                     62 |         1 |          0 |
|                   3805 |         1 |          1 |
+------------------------+-----------+------------+

This counts a domain as "spammy" if the mail we've seen signed by that domain is labeled as spam by Spamassassin at least 50% of the time, just as a starting point.  But if instead I report on less than 50% (relatively clean domains), the ratios are about the same:

+------------------------+-----------+------------+
| count(distinct domain) | hdr_canon | body_canon |
+------------------------+-----------+------------+
|                   2703 |         0 |          0 |
|                      6 |         0 |          1 |
|                   2238 |         1 |          0 |
|                  20573 |         1 |          1 |
+------------------------+-----------+------------+

So I don't think a conclusion's really possible here.

>       c-param vs ave # of hops (received lines)

+---------------------+-----------+------------+----------+
| avg(received_count) | hdr_canon | body_canon | count(*) |
+---------------------+-----------+------------+----------+
|              1.0976 |         0 |          0 |     2214 |
|              1.0000 |         0 |          1 |        7 |
|              1.0338 |         1 |          0 |     7569 |
|              2.3349 |         1 |          1 |    14086 |
+---------------------+-----------+------------+----------+

Canonicalizations of "0" mean "simple", "1" is "relaxed".  So there is possibly a correlation between use of relaxed/relaxed and the hop count for spam, but I have trouble envisioning that as something that's being actively considered by signers.

The same report for non-spam, however, shows that there's probably not much of a statistically significant difference:

+---------------------+-----------+------------+----------+
| avg(received_count) | hdr_canon | body_canon | count(*) |
+---------------------+-----------+------------+----------+
|              1.2570 |         0 |          0 |   220497 |
|              1.0971 |         0 |          1 |      412 |
|              1.4505 |         1 |          0 |   172136 |
|              2.0206 |         1 |          1 |   980337 |
+---------------------+-----------+------------+----------+

I don't know where all this is leading, but there you go.

-MSK

---

From: Hector Santos <[email-masked]>
Date: Wed, 18 May 2011 20:44:23 -0400

Murray S. Kucherawy wrote:
> Canonicalizations of "0" mean "simple", "1" is "relaxed".  So there 
> is possibly a correlation between use of relaxed/relaxed and the 
> hop count for spam, 

I just finished doing this test and got the following. I stored 
records (hops, hash, sdid) in a SQL tables and ran the following queries:

select hash, count(*) from c14n
      group by hash;

+--------------------------------+
| hash                  count(*) |
|--------------------------------|
| relaxed/relaxed       5420     |
| relaxed/simple        1115     |
| simple/relaxed        2        |
| simple/simple         1314     |
+--------------------------------+

select hash, hops, sdid, count(*) from c14n
      group by hops
      order by hops desc, hash;

+--------------------------------------------------------------+
| hash                hops    sdid                    count(*) |
|--------------------------------------------------------------|
| relaxed/relaxed     8       gmail.com               8        |
| relaxed/relaxed     7       talamasca.ocis.net      6        |
| relaxed/simple      6       mrochek.com             49       |
| relaxed/relaxed     5       yahoo.com               474      |
| relaxed/relaxed     4       gmail.com               184      |
| simple/simple       3       maimonides.edu          84       |
| relaxed/relaxed     2       coldwatercreek.com      1483     |
| relaxed/relaxed     1       facebookmail.com        5563     |
+--------------------------------------------------------------+

I had notice gmail.com messages had a wide degree of multi-hops, so I 
did a query just for it:

select hash, hops, sdid, count(*) from c14n
    where sdid="gmail.com"
    group by hops
    order by hops desc, hash;

+--------------------------------------------------------------+
| hash                hops     sdid                   count(*) |
|--------------------------------------------------------------|
| relaxed/relaxed     8        gmail.com              8        |
| relaxed/relaxed     7        gmail.com              4        |
| relaxed/relaxed     6        gmail.com              14       |
| relaxed/relaxed     5        gmail.com              14       |
| relaxed/relaxed     4        gmail.com              107      |
| relaxed/relaxed     2        gmail.com              130      |
+--------------------------------------------------------------+

Looking at these messages:

    hops=2   direct private emails to users
    hops=4   xml-dev list messages
    hops=5   pop3ext, ietf-smtp list messages
    hops=6   spf-help, ietf discuss list messages
    hops=7   spf-discuss list messages
    hops=8   spf-discuss list messages

> but I have trouble envisioning that as 
> something that's being actively considered by signers.

The reason we needed relaxed in the first place is because there are 
many long time systems that are still active and had evolved from UUCP 
(like us) and still have those backend internal I/O designs, including 
UI, report writers, text interfaces, etc, in place. The first change 
was just swapping the transport method UUCP to SMTP and the only 
interoperability requirement was to make sure the edge had the proper 
LF/CRLF interface translations in place.

Never an issue until DKIM came along. So for example, if the system 
backend storage is <LF>, you can imagine a standalone DKIM signer or 
verify utility needs to take this I/O into account when reading the 
file.  It can't assume that all mail storage is x822/5322 with CRLF 
delimiters.  We can state it but it is really none of anyone's 
business how the backend data is stored as long the end result is the 
same.

So are signers/operators aware mail mutations can happen?  I think so.

Are signers blasting 1 to Many messages "believe" they need a more 
relaxed integrity to maximize the DKIM verification across the many 
receivers?  I think so (although your stats are showing the similar 
passage rates for simple or relaxed).

I also think that if DKIM has a C14N option (i.e. STRIP) available to 
resolve legacy throughputs for particular streams, they will use it 
too maybe on per target basis only. :)

Anyway, thanks.

---

From: "John Levine" <[email-masked]>
Date: 19 May 2011 03:17:07 -0000

Since I more or less started this, my assertion was that relaxed doesn't
do much better than simple, which at this point I think we can categorize
as "not disproven."

The point I was making was that ever more complex ways to decide that
two mutated versions of a message are "the same" aren't likely to help
much, certainly not compared to the large cost of implementing code
even more complex than what relaxed does now.  And anyway, if your
goal is for your message to survive, you should armor it better, not
come up with more arcane ways to declare that it may be bleeding
heavily but it's not dead yet.

R's,
John

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 19 May 2011 10:21:08 +0000

> do much better than simple, which at this point I think we can categorize

Probably true, but if the difference between 10% broken and 8% broken signatures is independent of whether the email is spam, then actually "relaxed" seems to be producing a 20% reduction in signature breakage.

I'd argue that a 20% reduction in broken signatures *is* actually "much better". 

> even more complex than what relaxed does now.  

To determine that, we'd need a pareto analysis of breakage modes. Presumably lists that aren't re-signing are responsible for some of this, as are broken signing mechanisms. The questions remaining are, "is there anything left after excluding those two cases?", and "how much of that could be fixed easily?".

> And anyway, if your

---

From: Ian Eiloart <[email-masked]>
Date: Thu, 19 May 2011 11:00:10 +0000

> The contradiction is in the assertion that primarily spammers are using relaxed/relaxed, not the quantity of domains involved.

Right, but you can't address that by examining only the traffic from the high traffic domains. If the spammers are spreading across domains, then you won't see their traffic in that analysis.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 19 May 2011 10:14:27 -0700

> Sent: Thursday, May 19, 2011 4:00 AM
> To: Murray S. Kucherawy
> Right, but you can't address that by examining only the traffic from
> the high traffic domains. If the spammers are spreading across domains,
> then you won't see their traffic in that analysis.

But that's not what the report I ran produced.  There's no doubt spammers are spreading across domains, but their choice of canonicalization doesn't seem to make a difference in that regard.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 19 May 2011 10:19:56 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Ian Eiloart
> Sent: Thursday, May 19, 2011 3:21 AM
> To: John Levine
> Cc: <[email-masked]>
> Probably true, but if the difference between 10% broken and 8% broken
> signatures is independent of whether the email is spam, then actually
> "relaxed" seems to be producing a 20% reduction in signature breakage.
> 

That statistic is largely meaningless unless there's a basis for comparison.  For example, it would be useful to observe that the same message, sent with each of the four canonicalizations, to the same set of destinations using the same endpoint software, produced different results given that one changing variable.  But if domain X only ever tried sending with relaxed/relaxed and generally gets good results, there's nothing in that datum to say that simple/simple would not have worked just as well for the same sender with the same mail.  Thus I don't believe there's enough data to support your conclusion.

That relaxed/relaxed appears to survive 20% more might be based on the fact that the people using it send clean mail through clean paths, and it's as simple as that.

> To determine that, we'd need a pareto analysis of breakage modes.
> Presumably lists that aren't re-signing are responsible for some of
> this, as are broken signing mechanisms. The questions remaining are,
> "is there anything left after excluding those two cases?", and "how
> much of that could be fixed easily?".

Our stats are unable to tell what the problem is in all cases, but for mail we've received where the signer used the "z=" tag, the biggest signature breaker in terms of header changes is modified To: fields.  I suspect that's either rewriting by lists to add the list description as a comment, or improperly quoted comment fields that are corrected along the way.

---

From: Hector Santos <[email-masked]>
Date: Thu, 19 May 2011 13:32:57 -0400

Ian Eiloart wrote:

>> Levine was making a point:
>>

In a way, he was making a general reflection of Pareto Optimality.  It 
is Pareto Efficient when one thing takes up more resources without 
hurting others. It is not Pareto Efficient when the others demand more 
from the one thing that may alter the overall system efficiency :)

> Presumably lists that aren't re-signing are responsible for some of this, 
> as are broken signing mechanisms. The questions remaining are, "is there 
> anything left after excluding those two cases?", and "how much of that 
> could be fixed easily?".

Sadly, for many, unless one see the "oops" on their own system, it 
doesn't mean much for the rest - Pareto, Prisoner's Dilemma, Chaos 
Theory, Game Theorem, Unit Operations, etc.

Back to practical reality, here is my "basic" problem with DKIM:

Why are we doing it?  Why is the same message in two different 
streams, ok in one and not the other?  How should the user be 
"trained" to see author "PDQ Public" signed here but not signed there?

I started to use isdg.net to sign my mail for the IETF related list 
and I use the l= tag

  - PASS, IETF-DKIM, DKIM-aware MLM, 3rd party resigned,
          ATPS/ACL authorized

  - FAIL, IETF-SMTP, NON-DKIM-aware MLM, 1st party signed, no body change
          except extra top <CRLF>

  - PASS, IETF DISCUSS, NON-DKIM-aware MLM, 1st party signed, adds footer.

This WG consensus says the IETF-SMTP stream is unimportant and prefer 
to pass the buck to the software on that stream.  Could be an easy fix 
by the MLM but is the developer listening? If open source, who is 
making the change?

This it is a form of Pareto Analysis when one deems DKIM efficiency is 
reached and it includes an acceptable margin of error.  However, with 
this acceptable margin of error becomes an opening for exploitable 
with overwhelming receiver abuse, it weakens the overall DKIM efficacy 
to live in this growing indeterminate DKIM world.

DKIM will need to naturally evolve to target signing to reduce the 
inconsistent results in a 1 to Many environment.

Consider this:

Based on the way DKIM has been modeled in RFC4671bis, the most 
efficient optimization (lowest overhead) would be:

   - Before any HASH computation is done,

   - Extract all the DKIM-SIGNATURES and extract all the d=signer domain

   - Check to see if signer domain is in your local WHITELIST table or
     use some protocol shim, callout, "DNS WhiteListing" whatever.

   - If no signer is found, then no need to do anything else.

   - If one signer is found, then do the valid check.

   - If more than one signer is found, well, I guess only 1 is needed.

So in the same vain we interpret an calculated invalid signature:

    "No Valid Signature Exist!"

The trust failure interpretation is to says:

    "No Trusted Signature Exist!"

Why bother with all the redundant overhead waste with hash 
calculations, DNS public key lookups when the RFC4671 End Goal is to 
get a 3rd party certified trust?  If the signer is unknown, DKIM 
authenticity has no value.

---

From: Hector Santos <[email-masked]>
Date: Thu, 19 May 2011 14:58:43 -0400

Murray S. Kucherawy wrote:

> But that's not what the report I ran produced.  There's no doubt 
> spammers are spreading across domains, but their choice of 
> canonicalization doesn't seem to make a difference in that regard.

Quite possibly we can simply chalk it up by giving people the benefit 
of the doubt they are actually reading RFC4671 chapter 3.4 on 
canonicalization and they are simply looking for the DKIM path of 
least resistance.

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 22 May 2011 12:57:10 +0200

> even more complex than what relaxed does now.

Just to mention two of those ways, MIME rewriting is a capability
mayor MTAs introduced when MIME took root, HTML styles mangling is an
ongoing MUA exercise.

> And anyway, if your goal is for your message to survive, you should
> armor it better, not come up with more arcane ways to declare that
> it may be bleeding heavily but it's not dead yet.

Interesting, but not less intricate.  The semantics of authenticating
only the armored part of a message is not obvious.  Resorting to
base64 encoding is subject to varying interpretations, including
spammers attempts to avoid naive content filtering.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: "John R. Levine" <[email-masked]>
Date: 22 May 2011 10:44:59 -0400

>spammers attempts to avoid naive content filtering.

S/MIME and PGP MIME have been doing just that, authenticating just an
armored MIME body, for close to 20 years.  Your MUA probably has
support for S/MIME built in.  This is a wheel we do not need to
reinvent.

R's,
John

---

From: Hector Santos <[email-masked]>
Date: Sun, 22 May 2011 22:50:06 -0400

> spammers attempts to avoid naive content filtering.

+1.

What has made all this very difficult is trying to make a technology 
fundamentally based on authenticating integrity (DKIM) work with a 
natural breaking "Message Massager" (Mailing List) and as a result we 
have lowered its overall efficacy to an near absolute indeterminate 
state of conditions.  Even if the conversion is not required, you just 
don't know if BASE64 MIME is done by the signer and/or mailing list.

Case in point. Right now, I am looking at a gmail.com which I am 
pretty sure was not in Base64 MIME.  However, it was submitted to IETF 
DISCUSS group and its showing up as base64 MIME.  I don't know if the 
list did it one of the hops before the DKIM-Signature.

But the body hash failed.  So I took the message, made a copy and 
unbased64 the body, pulled the list footer and poof! - the body hash 
was ok again.

It would of been nice to have some DKIM-Signature flag that might 
indicate the Content-Transfer-Encoding, i.e.:

    et="base64"   <--- copy of the top level Content-Transfer-Encoding

then at least the verify will know the encoding type of original body. 
If et= value does not match the current Content-Transfer-Encoding, 
then it has some smarts do a unbase64 before hash verification.

Of course, it still would of failed due to the list footer so, but if 
l= was used, there is some avenue for success.

The main realization is that Sender/Signers need to be more aware of 
the target/path if they desire a higher rate of return.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 23 May 2011 10:21:29 +0100

>     et="base64"   <--- copy of the top level Content-Transfer-Encoding

Could you get the effect of this by including the  
Content-Transfer-Encoding header in the 'h=' and doing some fancy checks  
involving the 'bh=' (to detect whether it was the body or the headers or  
both that were broken)?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 23 May 2011 10:29:58 +0000

On 22 May 2011, at 15:44, John R. Levine wrote:

> support for S/MIME built in.  

It does. It tells me that your MIME signature was incorrect.

> This is a wheel we do not need to
> reinvent.
> 
> R's,
> John_______________________________________________

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 23 May 2011 10:11:14 +0000

> That statistic is largely meaningless unless there's a basis for comparison.  For example, it would be useful to observe that the same message, sent with each of the four canonicalizations, to the…

Well, nor do I really. I'm just pointing out that IF IF IF the 10% and 8% figures are real then it's the 10% and 8% that you need to be comparing (that's a 20% improvement) not the 90% and 92% (a trivial improvement).

However, someone else pointed out that spammers and non-spammers seem to have similar patterns of using simple and relaxed. If that's the case, then we've eliminated one potential problem with the data.

> That relaxed/relaxed appears to survive 20% more might be based on the fact that the people using it send clean mail through clean paths, and it's as simple as that.

True. Though I've found that maybe one third of messages that I see with a bad signature also carry a good signature. 

> Our stats are unable to tell what the problem is in all cases, but for mail we've received where the signer used the "z=" tag, the biggest signature breaker in terms of header changes is modified T…

For May, so far, I see these canonicalisations for good signatures:
206136 c=relaxed/relaxed
55748 c=relaxed/simple
   1 c=simple/relaxed
29207 c=simple/simple

I also see these problems:
5816 invalid - public key record (currently?) unavailable
	364 signing domains
3772 invalid - syntax error in public key record
	93 signing domains

And these for bad signatures:
22577 c=relaxed/relaxed (9.9%)
	but 6087 (27%) of these are from yahoogroups.com, and almost all also carry a good signature!
4065 c=relaxed/simple (6.7%)
   6 c=simple/relaxed 
3094 c=simple/simple (9.6%)

And these are the failure reasons:
simple/simple:
1992  body hash mismatch (body probably modified in transit)]
1102  signature did not verify (headers probably modified in transit)]

relaxed/simple:
2796  body hash mismatch (body probably modified in transit)]
1269  signature did not verify (headers probably modified in transit)]

relaxed/relaxed:
1824  body hash mismatch (body probably modified in transit)]
20753  signature did not verify (headers probably modified in transit)]

d=yahoogroups.com:
  9  body hash mismatch (body probably modified in transit)]
5949  signature did not verify (headers probably modified in transit)]
-- 
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 23 May 2011 10:25:50 +0000

Sorry, Pareto Efficiency (an economics concept) isn't something that I was familiar with. I was referring to an analysis Pareto charts (a quality engineering tool). Same guy, different concept.

A Pareto chart looks at the different causes of a problem (signature breakage) and determines the relative frequency of each cause. This helps you to determine the benefit (but not the cost) of addressing each cause.

My stats tell me that there's not much benefit in trying to persuade people to adopt relaxed versus simple. By and large, they're already doing that. For Exim users, that may be because relaxed is the software default.

But, they do suggest the header mismatches are 10 times more common than body mismatches, so it may be worth focussing here.

---

From: Hector Santos <[email-masked]>
Date: Mon, 23 May 2011 09:38:59 -0400

> involving the 'bh=' (to detect whether it was the body or the headers or  

Using the currently defined standard specification, the only way to do 
a reliable check is to add the z= header which contains the values of 
the h= header fields.  However, having z= is more for testing, can be 
exploited in some fashion and adds ugly DKIM-Signature bandwidth 
overhead.  So its not a good idea, I think, for everyday usage.

But since DKIM is "flexible" :),  adding maybe an et= tag should not 
break the system allowing for exploration to see if that could help 
lower failures for some streams.

In this test case with the gmail.com message submitted to a non-DKIM 
aware list, it would of succeeded knowing the original top level 
encoding type and using l= if destined to a list.

I think the other side point in my OP, is that it didn't seem it was 
necessary to base64 the message content.  I saw no 8 bit characters 
when I decoded it - just plain 7bit ascii.  So it seems some systems 
are always doing it regarding of 7/8 bit.

---

From: "John Levine" <[email-masked]>
Date: 23 May 2011 12:41:22 -0000

>involving the 'bh=' (to detect whether it was the body or the headers or  

No, of course not.  The bh= and h= are just hashes, so all they can
tell you is whether the text being hash has changed, not how the body
has changed or which header changed.  There's no separate hash for
individual header lines.

On the other hand, you might want to look at page 24 of RFC 4871.

R's,
John

---

From: Ian Eiloart <[email-masked]>
Date: Mon, 23 May 2011 15:51:17 +0000

On 23 May 2011, at 16:00, Hector Santos wrote:

> Actually, different guys (sciences, engineering disciplines), same concept. :)

Not according to http://en.wikipedia.org/wiki/Pareto which names Vilfredo Pareto as the originator of the concept of Pareto efficency, and says that the Pareto chart was also named after him. I guess the one connection between the two concepts is that a Pareto chart is most useful if the causes identified are independent, and that fixing one cause of error won't aggravate another.

---

From: Hector Santos <[email-masked]>
Date: Mon, 23 May 2011 11:00:00 -0400

Ian Eiloart wrote:
> Sorry, Pareto Efficiency (an economics concept) isn't something that 
> I was familiar with. I was referring to an analysis Pareto charts 
> (a quality engineering tool). Same guy, different concept.

Actually, different guys (sciences, engineering disciplines), same 
concept. :)

> A Pareto chart looks at the different causes of a problem (signature breakage) 
> and determines the relative frequency of each cause. This helps you to 
> determine the benefit (but not the cost) of addressing each cause.

And this provides an optimality. The purpose is to find an "condition" 
where it is Pareto Efficient (The best you can do without losing 
maximum benefit) because, like most things in life, there are always 
imperfections and most factors are interrelated in some fashion.  So 
while its a tool to help identify problems, benefits, weakest, 
critical points, etc, there is a cause and effects and Pareto 
Optimality always comes into play. In a closed environment you have 
boundary conditions, and how much of one thing is done to correct, 
tweak, turn a knob in a process control plant, it can have an effect 
the other things and most important, a change which can move you away 
from being Pareto Efficient (optimal).

> My stats tell me that there's not much benefit in trying to persuade 
> people to adopt relaxed versus simple. 

IMV, the specs should not be persuading other than explain what the 
purpose and reason for each one - let the reader decide.

> By and large, they're already doing that. For Exim users, that 
> may be because relaxed is the software default.

Agree, defaults is definitely a contributor especially for new 
deployments starting out.

> But, they do suggest the header mismatches are 10 times more 
> common than body mismatches, so it may be worth focussing here. 

But in what way?   We already have relaxed C14N for headers.  I 
proposed the STRIP method in 2005 and that resolved the CRLF related 
issues I saw back then.

Back in July 2005, I proposed the bh= among other diagnostic items for 
a "DKIM-Signature-Debug" debugging header of the header/body C14N thing:

    http://www.imc.org/ietf-mailsig/mail-archive/msg01817.html

The bh= was added to the August 2006 base-01 draft.  After the bh= 
hash is checked, its impossible to tell where the DKIM failure occurs 
afterwards (hash wise).  The only extra debugging item you have it 
adding z= to see if any header value was changed.

Personally, based on what I see, the signer should consider reducing 
the headers its signed and focus on the persistent core header never 
expected to change and even within the core headers, it definitely 
should avoid signing headers that are guarantee to change based on a 
known path it will take - like for an MKM, consider not hashing the 
5322.Subject tag and use "l=" when the target path is known to be a 
list adding a footer.

So with the Pareto Chart, we can include MLM and target/path as two of 
the items.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 23 May 2011 10:26:52 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of John Levine
> Sent: Monday, May 23, 2011 5:41 AM
> Cc: [email-masked]

If one were to encode somehow an extension indication that "this content was subjected to 8-to-7 downgrade" as a hint that a verifier should do the reverse before verifying, the verifier would have to manage to undo the downgrade in precisely, i.e. byte-for-byte, the same manner that the downgrade was done for it to work.  That's a pretty high requirement for interoperability (i.e., it's pretty error-prone), so it requires a specification and it would need to be consistent with the MIME RFCs.

So assuming it's a useful endeavour, it seems to me there's a lot of work to be done here.  Our latest report says Content-Transfer-Encoding changes aren't even in the top ten header field changes invalidating signatures, which means there aren't that many downgrades happening, so this probably isn't a significant problem right now.  I don't really think this is worth the effort.

It also seems to me it would be easier to either do what the RFC says and downgrade before signing, or perhaps if you're super-keen about it, generate a signature for the 8-bit form and the downgraded form; if no downgrade occurs, the 8-bit form will pass, and if one does, the 7-bit form will pass (all other things being equal).  But then there are those people that want to penalize invalid signatures despite the advice of the RFC, so you'd have that to deal with.

---

From: "John R. Levine" <[email-masked]>
Date: 23 May 2011 14:14:50 -0400

> If one were to encode somehow an extension indication that "this content was subjected to 8-to-7 downgrade" as a hint that a verifier should do the reverse before verifying, the verifier would have…

Seems to me that if someone were that desperate to get a signed message 
through a downgraded path, they should wrap the whole thing in a 
base64 encoded message/rfc822 mime part and send it that way.

This all strikes me as mostly hypothetical, and unlikely to affect more 
than a tiny sliver of mail.

The EAI group, which has way more experience with character set issues and 
downgrades than we do, tried all sorts of downgrade experiments and 
decided that none of them were workable. The current nearly final draft 
says that if you want to send an EAI message, you better find a path to 
the recipient that can deliver it as is.  Perhaps we should take the hint.

Regards,

---

From: Michael Deutschmann <[email-masked]>
Date: Mon, 23 May 2011 13:38:16 -0700 (PDT)

> base64 encoded message/rfc822 mime part and send it that way.

That's explicitly forbidden by RFC 2046.  Some idiot lazy MUA writer
sought a guarantee that his MUA would be able to look inside
forwarded-as-attachment mail without having to handle multiple layers of
encoding.  He got it.  Thus, no multipart or message/rfc822 object can be
base64 or QP encoded.

Because of that arrogant decision, a downconversion for a message
containing a message/rfc822 element under 8bit CTE (which *is* legal),
simply isn't defined.

---- Michael Deutschmann <[email-masked]>
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 24 May 2011 17:33:00 -0700

On 5/23/2011 10:26 AM, Murray S. Kucherawy wrote:
> If one were to encode somehow an extension indication that "this content was
> subjected to 8-to-7 downgrade" as a hint that a verifier should do the
> reverse before verifying, the verifier would have to manage to undo the
> downgrade in precisely, i.e. byte-for-byte, the same manner that the
> downgrade was done for it to work.  That's a pretty high requirement for
> interoperability (i.e., it's pretty error-prone), so it requires a
> specification and it would need to be consistent with the MIME RFCs.
>
> So assuming it's a useful endeavour, it seems to me there's a lot of work to
> be done here.

Let's make it be the right work.

To make a canonicalization algorithm that is more robust -- such as having it 
based on canonical forms of data, independent of encoding -- makes some sense. 
Trying to create the ability to "reverse" changes strikes me as far to complex 
and fragile to be reasonable.

d/
-- 

   Dave Crocker

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 24 May 2011 23:19:43 -0700

> From: Dave CROCKER [mailto:[email-masked]]
> Sent: Tuesday, May 24, 2011 5:33 PM
> To: Murray S. Kucherawy

I think your last sentence is what I was getting at.

One could have a 7-bit canonicalization that does the same sort of thing MIME does, I suppose, at least in terms of content conversion.  Not having given it much thought yet, though, I suspect doing this at the MIME level is more likely to be successful rather than at the message level, which is where DKIM lives.

I could swear I heard that idea presented recently...

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 24 May 2011 23:38:07 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Charles Lindsey
> Sent: Monday, May 23, 2011 2:21 AM
> To: DKIM

I don't think so, because a re-encoding causes a change to Content-Transfer-Encoding itself, which in turn changes both hashes, not just one of them.

You could use an extension tag to capture the original Content-Transfer-Encoding as a hint to the canonical form that was signed, but that means the verifier has to undo the conversion before computing the hashes, and it has to do that bytewise precisely as well as reconstructing the original MIME header fields.  Getting that even a byte off (modulo "relaxed") means you're toast.

At that point I suspect you may as well bite the bullet and start down the road of defining a new canonicalization that accommodates possible downgrades in transit, picking either 7-bit or 8-bit as the canonical form, and feeding that form to the hash to be used in signature generation.  But once you consider that a multipart can have some 7-bit and some 8-bit, this can get real messy real fast.

Once this becomes an actual problem, I imagine MIMEAUTH will start to get even more interesting.

---

From: Hector Santos <[email-masked]>
Date: Wed, 25 May 2011 04:03:55 -0400

Murray S. Kucherawy wrote:

> You could use an extension tag to capture the original Content-Transfer-Encoding 
> as a hint to the canonical form that was signed, but that means the verifier 
> has to undo the conversion before computing the hashes, and it has to do that 
> bytewise precisely as well as reconstructing the original MIME header fields.  
> Getting that even a byte off (modulo "relaxed") means you're toast.

The idea was to only do it for certain situations, i.e. when converted 
to base64.  Since a unbase64 function is already in your DKIM 
repertoire, no extra library is require and it would be one additional 
conditional statement.

> At that point I suspect you may as well bite the bullet and start down the 
> road of defining a new canonicalization that accommodates possible 
> downgrades in transit, picking either 7-bit or 8-bit as the canonical form, 
> and feeding that form to the hash to be used in signature generation.  But 
> once you consider that a multipart can have some 7-bit and some 8-bit, this 
> can get real messy real fast.

+1 (on the last sentence)

How would 7/8 bit be considered?

Personally, the STRIP C14N idea would work just fine by removing all 
trailing WSP (CR, LF, SP) and for QP text, decode it first.  I'm 
considering updating my 2006 I-D to include the QP decoding logic.

This, along with the et= idea, it would address at least three 
additional cases I have encountered.

    - MLM that add an extra line at the top (possibly because of any 
empty
      header file of size 2, crlf)
    - Intermediaries that expand QP to 8 bit
    - Intermediaries that reformat to BASE64

I personally have not seen anything else.

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 25 May 2011 12:39:04 +0200

> considering updating my 2006 I-D to include the QP decoding logic.

I propose a much more radical approach, something that will likely
land on the too-loose side.  Such kind of approach is justified by the
"most breakage is innocent" theory, and by already having two
canonicalizations on the too-tight side.

For example, consider these criteria for feeding the body hash:

1) For multipart MIME messages, completely remove the preamble, the
epilogue, and all boundaries and entity headers.

2) For MIME encoded parts, get back to the binary content.

3) For text parts, completely remove /any/ whitespace.  Additionally,
remove most punctuation, especially from begin and end of lines.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Hector Santos <[email-masked]>
Date: Wed, 25 May 2011 08:27:32 -0400

> remove most punctuation, especially from begin and end of lines.

Do we really need this?  Do you know of cases related to this?

We should identify and list the sort of transformation issues we are 
seeing. I have identified four so far:

  1 - We forgot a possible top CRLF. We dealt with the bottom <CRLF>
      but not the top,

  2 - Top level QP decoding,

  3 - Top Level reformatting to C-T-E: base64 (not MIME multi-part)

  4 - Lines over 998 (1000 with CRLF), this is an invalid RFC5322, but
      its possible some verifiers are designed to do a buffered C14N
      and don't check for RFC5322 line lengths between two memory points
      in the buffer as oppose as a line by line feed into the C14N
      function.  Why buffer vs line?  speed.

      I found 98 such buffer hash errors from various domains due to
      having at least 1 super long line.

   Domains with Illegal RFC5322 text
+-------------------------------------+
| count(*)   sdid                     |
|-------------------------------------|
| 78         coldwatercreek.com       |
| 4          jcprewards.com           |
| 4          news.redlobster.com      |
| 2          livingsocial.com         |
| 2          resultsmail.com          |
| 2          trl3.net                 |
| 2          xanthianoutswagger.net   |
| 1          numbersoft.info          |
+-------------------------------------+

and I just found a yet another problem which I was currently 
investigating to see where it this "mite" is occurring:

  5 - Incorrect handling of lines beginning with dots, for example
      I sent a message containing a line beginning with:

      ... blah blah blah.  blah.

      and it was received by my SMTP server as:

      .... blah blah blah.  blah.

      and its sent to this list, it comes back as:

      ..... blah blah blah.  blah.

I checked my SMTP server and its correct. I just now pretty much 
confirmed ThunderBird is causing this initial dot escaping error by 
sending mail to my gmail host and hotmail host accounts. The "...." 
showed up in my new mail bin. But it also appears there could be an 
intermediary with a bug as well adding yet another dot.

While we like to shift blame to the buggy software, IMV, since 
transport DOT escaping is a SMTP requirement and there could be these 
small issues, I'm thinking maybe we should consider a similar 
"relaxed" logic now done for reducing multiple <WSP> to reduce 
multiple <DOT> as well but only when its begins on a new line (or for 
block transfers, preceded by <CR><LF>).  So a line like

      <DOT><DOT><DOT><SP>blah<SP>blah<DOT><SP><SP>blah<DOT>

it becomes:

      <DOT><SP>blah<SP>blah<SP>blah<DOT>

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 25 May 2011 17:33:28 +0200

> Do we really need this?  Do you know of cases related to this?

The idea is to anticipate any unknown signature breaker.  My three
points above are rather generic.  They are meant to be expanded so as
to include your five points below, and more.

I think it is quite obvious that MIME rewriting generates new
boundaries, and may alter an entity's header.

Non-text binary content that arrives corrupted deserves breaking a
signature.  However, a rewriter may decode a base64 entity for local
storage, and then re-encode it with a different line length.

Text undergoes any kind of massage, trailing "=" may be leftover,
CRLFs may be doubled, "From " turned into ">From ", besides the
leading dots you mention in point five.

> seeing.

Erring on the too-lose side implies some generalization.

> I have identified four so far:
> 

I imagined the C14N function reads characters one by one.  On finding
CRLF it can go back a few bytes to remove end-of-line punctuation.
However you code c14n(), it will be sparklingly faster than sha256().

However, distinguishing begin middle of line versus begin/end is
possibly inconsistent, since line breaks may be altered because of
invalidly long lines or RFC3676 rewrapping.

>       having at least 1 super long line.

Some MUAs consistently keeps paragraphs on a single line.

>       <DOT><SP>blah<SP>blah<SP>blah<DOT>

Yes, dot is one of the punctuation characters that should be removed.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Hector Santos <[email-masked]>
Date: Wed, 25 May 2011 12:42:38 -0400

> leading dots you mention in point five.

Why not just DKIM Signer ALWAYS  base64 a MIME message/rfc822, sign it 
and thats it? :)

> However you code c14n(), it will be sparklingly faster than sha256().

The code can be written to do the C14N first, then do a time hash 
call, or you can do it on the fly C14N/HASH.  I use memory file maps 
to deal any potential large mail and the line parsing doesn't hurt speed.

> Yes, dot is one of the punctuation characters that should be removed.

This turned out to be a bug in our beta code, revamped to support I/O 
completion ports and the code for undotting of the leading dot (per 
RFC5321 4.5.2) fell thru the crack.  So we can nix this one. :)

---

From: "John Levine" <[email-masked]>
Date: 25 May 2011 16:59:29 -0000

>The idea is to anticipate any unknown signature breaker.

I'm pretty sure that's specifically out of scope.

And I promise that whatever you do, short of wrapping the whole
message in opaque armor, I can come up with something that will
break it.

Regards,

---

From: "MH Michael Hammer (5304)" <[email-masked]>
Date: Wed, 25 May 2011 13:28:18 -0400

> From: [email-masked] [mailto:ietf-dkim-
> [email-masked]] On Behalf Of John Levine
> Sent: Wednesday, May 25, 2011 12:59 PM
> Cc: [email-masked]

+1

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 25 May 2011 11:23:44 -0700

> message in opaque armor, I can come up with something that will

One might have a goal of attempting to be robust against all forms of potential 
breakage.

That's not likely to be the goal of this sort of exercise.  Rather, it will be 
to choose a set of particular types of breakage, ignoring others.  For an effort 
like that, it is not meaningful to come up with additional types of breakage, 
since there is no attempt to cover such additional examples.

d/

-- 

   Dave Crocker

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 26 May 2011 10:22:56 +0100

> and fragile to be reasonable.

Sure. But this thread is simply trying to see whether some minimal  
assistance from the signer might be useful for helping any geek who was  
mad enough to try reversing the changes. For sure there will such exist  
geeks out there somewhere - but I don't expect such reversing to become  
normal standard practice.

---

From: Hector Santos <[email-masked]>
Date: Thu, 26 May 2011 06:03:58 -0400

> Some MUAs consistently keeps paragraphs on a single line.

Or rather on the WRITER side, if set, and they can use:

   - use QP to send it (not necessarily save it as QP on the local
     user side).

   - use automatic word wrapping based on the width set (usually
     < 70 to stay consistent with console terminal types).

But when not, on the READER side, as you know, reading can be 
difficult under the display device that doesn't word wraps it for you. 
   This is often a design problem for WEB based viewing because it may 
depend on the HTML tag used to display it.

    <pre> preformatted - viewer will display as is.
    <p>   paragraph - viewer will word wrap

When I read this list mail, for example, via the archive on the web, 
some participant's mail are not word wrapped. So I often just copy and 
paste it into my editor, hit ALT-B to word wrap it just so I can read 
it.   This is an old problem where people assumed (or didn't) everyone 
is using the same reading devices.

In our WEB mail viewer, I forget the logic but it has been adjusted 
over time, and "security" is part of it.   The best way (under web) to 
do it is to use multipart/alternative with text/plain and text/html to 
allow reading devices to be smart.

When you made your suggestion, I thought you and I were thinking the 
same thing in regards to the C14N/HASH should be on what the user 
"wrote" and what is "read," i.e. the actual context, with all the 
"color" and formatting removed.  I think the idea has theoretical 
merit but it is definitely extra processing on both ends (signer and 
verifier) so it may not be feasible.

Just wish to make a few other points regarding the larger buffer readers.

     Messages with illegal (length) text lines
+------------------------------------------------+
| signer                    total  illegal  hops |
|------------------------------------------------|
| coldwatercreek.com        78     78       2    |
| livingsocial.com          41     2        1    |
| jcprewards.com            23     4        2    |
| news.redlobster.com       11     4        1    |
| xanthianoutswagger.net    2      2        1    |
| resultsmail.com           2      2        1    |
| trl3.net                  2      2        1    |
| numbersoft.info           1      1        1    |
+------------------------------------------------+

all spam.  Obviously there are a lot of spammers and eMarketers that 
believe that it (illegal lines) is not checked, and for practical 
reasons, it isn't, or it isn't done to a latter point.

Most SMTP server use a larger buffer data blocks to read the DATA 
stream.  Reading this line by line is extremely inefficient and poor 
TCP and networking performance. It can be the difference between a 2 
second upload to a 1 minute upload.   It wouldn't be hard to do a line 
length check between two <CRLF> points, but for us, this (illegal 
lines) wasn't detected until DKIM checking was added.

So if there anything positive about DKIM is that its helping system do 
things they probably never bother (didn't need to or want to) check 
before. The multiple From: header was among them.   The main point is 
that there are many systems with a transport mechanism that really 
don't care what is in the payload. But RFC5322 compliancy is becoming 
a more aspect to take into account, and including line lengths should 
be among them.

---

From: Hector Santos <[email-masked]>
Date: Thu, 26 May 2011 06:26:09 -0400

Charles Lindsey wrote:
>> To make a canonicalization algorithm that is more robust -- such as  
>> having it based on canonical forms of data, independent of encoding -- 
>> makes some  sense.  Trying to create the ability to "reverse" changes 
>> strikes me as far to complex and fragile to be reasonable.

> normal standard practice.

Sure, but its not just being a geek but simply understanding the 
environment and problems that exist.  I think most people will want to 
know the "Why" this or that happens when you do complex work with a 
presumption and natural human expectation that the yields will be 
correct side of things.

Most good and experienced engineers (or any discipline in general) 
have insights into potential problems without having to see it really 
happen.  If you can predict something, it can often serve as a 
preemption solution.  Good Practical Engineers tend to look at the 
feasibility aspects of a solution. The ideas I proposed (STRIP, "et=" 
encoding tag) are ones I believe MAY be a feasible solution for a few 
of these inadvertent incorrect body hash cases. I am 100% aware that 
it may not be necessary or feasible in the overall practical sense. 
  But its not just thinking out of the box today, this was a long time 
issue (since 2005) and in my view, it was already a bad idea to try to 
make DKIM work within LIST environments that could only work in a 
fundamentally changed framework highlighting all the legacy hiccups of 
the system.

Levine's answer is to forget what you receive, just resign on the way 
out.  That is not unreasonable, but its not a feasible and secured 
solution neither, and ultimately, it also comes with a change 
requirement.

At the end of the day, no matter what, people software (Old and New) 
need to change to make DKIM work better.

---

From: Hector Santos <[email-masked]>
Date: Thu, 26 May 2011 09:22:43 -0400

Dave CROCKER wrote:
> One might have a goal of attempting to be robust against all forms
> of potential breakage.
> 
> That's not likely to be the goal of this sort of exercise.  Rather, it 
> will be to choose a set of particular types of breakage, ignoring others.  
> For an effort like that, it is not meaningful to come up with additional 
> types of breakage, since there is no attempt to cover such additional 
> examples.
> 

Dave,

While one reasonably understand the statement for feasibility, I have 
trouble with the vexing conflict exhibited when picking your battles 
yet a particular skirmish (i.e. a particular mail stream) is not part 
of the general solution.

For example, IETF-SMTP, when 100% of signing domain participating in 
such a list constantly fails due to minute transparent changes, and it 
includes your own domains, one tends to be more interested in looking 
for a solution because it will water down the branding of your domains.

IOW, the idea of mixed results does not help DKIM when it all falls 
under the same category.  The list that does not change anything about 
the message except adds a footer is the principle reason that gives 
life to the "l=" tag and for domains to use it for a list known to add 
only a footer.

For targeted list addresses I am a member of, I have my signer setup to:

    - use "l="
    - do not sign the "Subject:" header

So we did our job here.  The only problem is the list that adds a 
<CRLF> to the top of the message.

Sure, we are presuming this is a BUG and most likely is, but it could 
be a simple matter that the list operator had an EMPTY TopHeader file 
but has 2 bytes only in it <CRLF>.  So we don't know what's actually 
going on. But rest assured, we (well, I did) did not pay attention to 
the extra TOP <CRLF>, only the extra BOTTOM <CRLF> potential.

I see that you are not signing your mail for IETF-SMTP.  I am and as 
well as others.  Why is that not generally important? It is actually 
one of the simpler C14N issues to deal with.  It may be minor, but it 
still an issue for a LIST that DKIM mail passes thru. I feel should be 
part of a DKIM C14N consideration and also an MLM awareness issue, not 
necessary for my sake but for DKIM sake.

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 27 May 2011 19:09:09 +0200

> since there is no attempt to cover such additional examples.

Of course, a signature cannot survive a deliberate attempt at breaking
it.  However, earlier analysis considered man-in-the-middle attacks
like changing, e.g., "Amoeba yeast" into "Amo ebay east" [Bryan
Costales, Thu, 04 Aug 2005].  We don't know how likely such kind of
attacks may be, since only tight canonicalizations were standardized.

By introducing a loose canonicalization we may learn whether signature
survivability affects DKIM adoption.  If wider usage introduces
attacks, we can switch back to current canonicalizations --in case
downgrades will have gone away-- or design yet another one,
approaching just the tightness we need.  My appeal is for not imposing
monotonicity to successive approximations, and allow erring on the
too-lose side as well.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 27 May 2011 19:11:30 +0200

> RFC5321 4.5.2) fell thru the crack.  So we can nix this one. :)

Yet, it wouldn't have hurt if those signatures had passed.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 27 May 2011 10:16:38 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Alessandro Vesely
> Sent: Friday, May 27, 2011 10:09 AM

So what, for example, would you do differently?  The unfortunate thing about the way the crypto works is that you get a failure, but you don't know for sure what changed other than "it was in the header" or "it was in the body".  "z=" sometimes gives you details about the former but it's not in widespread use.

I'm all for including experimental code in future releases of our stuff, especially if it's an experiment other implementations are trying.  But I need to see a spec first, or enough detail that I could write one.

---

From: "John R. Levine" <[email-masked]>
Date: 27 May 2011 14:05:16 -0400

> survivability affects DKIM adoption.

Feel free to do some experiments.  One of the reasons that DKIM has had 
relatively few implementation surprises is that we already knew how DK 
worked.

Regards,

---

From: Hector Santos <[email-masked]>
Date: Fri, 27 May 2011 16:39:42 -0400

> relatively few implementation surprises is that we already knew how DK 

Overall, I would like to agree, but then again, there are still 
problems, including in how DK worked and we still coming up with 
surprises.

The issues we come across and how we deal with it, is par for the 
course for how most things are considered important or not - Who it 
happens to (the messenger), is a very important factor in all this.

---

From: Hector Santos <[email-masked]>
Date: Sat, 28 May 2011 08:05:15 -0400

> monotonicity to successive approximations, and allow erring on the

It seems the DKIMnomics are proving not to be there. It's like beating 
a dead horse dealing with the inconsistent considerations.  With so 
much failure with your own domain usages, it has become a big concern.

Its not a big deal when your own domain failures come back into your 
own system (via list distribution) because you can make the 
exceptions. But that knowledge is not known by others seeing the same 
message.  Its not like the world has "SPF-like" exposed information or 
3rd party vouching information for DKIM receivers, or users to at 
least to get a clue.

The irony is the seeing a high validation success among the unknown 
spammers w/o using unknown list to distribute their junk.  Since we 
separated the "question" between the resigner and the purported 
author, where are the benefits?  Your own domains are lost on list 
that resign as well on those that don't resign.

Seeing the following does not give one much hope:

+-----------------------------------------------------------------------+
| LIST-ID                                   SIGNER               % FAIL |
|-----------------------------------------------------------------------|
| 6522.31.email.enviomasivoempresarial.com  esp2.it                 0   |
|-----------------------------------------------------------------------|
| 8388.19.h3h8.s05.it                       emailingtools.it        0   |
| 8388.23.h3h8.s05.it                       emailingtools.it        0   |
| 8388.40.h3h8.s05.it                       emailingtools.it        0   |
| 8388.51.h3h8.s05.it                       emailingtools.it        0   |
| 8974.19.h9g4.s06.it                       emailingtools.it        0   |
|-----------------------------------------------------------------------|
| Developers.winserver.com                  mapurdy.com.au        100   |
|-----------------------------------------------------------------------|
| dkim-ops.mipassoc.org                     mipassoc.org            0   |
|-----------------------------------------------------------------------|
| ietf-822.imc.org                          cybernothing.org      100   |
| ietf-822.imc.org                          tana.it               100   |
| ietf-822.imc.org                          mrochek.com           100   |
| ietf-822.imc.org                          isdg.net              100   |
| ietf-822.imc.org                          messagingengine.com   100   |
|-----------------------------------------------------------------------|
| ietf-dkim.mipassoc.org                    mipassoc.org          0.2   |
|-----------------------------------------------------------------------|
| ietf-pop3ext.imc.org                      gmail.com             100   |
|-----------------------------------------------------------------------|
| ietf-smtp.imc.org                         santronics.com        100   |
| ietf-smtp.imc.org                         tana.it               100   |
| ietf-smtp.imc.org                         gmail.com             100   |
| ietf-smtp.imc.org                         sonnection.nl         100   |
| ietf-smtp.imc.org                         taugh.com             100   |
| ietf-smtp.imc.org                         resistor.net          100   |
| ietf-smtp.imc.org                         mrochek.com           100   |
| ietf-smtp.imc.org                         messagingengine.com   100   |
|-----------------------------------------------------------------------|
| ietf.ietf.org                             resistor.net          100   |
| ietf.ietf.org                             cybernothing.org      100   |
| ietf.ietf.org                             cisco.com             100   |
| ietf.ietf.org                             mrochek.com           100   |
| ietf.ietf.org                             iecc.com              100   |
| ietf.ietf.org                             isdg.net              100   |
| ietf.ietf.org                             gmail.com             100   |
|-----------------------------------------------------------------------|
| pak-business-26.googlegroups.com          googlegroups.com        0   |
| pak-business-29.googlegroups.com          googlegroups.com        0   |
| pak-business-60.googlegroups.com          googlegroups.com        0   |
|-----------------------------------------------------------------------|
| rss-public.yahoogroups.com                yahoogroups.com         0   |
|-----------------------------------------------------------------------|
| [email-masked]                   talamasca.ocis.net    100   |
| [email-masked]                   gmail.com             100   |
| [email-masked]                   listbox.com             0   |
|-----------------------------------------------------------------------|
| [email-masked]                      gmail.com             100   |
| [email-masked]                      listbox.com             0   |
|-----------------------------------------------------------------------|
| WINServer.santronics.com                  megabytecoffee.com    100   |
| WINServer.santronics.com                  mapurdy.com.au        100   |
| WINServer.winserver.com                   megabytecoffee.com    100   |
| WINServer.winserver.com                   gmail.com             100   |
| WINServer.winserver.com                   mapurdy.com.au        100   |
+-----------------------------------------------------------------------+

All my signing domain, isdg.net, fails on all the above list and you 
can see others in the same list.   Its interesting to note, like on 
the spf-discuss and spf-help list, only its own signature survive, but 
not others. For our winserver.com and santronics.com list, everyone 
else signature fails.

Overall, its not about just how it all looks on your own system or 
MUA, but how it looks at other systems.

So what is DKIM for?  For the unknown?  Giving these Valid Signature 
SPAMMERS some sort of seal of approval?  I don't think so.

+-----------------------------------------------------------------------+
| LIST-ID                                   SIGNER               % FAIL |
|-----------------------------------------------------------------------|
| 6522.31.email.enviomasivoempresarial.com  esp2.it                 0   |
|-----------------------------------------------------------------------|
| 8388.19.h3h8.s05.it                       emailingtools.it        0   |
| 8388.23.h3h8.s05.it                       emailingtools.it        0   |
| 8388.40.h3h8.s05.it                       emailingtools.it        0   |
| 8388.51.h3h8.s05.it                       emailingtools.it        0   |
| 8974.19.h9g4.s06.it                       emailingtools.it        0   |
|-----------------------------------------------------------------------|
| pak-business-26.googlegroups.com          googlegroups.com        0   |
| pak-business-29.googlegroups.com          googlegroups.com        0   |
| pak-business-60.googlegroups.com          googlegroups.com        0   |
|-----------------------------------------------------------------------|
| rss-public.yahoogroups.com                yahoogroups.com         0   |
+-----------------------------------------------------------------------+

As far as I am concern the highest benefit DKIM can provide is an 
explicit POLICY of failure protection.  That is where the benefits 
comes - when there is a violation of a given practice. But MLM doesn't 
work with DKIM with or without policy.  So where is the DKIMnomics?

--
HLS

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 28 May 2011 18:29:23 +0200

On 27/May/11 19:16, Murray S. Kucherawy wrote:
> I'm all for including experimental code in future releases of our
> stuff, especially if it's an experiment other implementations are
> trying.  But I need to see a spec first, or enough detail that I
> could write one.

For the body, I brought some ideas[1].  For MIME header fields,
punctuation and boundaries need to be omitted as well.  For other
header fields, including the DKIM-Signature, it is probably enough to
remove just any white space.
[1] http://mipassoc.org/pipermail/ietf-dkim/2011q2/016692.html

IMHO, the "hard parts" of the code are (i) selecting a MIME parser,
and (ii) finding a good way to structure experimental C14Ns and handle
double (triple?) signatures in the existing code.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Hector Santos <[email-masked]>
Date: Sun, 29 May 2011 22:47:53 -0400

Hi Alessandro,

Can you come up with a before and after example?  An example showing a 
snippet of what would be the original MIME message text and what would 
be final stripping down C14N version?

--
HLS

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 29 May 2011 21:04:38 -0700

> Sent: Saturday, May 28, 2011 9:29 AM

One of the elegant things about the current canonicalizations is that they can stream.  I think a system that's MIME-aware can too, but possibly not, and in any case having to teach a DKIM implementation about MIME will make it a lot more complicated and expensive.  If we have to go down that road, I think working on DOSETA and MIMEAUTH is the way to go.

If we want the lower-hanging fruits, we might take the list of things MLMs like to do to messages and find ways to canonicalize those.  Fortunately, we made a list of the common ones in the MLM document.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 30 May 2011 10:21:49 +0100

> double (triple?) signatures in the existing code.

For a Mime Parser, see http://www.cs.man.ac.uk/~chl/uncode/uncode.html.

Feel free to experiment and improve.

---

From: Steve Atkins <[email-masked]>
Date: Mon, 30 May 2011 09:14:26 -0700

> One of the elegant things about the current canonicalizations is that they can stream.  I think a system that's MIME-aware can too, but possibly not, and in any case having to teach a DKIM implemen…

Streaming MIME parsers are possible, but they're not simple. Also, MIME is more complex than most people think it is, and even coming up with a canonicalization specification that is both MIME aware and does what the author intended is likely to be quite difficult.

> If we want the lower-hanging fruits, we might take the list of things MLMs like to do to messages and find ways to canonicalize those.  Fortunately, we made a list of the common ones in the MLM doc…

Even that seems to be adding complexity simply for the sake of adding complexity, rather than because there's an obvious benefit.

The most obvious thing that MLMs do that invalidate signatures are 1. append content to the body and 2. prepend content to the subject line. Any approach that allows me to replay messages while making those changes seems to open the door to abuse.

Cheers,
  Steve

---

From: Hector Santos <[email-masked]>
Date: Mon, 30 May 2011 15:12:14 -0400

Steve Atkins wrote:

> The most obvious thing that MLMs do that invalidate signatures are 1. append 
> content to the body and 2. prepend content to the subject line. 

+1

> Any approach that allows me to replay messages while making those changes 
> seems to open the door to abuse.

However, we need to see if we can address survivability first before 
we can try to mitigate this.  I'm concern not just about what I see 
but what is everyone else going to see. We know what is going on.  The 
rest are not expected to realize the details.

Of the list I am personally involved with and beginning to sign mail, 
I see 100% body hash fail, but when the extra <CRLF> top line is 
considered, there is a significant improvement.

+----------------------------------------------------------------+
| LIST-ID                 SIGNER                FAILS   CRLF-FIX |
|----------------------------------------------------------------|
| ietf-822.imc.org        cybernothing.org      100%    100%     |
| ietf-822.imc.org        messagingengine.com   100%     61%     |
| ietf-822.imc.org        tana.it               100%      0%     |
| ietf-822.imc.org        mrochek.com           100%      0%     |
| ietf-822.imc.org        ISDG.NET              100%      0%     |
|----------------------------------------------------------------|
| ietf-smtp.imc.org       resistor.net          100%    100%     |
| ietf-smtp.imc.org       messagingengine.com   100%     71%     |
| ietf-smtp.imc.org       gmail.com             100%     58%     |
| ietf-smtp.imc.org       SANTRONICS.COM        100%     11%     |
| ietf-smtp.imc.org       tana.it               100%      0%     |
| ietf-smtp.imc.org       sonnection.nl         100%      0%     |
| ietf-smtp.imc.org       taugh.com             100%      0%     |
| ietf-smtp.imc.org       mrochek.com           100%      0%     |
|----------------------------------------------------------------|
| ietf.ietf.org           resistor.net          100%    100%     |
| ietf.ietf.org           cybernothing.org      100%    100%     |
| ietf.ietf.org           cisco.com             100%    100%     |
| ietf.ietf.org           mrochek.com           100%    100%     |
| ietf.ietf.org           qualcomm.com          100%    100%     |
| ietf.ietf.org           iecc.com              100%    100%     |
| ietf.ietf.org           gmail.com             100%    100%     |
| ietf.ietf.org           ISDG.NET              100%      0%     |
+----------------------------------------------------------------+

The domains in caps are mine. I'm signing with l= and not signing the 
subject line, except for santronics.com in IETF-SMTP and the 89% 
survival are with replies after the subject line has been altered. 
But you see other domains survive because the first two list do not 
add footers. The last one does and only "l=" people will survive that 
(after the CRLF adjustment).

For example, Ned with mrochek.com, all fails but fixed with the <CRLF> 
adjustment in the first two list. But since he doesn't sign with "l=", 
he still fails the ietf.ietf.org (IETF Discuss) list which does add a 
footer.

Overall, I agree with you that the most common MLM changes are the 
subject [list-name] tag prefix and the added footer.  We need to deal 
with that, for sure, as a highlighted signer recommendation targeting 
list mail. But as the table above shows, without the <CRLF> fix it 
doesn't matter.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 30 May 2011 15:23:20 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Steve Atkins
> Sent: Monday, May 30, 2011 9:14 AM
> To: DKIM List
> The most obvious thing that MLMs do that invalidate signatures are 1.
> append content to the body and 2. prepend content to the subject line.
> Any approach that allows me to replay messages while making those
> changes seems to open the door to abuse.

Agree on all counts.  And I talked to the Mailman people, for example, about a modified header canonicalization that deals with Subject: tagging, and they also agreed it wouldn't help that much since that's not the most common change made that would invalidate the signatures.

So as far as I can tell, we're at a point where lots of people think they want MLM survivability of signatures, or at least the chain-of-trust capability, but no proof that the increased risk is worth the increased gain.

---

From: "John R. Levine" <[email-masked]>
Date: 30 May 2011 18:44:23 -0400

> So as far as I can tell, we're at a point where lots of people think they want MLM survivability of signatures, or at least the chain-of-trust capability, but no proof that the increased risk is wo…

I would quibble with the word "lots".  Perhaps "a few highly vocal."

Put me in the camp that says there's no problem that's come up in 40 years 
of MLMs that this would solve, and in the unlikely event that it actually 
were a problem, signing an A-R header would work lot better, since it 
includes a signature from the MLM, which is what we really want.

To the claim that there are MLMs that won't do that, if I count the number 
of MLMs in the world vs. the number of sending and receiving mail hosts, 
upgrading all the MLMs is a whole lot more likely than upgrading all of 
the mail hosts.

Regards,

---

From: Steve Atkins <[email-masked]>
Date: Mon, 30 May 2011 23:14:10 -0700

> Agree on all counts.  And I talked to the Mailman people, for example, about a modified header canonicalization that deals with Subject: tagging, and they also agreed it wouldn't help that much sin…

Yup, that too.

> So as far as I can tell, we're at a point where lots of people think they want MLM survivability of signatures,

Maybe. If you can't name ten, there aren't lots.

> or at least the chain-of-trust capability, but no proof that the increased risk is worth the increased gain.

Chain of trust is a somewhat different thing, and could likely be implemented with little, if any, increased risk in the case where the MLM is trusted (for some meaning of the word that probably boils down to manual whitelist or positive reputation of the MLM operator) by the recipient.

The MLM signing the re-sent message, including an A-R header or some slight variant, is the obvious way. I don't think there's much gain to be had there, but it can be done with little effort and little risk.

Cheers,
  Steve

---

From: Dave Crocker <[email-masked]>
Date: Tue, 31 May 2011 08:13:17 -0700

Steve Atkins <[email-masked]> wrote:

>> or at least the chain-of-trust capability, but no proof that the
>increased risk is worth the increased gain.
>>
>Chain of trust is a somewhat different thing, and could likely be
>implemented with little, if any, increased risk in the case where the
>MLM is trusted (for some meaning of the word that probably boils down
>to manual whitelist or positive reputation of the MLM operator) by the
>recipient.
>
>The MLM signing the re-sent message, including an A-R header or some
>slight variant, is the obvious way. I don't think there's much gain to
>be had there, but it can be done with little effort and little risk.

Chain of trust is always an appealing model.  Unfortunately, it hasn't been used successfully over the open Internet.  The closest we are coming to having an example of its working is DNSSec, which actually has a very, very constrained model and relatively short chain.  It's utility as a demonstration of success is also very new.  It's not a 'complete' example.

There is a tendency to believe that operational changes are preferred over protocol changes.  That's essentially the difference between formulatng a model of trusting the sequence of message handlers, versus devising an authentication technique that survives the sequence of handlers.

Unfortunately, operational changes for security tend to make a more fragile model.

d/

---

From: "John R. Levine" <[email-masked]>
Date: 31 May 2011 11:28:24 -0400

> Chain of trust is always an appealing model.  Unfortunately, it hasn't 
> been used successfully over the open Internet.

I agree with your doubts about the utility of chain of trust, but I would 
have to say that SSL signed web sites are used successfully over the open 
Internet.

Regards,

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 31 May 2011 18:43:44 +0200

On 31/May/11 00:23, Murray S. Kucherawy wrote:
>> From:  On Behalf Of Steve Atkins
>> 

While that's true for MLM, I'm not sure it correctly reflects MTAs'
behaviors.  In particular, the X-MIME-AUTOCONVERT feature and whatever
may cause MIME rewriting.  This is MTA-specific, and affects MLMs as
well as dot-forwards.

Pareto has been discussed enough, so I don't comment on the fact that
such minor part of the traffic would demand complicated and expensive
implementations to go through correctly.

> Agree on all counts.  And I talked to the Mailman people, for
> example, about a modified header canonicalization that deals with
> Subject: tagging, and they also agreed it wouldn't help that much
> since that's not the most common change made that would invalidate
> the signatures.

Yeah, reply messages have subject-tags already in place.  If MLM
subscriptions were known at submission time, tag addition before
signing could be easily done by MSAs, MUAs, or manually by users.