Subject: Re: [dmarc-ietf] A policy for direct mail flows only, was ARC questions

From: Alessandro Vesely <[email-masked]>
Date: Tue, 1 Dec 2020 11:37:54 +0100

On Tue 01/Dec/2020 05:56:46 +0100 Brandon Long wrote:
> On Thu, Nov 26, 2020 at 12:59 AM Alessandro Vesely <[email-masked]> wrote:
> 
>> On 25/11/2020 20:16, Michael Thomas wrote:
>>> On 11/25/20 11:11 AM, Alessandro Vesely wrote:
>>>> On 25/11/2020 19:24, Jesse Thompson wrote:
>>>>> On 11/25/20 11:30 AM, Alessandro Vesely wrote:
>>>>>> Without resorting to ARC, it is still possible to validate author
>>>>>> domain's signatures directly if the MLM just adds a subject tag
>>>>>> and a footer>>>>>
>>>>> I agree that ARC isn't really needed to do this (trust the last hop
>>>>> from the MLM and determine the original authenticity from the MLM's
>>>>> perspective)>>>>
>>>> I didn't mean to trust the MLM.  I meant remove the subject tag and
>>>> the footer, then the original DKIM signature verifies.  See:
>>>> https://datatracker.ietf.org/doc/draft-vesely-dmarc-mlm-transform/
>>>
>>> When I was at Cisco, with l= and some subject line heuristics I could get
>>> probably like 90+% verification rate across the entire company, a company that
>>> uses external mailing lists a lot. Definitely not 100% though.
>>
>>
>> DKIM itself is not 100%.  You always have lines beginning with "From " or
>> occasional autoconversions.
>>
>> l= doesn't cover multipart/alternative nor Content-Transfer-Encoding: 
>> base64. In addition, the DKIM spec discourages its usage and suggests
>> that "Assessors might wish to ignore signatures that use the tag.">>
> 
> Right, some of the other dkim-light or diff concepts we discussed would be
> better than using l=
> 
> We again got hung up on the 100% solution, though... something that handled 
> subject-prefix and footer in a transport agnostic way might have worked.

I'm not clear about the meaning of "100%".  If an author domain puts no DKIM signatures, there is no way to verify them.  Hence, some compliance of the author domain has to be required.

The same holds for conditional signatures.

The same holds for MLM transformations.

> The fact that DKIM isn't transport agnostic is an achilles heel to even
> that, though, since we'd have to come up with a new canonicalization and get
> it to widespread adoption before the simple diff could work.

As DKIM works in the vast majority of cases (yes, not 100%, but nearly), the idea to discombobulate it for the sake of a meagre set of old-fashioned individuals who still dislike mass social media is a showstopper.  Yet, since DMARC standardization itself avails of mailing lists, we may want to get some coherence.

> Or require mailing lists to be a lot more strict in how they do their email
> rewriting, but I imagine that's harder work than even ARC.

They are quite strict already.  For example, I received the message I'm replying to in both my top and dmarc folders.  Both copies of the message have the correct From:.  The MLM copy was authenticated like so:

Authentication-Results: wmail.tana.it;
  spf=pass smtp.mailfrom=ietf.org;
  dkim=pass reason="Original-From: transformed" (whitelisted) header.d=google.com;
  dkim=pass (whitelisted) header.d=ietf.org
    header.b=zA0RRbSR;
  dkim=fail (signature verification failed, whitelisted) header.d=ietf.org
    header.b=F65GxYqF

From: restoring is documented here:
https://www.tana.it/sw/zdkimfilter/zdkimfilter.html#mlmtrans

Best
Ale
--

---

From: Michael Thomas <[email-masked]>
Date: Tue, 1 Dec 2020 10:07:27 -0800

On 11/30/20 8:56 PM, Brandon Long wrote:
>
>
> On Thu, Nov 26, 2020 at 12:59 AM Alessandro Vesely <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     l= doesn't cover multipart/alternative nor
>     Content-Transfer-Encoding: base64.
>     In addition, the DKIM spec discourages its usage and suggests that
>     "Assessors
>     might wish to ignore signatures that use the tag."
>
Nobody said it can fix everything. It's just that those things are in 
the long tail. And that admonition sounds like it crept in there after 
rfc4871 when i wasn't paying attention. i find that a ridiculous 
overreaction and begs the question why.

Being able recover 90%+ signatures going through mailing lists made our 
scheme of spear-phish alerting very close to viable. We never got to the 
point of having to make that call because there was just too many crufty 
email servers in the company that didn't use the centralized email path 
to insure their messages were signed. That had nothing to do with 
whether our scheme could work or not.

>
> Right, some of the other dkim-light or diff concepts we discussed 
> would be better than using l=
>
> We again got hung up on the 100% solution, though... something that 
> handled subject-prefix and
> footer in a transport agnostic way might have worked. The fact that 
> DKIM isn't transport agnostic
> is an achilles heel to even that, though, since we'd have to come up 
> with a new canonicalization
> and get it to widespread adoption before the simple diff could work.  
> Or require mailing lists to
> be a lot more strict in how they do their email rewriting, but I 
> imagine that's harder work than
> even ARC.
>
>

Frankly all it would take is a google or another large mail provider to 
publicly state that unless a mailing list supports BCP XYZ, your mail 
will be subject to very strict scrutiny and likely not delivered to get 
the attention of mailing list providers. That was my suggestion back in 
the day but it was scoffed at because people could point to some edge 
case that generates .001% of list traffic and thus invalidating the 
entire approach. The best is definitely the enemy of the good here.

People really need to keep in mind that service provider email is not 
the only game in town. That point keeps getting lost.

Mike

---

From: Brandon Long <[email-masked]>
Date: Tue, 1 Dec 2020 18:14:46 -0800

On Tue, Dec 1, 2020 at 2:37 AM Alessandro Vesely <[email-masked]> wrote:

> handled
> I'm not clear about the meaning of "100%".  If an author domain puts no
> DKIM signatures, there is no way to verify them.  Hence, some compliance of
> the author domain has to be required.
>

Yes, by 100% I meant of messages that were already authenticated and
therefore should continue to be authenticated through
the relay.

Some of the conditional signatures of the "include a diff you can remove to
validate the original" attempt seemed to
fail on the theory that there were too many things that couldn't be
handled.  Ie, if your relay removes attachments,
including them back in a diff kind of breaks the whole point of that... but
how common is that (even less now with
Yahoo Groups gone, but possibly still some av/malware relays still do this).

I think that one issue we've had is that DMARC is very mechanical and
straight-forward, so anything that's fuzzy
in response seems more complicated.

Anyways, what I'm saying is that I agree, we may have discarded things that
might have been much easier/simpler
for the 90% case due to wanting something that worked for everything.

> As DKIM works in the vast majority of cases (yes, not 100%, but nearly),
> the idea to discombobulate it for the sake of a meagre set of old-fashioned
> individuals who still dislike mass social media is a showstopper.  Yet,
> since DMARC standardization itself avails of mailing lists, we may want to
> get some coherence.
>

There are decided benefits to DKIM being MIME/body unaware, I was merely
pointing
out that there are things we could do if it was not and part of why some of
these alternatives were not pursued.

> They are quite strict already.  For example, I received the message I'm
> replying to in both my top and dmarc folders.  Both copies of the message
> have the correct From:.  The MLM copy was authenticated like so:
>

That's cool, I thought when I looked at the mailman code (admittedly a long
time
ago) that it didn't really concern itself with that level of maintaining
the format
of the message.  Unfortunately, the Google Groups code isn't that strict
either,
though we could have fixed that for a lot of cases (there's always the
challenge
of inserting a footer that requires changing the charset or cte).

Brandon

---

From: Brandon Long <[email-masked]>
Date: Tue, 1 Dec 2020 18:21:51 -0800

On Tue, Dec 1, 2020 at 10:07 AM Michael Thomas <[email-masked]> wrote:

> footer in a transport agnostic way might have worked.  The fact that DKIM
> isn't transport agnostic
> is an achilles heel to even that, though, since we'd have to come up with
> a new canonicalization
> and get it to widespread adoption before the simple diff could work.  Or
> require mailing lists to
> be a lot more strict in how they do their email rewriting, but I imagine
> that's harder work than
> even ARC.
>
> publicly state that unless a mailing list supports BCP XYZ, your mail will
> be subject to very strict scrutiny and likely not delivered to get the
> attention of mailing list providers. That was my suggestion back in the day
> but it was scoffed at because people could point to some edge case that
> generates .001% of list traffic and thus invalidating the entire approach.
> The best is definitely the enemy of the good here.
>
> People really need to keep in mind that service provider email is not the
> only game in town. That point keeps getting lost.
>
arguably we're all here because a large mail provider did make such a
change (though to be fair, there were plenty of others who wanted to make
that change).

While Google might be able to help move things along, there would need to
be strong community support for that, no one wants to go this alone and
look like the big bully.

I also think that you're overestimating what we could do.  Ultimately, we
serve our customers, and they want their legitimate email, even if it
doesn't support BCP XYZ.

Brandon

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 2 Dec 2020 12:11:31 +0100

On Wed 02/Dec/2020 03:14:46 +0100 Brandon Long wrote:
> therefore should continue to be authenticated through the relay.

That's ARC.  If a message lacks DKIM and was SPF-authenticated, there's no way 
it can continue to be authenticated through a relay.

OTOH, mailing lists and relays are two different beasts.  For one thing, it is 
very unusual for a mailing list to send to another mailing list.  Thus, we can 
safely specify a non-stackable authentication method.

> validate the original" attempt seemed to fail on the theory that there were
> too many things that couldn't be handled.  Ie, if your relay removes
> attachments, including them back in a diff kind of breaks the whole point of
> that... but how common is that (even less now with Yahoo Groups gone, but
> possibly still some av/malware relays still do this).

Not to mention anonymous lists, which remove the OP identity completely.  They 
are DMARC-proof by themselves, with no additional twists.  My draft restricts 
footers to text/plain MIME type, to overcome the objection to l=.  Hence, if a 
list appends HTML parts (e.g. to use <hr>), it doesn't qualify as DMARC-proof.

> straight-forward, so anything that's fuzzy in response seems more
> complicated.

It may seem fuzzy, but it's not.  The ietf list ([email-masked]), for example, 
adds no subject tag and no footer.  DKIM signatures should remain valid, then. 
  Yet, if posters sign Sender:, they fail.  I wouldn't call that fuzziness.  It 
is the very nature of the spec.  If you sign Received:, no relay can hold your 
signature over.

Best
Ale
--

---

From: [email-masked]
Date: Tue, 1 Dec 2020 14:27:04 +0100

Le 01/12/2020 à 11:37, Alessandro Vesely a écrit :
> 
> [...] a meagre set of old-fashioned individuals who still dislike mass social media [...]

Can decisions please be made based on sound technical reasons rather
than intolerance and zealotry?

Setting aside the form of your argument: no, contracting with Facebook
et al should not be a prerequisite for using the Internet for any
purpose. By the very definition of "open network".

Cheers,
BC

---

From: Michael Thomas <[email-masked]>
Date: Wed, 2 Dec 2020 11:29:19 -0800

On 12/1/20 6:21 PM, Brandon Long wrote:
>
>
> On Tue, Dec 1, 2020 at 10:07 AM Michael Thomas <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>>     We again got hung up on the 100% solution, though... something
>>     that handled subject-prefix and
>>     footer in a transport agnostic way might have worked.  The fact
>>     that DKIM isn't transport agnostic
>>     is an achilles heel to even that, though, since we'd have to come
>>     up with a new canonicalization
>>     and get it to widespread adoption before the simple diff could
>>     work.  Or require mailing lists to
>     Frankly all it would take is a google or another large mail
>     provider to publicly state that unless a mailing list supports BCP
>     XYZ, your mail will be subject to very strict scrutiny and likely
>     not delivered to get the attention of mailing list providers. That
>     was my suggestion back in the day but it was scoffed at because
>     people could point to some edge case that generates .001% of list
>     traffic and thus invalidating the entire approach. The best is
>     definitely the enemy of the good here.
>
>     People really need to keep in mind that service provider email is
>     not the only game in town. That point keeps getting lost.
>
> change (though to be fair, there were plenty of others who wanted to 
> make that change).
>
> While Google might be able to help move things along, there would need 
> to be strong community support for that, no one wants to go this alone 
> and look like the big bully.
>
> I also think that you're overestimating what we could do. Ultimately, 
> we serve our customers, and they want their legitimate email, even if 
> it doesn't support BCP XYZ.
>

Well obviously the BCP would have to come first and there would have to 
be community buy-in to create such a BCP, but a Google participating in 
creating such a BCP ought pique the interests of people who make mailing 
list software who obviously have the biggest stake in this.

It occurs to me that it might not even be a BCP. Maybe mailing lists can 
just create a header with sed commands to undo its changes :)

Mike

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 3 Dec 2020 12:05:37 +0100

> than intolerance and zealotry?

That was not meant to prompt the making of any decision.  It is a fact that 
mailing lists are not among the most popular communication tools.

> purpose. By the very definition of "open network".

I agree with that "should".

Best
Ale
--

---

From: Brandon Long <[email-masked]>
Date: Fri, 4 Dec 2020 14:45:47 -0800

> safely specify a non-stackable authentication method.

mailing list to mailing list mail is very common in GSuite, but maybe we're
a special
case.  Part of that was that aliases were moved to the mailing list infra
(which is definitely
more of an "our problem" than everyone), but we also see common usage where
companies make groups matching their hierarchy (the all company group goes
to
all department groups which go to all local groups etc).  We also see some
where
announce groups mix contrib groups, etc.  Especially since we also use the
same
groups as ACL groups for GSuite and Google Cloud, so hierarchical groups
are very
common.

(there was a recent common where someone said who needs an org chart when we
have email, showing the list of footers for all of the org level mailing
lists the message
went to to get to them)

(one construction of hierarchical announce lists that was not well
considered did result
in every person on the combined list getting 9000 copies of messages to the
list, so
flattening lists is sometimes much preferred)

Which isn't to say we couldn't work around it or do the work to flatten the
lists, though
it increases the complexity of the product (can you unsubscribe from the
top level list
but not a sub list?)

Brandon

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 5 Dec 2020 14:20:55 +0100

On Fri 04/Dec/2020 23:45:47 +0100 Brandon Long wrote:
>>>>>>>>>> Without resorting to ARC, it is still possible to validate
>>>>>>>>>> author domain's signatures directly if the MLM just adds a
>>>>>>>>>> subject tag and a footer >>>>>>>>>
>>>>>>>>> I agree that ARC isn't really needed to do this (trust the
>>>>>>>>> last hop from the MLM and determine the original
>>>>>>>>> authenticity from the MLM's perspective) >>>>>>>>
>>>>>>>> I didn't mean to trust the MLM.  I meant remove the subject
>>>>>>>> tag and the footer, then the original DKIM signature verifies.
>>>>>>>> See:
>>>>>>>> https://datatracker.ietf.org/doc/draft-vesely-dmarc-mlm-transform/ >>>>>>>
>>>>>>> When I was at Cisco, with l= and some subject line heuristics I
>>>>>>> could get probably like 90+% verification rate across the entire
>>>>>>> company, a company that uses external mailing lists a lot.
>>>>>>> Definitely not 100% though. >>>>>>
>>>>>> DKIM itself is not 100%.  You always have lines beginning with
>>>>>> "From " or occasional autoconversions. >>>>>>
>>>>>> Content-Transfer-Encoding: base64. In addition, the DKIM spec
>>>>>> discourages its usage and suggests that "Assessors might wish to
>>>>>> ignore signatures that use the tag." >>>>>
>>>>> would be better than using l= >>>>>
>>>>> handled subject-prefix and footer in a transport agnostic way might
>>>>> have worked. >>>>
>>>> I'm not clear about the meaning of "100%".  If an author domain puts
>>>> no DKIM signatures, there is no way to verify them.  Hence, some
>>>> compliance of the author domain has to be required. >>>>
>>> therefore should continue to be authenticated through the relay. >>
>> That's ARC.  If a message lacks DKIM and was SPF-authenticated, there's
>> no way it can continue to be authenticated through a relay. >>
>> OTOH, mailing lists and relays are two different beasts.  For one thing,
>> it is very unusual for a mailing list to send to another mailing list.
>> Thus, we can safely specify a non-stackable authentication method. >
> a special case.  Part of that was that aliases were moved to the mailing 
> list infra (which is definitely more of an "our problem" than everyone), but 
> we also see common usage where companies make groups matching their 
> hierarchy (the all company group goes to all department groups which go to 
> all local groups etc).  We also see some where announce groups mix contrib 
> groups, etc.  Especially since we also use the same groups as ACL groups for 
> GSuite and Google Cloud, so hierarchical groups are very common.

I'm not familiar with that feature.  How does a list get subscribed to another 
list?

In general, a list accepts messages from subscribed users, so if dmarc-ietf 
distributed this message to another list which I'm not a subscriber of, it 
would not pass.  (From: rewriting changes that behavior.  For example, one 
could subscribe to a list which routinely rewrites From: on behalf of this 
list, dmarc-ietf.  If the former list traffic is very high, that subscribing 
could be a sort of DoS attack to us.)

> (there was a recent common where someone said who needs an org chart when 
> we have email, showing the list of footers for all of the org level mailing 
> lists the message went to to get to them)

The mlm-transform draft (referenced above) limits footers to 10 lines of text, 
each shorter than 80 characters.  If the hierarchy is not deeper than 10,  the 
chart can fit within a single footer.  Note that every added line breaks all 
previous signatures.  However, only the author domain's signature is worth 
being recovered, as far as MLM transformations are concerned.  ARC can still 
certify the whole chain.

The 10 lines limit is arbitrary.  It is meant to avoid that recipients mistake 
the footer for the main part of the message.

> considered did result in every person on the combined list getting 9000 
> copies of messages to the list, so flattening lists is sometimes much 
> preferred) >
> lists, though it increases the complexity of the product (can you 
> unsubscribe from the top level list but not a sub list?)

I guess some coordination among sublists is required anyway.

To recognize that a message arrives from a sublist, and hence mangle the 
existing footer instead of adding a new one, is certainly an added complexity. 
  Maybe, the number of lists in such condition is low enough to let MLM 
transformation reversion still worth being deployed?

Best
Ale
--

---

From: "John Levine" <[email-masked]>
Date: 5 Dec 2020 13:29:29 -0500

In article <CABa8R6spYyZDhzEnQ=[email-masked]> you write:
>a special case. 

I dunno how special that case is, but there are lots of cases where mail passes
through multiple layers of ARC signing mutations.

I routinely get mail from Microsoft's farm with an ARC seal or two
that has never been near a mailing list. Any time a MS user sends a
message to one of my lists, it'll emerge with at least two ARC
signatures.

R's,
John

---

From: Michael Thomas <[email-masked]>
Date: Sat, 5 Dec 2020 10:37:37 -0800

> message to one of my lists, it'll emerge with at least two ARC

Getting multiple signatures from the originating domain doesn't hurt 
anything. It's a little wasteful, but that's it.

Mike

---

From: "John Levine" <[email-masked]>
Date: 5 Dec 2020 15:56:07 -0500

>anything. It's a little wasteful, but that's it.

a) The point of ARC seals is that the cv=pass in seal N tells you that
the signature in seal N-1 was good when the message arrived, so you
can do your filtering based on the state of the message each time it
was modified. DKIM doesn't do that.

2) Last week someone was complaining about the expense of the
signatures in ARC seals, now multiple signatures don't hurt anything.
While I agree with the latter sentiment, what changed?

R's,
John

---

From: Michael Thomas <[email-masked]>
Date: Sat, 5 Dec 2020 13:04:39 -0800

> While I agree with the latter sentiment, what changed?
It means that you can't control somebody else's infrastructure. We can 
control whether standards are created carefully and thoughtfully.

Mike

---

From: Brandon Long <[email-masked]>
Date: Mon, 7 Dec 2020 17:36:41 -0800

> I'm not familiar with that feature.  How does a list get subscribed to

Either by direct add via the groups interface, via carefully
crafted +subscribe emails,
or via our directory API:
https://developers.google.com/admin-sdk/directory/v1/guides/manage-group-members

The consumer version of groups tries to not let other groups subscribe by
looking at the
subscribing domain, but the workspace version doesn't care.

> could be a sort of DoS attack to us.)

Yes, having lists subscribed to lists creates all sorts of complications.

> I guess some coordination among sublists is required anyway.

Note that sub-lists may not be from the same list manager, ie one could
subscribe alias type lists to lists at other domains.

> transformation reversion still worth being deployed?

I think something like mlm-transform is not a 100% solution, though how
useful would
depend on whether it's a 50% or 99% solution.  Frankly, I think some of the
other restrictions
might eliminate more email, but we'd have to look at the numbers.

Brandon