Subject: Re: [dmarc-ietf] same old org domain, Topic for IETF 112 - Policy Discovery

From: Alessandro Vesely <[email-masked]>
Date: Mon, 1 Nov 2021 20:17:05 +0100

On Sun 31/Oct/2021 16:27:14 +0100 Scott Kitterman wrote:
> On October 31, 2021 11:29:41 AM UTC, Alessandro Vesely <[email-masked]> wrote:
>> On Sat 30/Oct/2021 22:56:00 +0200 Scott Kitterman wrote:
>>> On October 30, 2021 8:47:51 PM UTC, John Levine <[email-masked]> wrote:
>>>> According to Scott Kitterman  <[email-masked]>:
>>>>>> That usage has proven to work quite well.  And some respect for the installed
>>>>>> base wouldn't hurt.
>>>>>
>>>>> The alternative I suggested is 100% compatible with the installed base.
>>>>> If a domain has published DMARC policy per RFC 7489, the proposed new
>>>>> approach will still find it.
>>
>> Yes, but would PSL-based DMARC filters have to be re-written, re-tested,
>> re-installed?
> 
> I don't think so.  Because, modulo the backwards compatibility note mentioned below (which I think is primarily a theoretical concern), existing implementations would find policies published under the new design, they wouldn't need to be changed to support this new policy discovery definition.

Since policy publishing is 100% compatible, so is designing it.  An organization shouldn't publish an intermediate record just to "help" DMARC filters.

>> Another criterion, beside tree-walk and PSL, could be to look at the d= tag of
>> the DKIM signatures that are aligned with the From: domain.  Would that be
>> semantically equivalent to the procedure described in the current Section 6.7.2?
> 
> Sorry, 6.7 doesn't have numbered sub- paragraphs.  Did you mean 6.6.2?

No and yes, I meant https://datatracker.ietf.org/doc/html/draft-ietf-dmarc-dmarcbis-03#section-6.7.2

> Once you dig into 6.6.2 you end up at 3.1 identifier alignment.  For strict alignment, it works unchanged.  For relaxed, there's a potential for an issue.
> 
> Since since both 3.1.1 (DKIM) and 3.1.2 (SPF) define relaxed based on a common org domain, we would have to come up with a new definition.
> 
> We can't, under the new design, use the policy domain as a substitute for org domain since that would potentially allow unrelated entities to generate aligned results if records are published above the current org domain level (as a totally hypothetical example, messages could be sent signed by state.gov From maryland.gov and they would align).

That might happen only because they're under an "experimental" public suffix domain.

The old spec obviously assumes that the policy domain and the From: domain are aligned.

> I think that if we changed the relaxed definition to the same as or a sub-domain of the From domain it would avoid potential issues like that without practical impact.  I don't think I have ever seen legitimate mail where Mail From or DKIM signing domain wasn't either the same or a sub-domain of From that were in the same org domain.

From: [email-masked], signed by example.org which also publishes a policy has to be valid.  We have to work out the PSD case.

>>> I think it would be appropriate to have some kind of note warning not to assume intermediate domains are queried for policy due to legacy code
>>
>> Agreed.  Intermediate domains, the ones between the From: domain and its
>> Organizational Domain, would be considered like some kind of proxy, or even a
>> more specific alternative to the Organizational Domain or the possible PSD.
>> Still, checking each and every intermediate domain may remain optional.
> 
> As a practical matter, it would be optional since no one is likely to publish records for those domains, but we wouldn't specify it that way.  It would be a local optimization an implementer might choose.

We can say that any policy found at the From: domain or any parent thereof is valid, even at TLD.

>> The concept of Organizational Domain is still useful for receivers, as it helps
>> setting up reputation databases.  In this respect, the PSL is also useful
>> outside the DMARC protocol; for example, to get the organizational domain of
>> HELO arguments.
> 
> Perhaps, but if it's not DMARC related, it's out of scope for our consideration.  In any case, none of those uses require commonality to support interoperability, they're all local functions.  I don't think we need any RFC definition to enable those uses.

Having a useful concept defined by an RFC is a Good Thing.

Best
Ale
--

---

From: Joe Humphreys <[email-masked]>
Date: Mon, 1 Nov 2021 15:30:13 -0400

Scott, your proposal says to remove any reference to organizational domain,
but you don't explain how you will define DKIM and SPF alignment without
reference to organizational domain.

 In your proposal, if the query at step 1 does return a record, and the
record specifies relaxed alignment, then the receiver does not have enough
information to evaluate alignment.

This is not an academic point. The organization I work for relies on
relaxed alignment, and we sometimes create DMARC records for subdomains
(usually because we've identified a subdomain that we're not yet ready to
apply p=none to). I don't see how this will work under your proposal.

Regards,
Joe Humphreys

On Sun, Oct 31, 2021 at 10:46 PM Scott Kitterman <[email-masked]>
wrote:

>
>
> On November 1, 2021 2:19:02 AM UTC, Douglas Foster <
> [email-masked]> wrote:
> >Currently, the Publicssuffix.org list protects the PSL names.   Although I
> >don't believe it is covered anywhere in the DMARC v1 document, any DMARC
> >implementer will have to notice that a PSL name must produce DMARC FAIL,
> >because it cannot be authenticated itself, and it cannot be authenticated
> >by alignment with any other domain name.
> >
> >If we could publish a rule which limits alignment to only work from an
> >authenticated domain name downward to a child subdomain, we would have a
> >secure design, because we would not need to worry about leaving the
> >authenticated organization.   But as long as upward alignment is
> >necessitated by current practice, we need to be concerned about leaving
> the
> >authenticated organization in the upward direction into the PSL.
> >
> >A specific example:
> >MailFrom is "[email-masked]" and From is "trustme@com".
> >Any acceptable replacement for the PSL must ensure that this identifier
> >configuration is rejected.
> >
> >When moving up the domain tree, the only way to avoid transiting from an
> >authenticated organization into the PSL, is to know what names are in the
> >PSL.    The alternatives seem to be (a) an externally obtained list, no
> >matter how imperfect, or (b) DNS entries, no matter how imperfect.    The
> >list seems the best option in the near term, but the DNS option might
> prove
> >valuable over time.
>
> What's wrong with what I already proposed?
>
> Your analysis is backwards.  Modulo the few PSDs that have published
> records for PSD DMARC and for receivers that check for it, the DMARC
> results for mail to such domains is none, not fail.
>
> Scott K
>
> _______________________________________________
> dmarc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dmarc
>

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 01 Nov 2021 16:24:00 -0400

On Monday, November 1, 2021 3:17:05 PM EDT Alessandro Vesely wrote:
> From: [email-masked], signed by example.org which also publishes a
> policy has to be valid.

Why?  Do you know of this construct being used in any significant way?

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 01 Nov 2021 16:26:07 -0400

That was addressed in Message-ID: 
<[email-masked]>.

Scott K

---

From: Joe Humphreys <[email-masked]>
Date: Mon, 1 Nov 2021 18:03:06 -0400

Sorry, I missed that. But I disagree with this statement:

I think that if we changed the relaxed definition to the same as or a
sub-domain of the From domain it would avoid potential issues like that
without practical impact.  I don't think I have ever seen legitimate mail
where Mail From or DKIM signing domain wasn't either the same or a
sub-domain of From that were in the same org domain.

We send tens of millions of such messages daily. These are messages where
the From address is [email-masked], and the DKIM
signing domain is just organization.com.

I suggest again that the simple answer is for the DMARC record itself to
specify the organizational domain. This is orthogonal to how you discover
the DMARC record.

Regards,
Joe Humphreys

On Mon, Nov 1, 2021 at 4:26 PM Scott Kitterman <[email-masked]> wrote:

> > > >Currently, the Publicssuffix.org list protects the PSL names.
>  Although
> > > >I

---

From: Tobias Herkula <[email-masked]>
Date: Mon, 1 Nov 2021 22:07:46 +0000

Yes this is used in a significant way, dropping the mechanic of the org-domain would make a lot of things in processing inbound mail streams a lot more complicated.

The PSL does not exists for DKIM or DMARC, it is a product of the CAB forum. And the idea was borrowed for DMARC, but without it, DMARC will have a hard time, and depending standards as well. I don't want to discuss how good or bad BIMI is, but without an "org-domain" it doesn't work. But if DMARC as one of the base requirements for BIMI drops the "org-domain" mechanic, you really need to produce a better alternative than, simply stating that things that are currently OK to do, are not used by enough entities and could be abandoned.

I see a couple billion mails per week and can assure you that 5322.From's with a Sub-Domain but signed with the org-domain are a regular picture of totally valid mail streams, and this whole concept goes even deeper for large mail processors. It makes a huge difference for measuring reputation and responsibilities. And I think that this should be the baseline for the discussion here. As a mail receiver, I would at least assume, I and most of my colleagues use the org-domain concept to pin responsibilities to a clear and dedicated entity. If we abandon this, we are opening additional attack vectors without any increase in functionality and even increasing the complexity for almost all parties, only for the sake of getting the PSL out of the equation. 

Querying the PSL in a compiled trie data structure is much faster than even doing one DNS request, and even with the private part of the PSL this is a couple MB of memory. I get Mails that are larger than downloading the PSL once per day for a year. So why are we having this discussion? I know the PSL is not perfect, and I'm totally in for change if something doesn't work, but we have seen that DBound didn't made it and there are no real heavy usage PSL alternatives.

And one thing I really don't get, why do we want to solve that so heavily that we use scare tactics with phrasing like "if we don't solve it now, we would need to write another RFC in a couple of years", isn't that totally fine, for a standard to evolve and update it if it needs an update?

-----Ursprüngliche Nachricht-----
Von: dmarc <[email-masked]> Im Auftrag von Scott Kitterman
Gesendet: Montag, 1. November 2021 21:24
An: [email-masked]
Betreff: Re: [dmarc-ietf] same old org domain, Topic for IETF 112 - Policy Discovery

On Monday, November 1, 2021 3:17:05 PM EDT Alessandro Vesely wrote:
> From: [email-masked], signed by example.org which also publishes 
> a policy has to be valid.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 01 Nov 2021 22:25:30 +0000

Great.  As I mentioned, I don't currently have access to a large, diverse corpus of email to check against.

Could you perhaps do an assessment of the percentages of email you see where

5322.From == Org domain
5322.From is a sub-domain of Org domain

5321.MailFrom == 5322.From
5321.MailFrom is a sub-domain of 5322.From
5322.From is a sub-domain of 5321.From

d= domain == 5322.From
d= domain is a sub-domain of 5322.From
5322.From is a sub-domain of d= domain

I think that something like this would be very helpful for the group to understand what is currently used.

Clearly we need to carefully address current practice.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 01 Nov 2021 22:26:19 +0000

That might do it.

Scott K

---

From: "John Levine" <[email-masked]>
Date: 1 Nov 2021 18:29:46 -0400

>specify the organizational domain. This is orthogonal to how you discover

One of the points of the org domain is to provide a default policy for
hostnames like sharepoint12.dallas.organization.com that aren't
supposed to send mail and don't have a DMARC record at all.

I see how a pointer in a DMARC record could handle this other domain
is allowed to sign my mail, but I don't see how it provides the
default for all of the random hosts under organziation.com.

R's,
John

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 01 Nov 2021 22:36:44 +0000

>default for all of the random hosts under organziation.com.

Isn't that policy discovery?  I think we have that covered already.

Scott K

---

From: Dotzero <[email-masked]>
Date: Mon, 1 Nov 2021 18:51:13 -0400

On Mon, Nov 1, 2021 at 6:08 PM Tobias Herkula <[email-masked]>
wrote:

> Yes this is used in a significant way, dropping the mechanic of the
> org-domain would make a lot of things in processing inbound mail streams a
> lot more complicated.
>
> The PSL does not exists for DKIM or DMARC, it is a product of the CAB
> forum. And the idea was borrowed for DMARC, but without it, DMARC will have
> a hard time, and depending standards as well. I don't want to discuss how
> good or bad BIMI is, but without an "org-domain" it doesn't work. But if
> DMARC as one of the base requirements for BIMI drops the "org-domain"
> mechanic, you really need to produce a better alternative than, simply
> stating that things that are currently OK to do, are not used by enough
> entities and could be abandoned.
>
> I see a couple billion mails per week and can assure you that 5322.From's
> with a Sub-Domain but signed with the org-domain are a regular picture of
> totally valid mail streams, and this whole concept goes even deeper for
> large mail processors. It makes a huge difference for measuring reputation
> and responsibilities. And I think that this should be the baseline for the
> discussion here. As a mail receiver, I would at least assume, I and most of
> my colleagues use the org-domain concept to pin responsibilities to a clear
> and dedicated entity. If we abandon this, we are opening additional attack
> vectors without any increase in functionality and even increasing the
> complexity for almost all parties, only for the sake of getting the PSL out
> of the equation.
>
> Querying the PSL in a compiled trie data structure is much faster than
> even doing one DNS request, and even with the private part of the PSL this
> is a couple MB of memory. I get Mails that are larger than downloading the
> PSL once per day for a year. So why are we having this discussion? I know
> the PSL is not perfect, and I'm totally in for change if something doesn't
> work, but we have seen that DBound didn't made it and there are no real
> heavy usage PSL alternatives.
>
> And one thing I really don't get, why do we want to solve that so heavily
> that we use scare tactics with phrasing like "if we don't solve it now, we
> would need to write another RFC in a couple of years", isn't that totally
> fine, for a standard to evolve and update it if it needs an update?
>

Thank you for your voice of reason Tobias. It would appear that some are
willing to create a larger problem in order to address a smaller problem.

Michael Hammer

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 01 Nov 2021 23:03:41 +0000

>willing to create a larger problem in order to address a smaller problem.

Let me assure you, I'm not trying to break anything, very much the opposite.  It's difficult for people like me without access to tons of data to generalize from what we do have.  It appears that some of my assumptions are in error, so we shouldn't act on them.

That said, I don't think BIMI's needs should be a factor in the DMARC design.  If BIMI needs an org domain and it turns out DMARC doesn't, then they can define it to support their needs.

I think we've just started a conversation on how to address change (if any) in the DMARC alignment algorithm.  We're nowhere near the end, so please don't panic.

Scott K

---

From: Tobias Herkula <[email-masked]>
Date: Mon, 1 Nov 2021 23:16:06 +0000

I'm more or less thinking we should keep the alignment method as is, and only rephrase the subordinate clause to something that puts an emphasis on the fact that we currently don't have a perfect solution, and the PSL has shown to be a very stable external entity that is fine to use until something better comes around. If my google-foo is not failing me, the cab forum already discussed that at the beginning of the last decade, and still here we are at the end of 2021 and are kind of fine with what the PSL provides.

Every big change, like reference tags inside of DMARC records or even a more sophisticated solution that tries to solve that not only for DMARC but for others as well should be handled outside of this discussion here. We have enough examples with other standards that opted for a dependency instead of trying to solve it alone. And if that is so important, "dbound" is not dead, only concluded...

-----Ursprüngliche Nachricht-----
Von: dmarc <[email-masked]> Im Auftrag von Scott Kitterman
Gesendet: Dienstag, 2. November 2021 00:04
An: [email-masked]
Betreff: Re: [dmarc-ietf] same old org domain, Topic for IETF 112 - Policy Discovery

On November 1, 2021 10:51:13 PM UTC, Dotzero <[email-masked]> wrote:
>> org-domain would make a lot of things in processing inbound mail 
>> streams a lot more complicated.
>>
>> forum. And the idea was borrowed for DMARC, but without it, DMARC 
>> will have a hard time, and depending standards as well. I don't want 
>> to discuss how good or bad BIMI is, but without an "org-domain" it 
>> doesn't work. But if DMARC as one of the base requirements for BIMI drops the "org-domain"
>> mechanic, you really need to produce a better alternative than, 
>> simply stating that things that are currently OK to do, are not used 
>> by enough entities and could be abandoned.
>>
>> I see a couple billion mails per week and can assure you that 
>> 5322.From's with a Sub-Domain but signed with the org-domain are a 
>> regular picture of totally valid mail streams, and this whole concept 
>> goes even deeper for large mail processors. It makes a huge 
>> difference for measuring reputation and responsibilities. And I think 
>> that this should be the baseline for the discussion here. As a mail 
>> receiver, I would at least assume, I and most of my colleagues use 
>> the org-domain concept to pin responsibilities to a clear and 
>> dedicated entity. If we abandon this, we are opening additional 
>> attack vectors without any increase in functionality and even 
>> increasing the complexity for almost all parties, only for the sake of getting the PSL out of the equation.
>>
>> Querying the PSL in a compiled trie data structure is much faster 
>> than even doing one DNS request, and even with the private part of 
>> the PSL this is a couple MB of memory. I get Mails that are larger 
>> than downloading the PSL once per day for a year. So why are we 
>> having this discussion? I know the PSL is not perfect, and I'm 
>> totally in for change if something doesn't work, but we have seen 
>> that DBound didn't made it and there are no real heavy usage PSL alternatives.
>>
>> And one thing I really don't get, why do we want to solve that so 
>> heavily that we use scare tactics with phrasing like "if we don't 
>> solve it now, we would need to write another RFC in a couple of 
>> years", isn't that totally fine, for a standard to evolve and update it if it needs an update?
>>
>
>Thank you for your voice of reason Tobias. It would appear that some 
>are willing to create a larger problem in order to address a smaller problem.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 01 Nov 2021 19:35:47 -0400

I'm less sanguine about the felicity of the current state of affairs, but it 
may be the best we can do.  At the very least we should make it clear that 
only the ICANN DOMAINS section of the PSL should be used.  Not excluding the 
private domains from the beginning was a clear mistake (and I just submitted 
an erratum to document that).

Scott K

On Monday, November 1, 2021 7:16:06 PM EDT Tobias Herkula wrote:
> I'm more or less thinking we should keep the alignment method as is, and
> only rephrase the subordinate clause to something that puts an emphasis on
> the fact that we currently don't have a perfect solution, and the PSL has
> shown to be a very stable external entity that is fine to use until
> something better comes around. If my google-foo is not failing me, the cab
> forum already discussed that at the beginning of the last decade, and still
> here we are at the end of 2021 and are kind of fine with what the PSL
> provides.
 
> Every big change, like reference tags inside of DMARC records or even a more
> sophisticated solution that tries to solve that not only for DMARC but for
> others as well should be handled outside of this discussion here. We have
> enough examples with other standards that opted for a dependency instead of
> trying to solve it alone. And if that is so important, "dbound" is not
> dead, only concluded...

> >> doesn't work. But if DMARC as one of the base requirements for BIMI drops
 mechanic, you really need to produce a better
> >> alternative than,
> Let me assure you, I'm not trying to break anything, very much the opposite.
>  It's difficult for people like me without access to tons of data to
> generalize from what we do have.  It appears that some of my assumptions
> are in error, so we shouldn't act on them.
 
> That said, I don't think BIMI's needs should be a factor in the DMARC
> design.  If BIMI needs an org domain and it turns out DMARC doesn't, then
> they can define it to support their needs.
 
> I think we've just started a conversation on how to address change (if any)
> in the DMARC alignment algorithm.  We're nowhere near the end, so please
> don't panic.

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 6 Nov 2021 17:13:33 +0100

> I think that something like this would be very helpful for the group to understand what is currently used.

I don't currently have access to a large corpus of email either.  However, I managed to find the time to produce some stats.  On my disk I got:

     307655   80.27%  5322.From == Org domain
      75614   19.73%  5322.From is a sub-domain of Org domain
     204514   53.36%  5321.MailFrom == 5322.From
      25384    6.62%  5321.MailFrom is a sub-domain of 5322.From
       6348    1.66%  5322.From is a sub-domain of 5321.From
     145138   37.87%  5322.From not related to 5321.From
     177731   46.37%  d= domain == 5322.From
       5896    1.54%  d= domain is a sub-domain of 5322.From
      13954    3.64%  5322.From is a sub-domain of d= domain
      93398   24.37%  d= domain not related to 5322.From
     150056   39.15%  d= domain == Org domain
      47877   12.49%  d= domain is a sub-domain of Org domain
          1    0.00%  Org domain is a sub-domain of d= domain
      93045   24.28%  d= domain not related to Org domain

     383269 files processed
      38433 files skipped (37837 silently)

Files with neither a Return-Path nor a DKIM-Signature (no identifier) were silently skipped.  Other files were skipped for various reasons.  For each file, only one DKIM signature per type is counted.

The utility to produce that list is here, if someone wants to give it a try:
https://www.tana.it/alignmentstats-0.tar.gz

Best
Ale
-- 

ale@pcale:~/tmp/zdkimfilter/assess$ sha256sum alignmentstats-0.tar.gz
aa3eb2f036789006dcc5b65ce0492061d5d9827c9f2c5ed8638e2736e4c97ca0  alignmentstats-0.tar.gz