Subject: [dmarc-ietf] 3.2.6 The meaning of non-existence

From: Douglas Foster <[email-masked]>
Date: Tue, 14 Dec 2021 23:16:13 -0500

What does we mean for an RFC5322.From address to be “non-existent”?

We have said that it is non-existent because it fails the MX/A/AAAA test,
but we have not documented what that test represents.  Perhaps it seemed
obvious, but let's make it clear:

A failed MX/A/AAAA test is a very reliable indicator that the From address
does not have a mailbox, because the associated domain does not have a mail
server which accepts messages.  “Does not exist” means that the message
does not exist as a destination mailbox.

But is that result information useful, and if so, how?   What problem does
it resolve?

I estimate that 70% of the legitimate mail entering my organization is
unidirectional – messages which do not expect a reply by email.
Unidirectional traffic does not require an inbox.  When we determine that a
message does not have an inbox, we determine that it is definitely part of
the 70%.   I don't find anything actionable in that information.

The RFC5322.From identifier is an abstraction which represents a message
stream from a single entity acting as author.   Everything that the author
mails can be done through agents, where the agent is the  SMTP From
address.  A review of actual mail messages will show that legitimate
messages come from domains that do not have a mail server.

In the general case, an author account or domain exists simply because the
domain owner (or PSD) authorizes someone or something to use that name.
Our goal needs to be a test which identifies domain names which have never
been authorized by the domain owner or PSD.   We need a different test.

Doug Foster

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 15 Dec 2021 04:40:04 +0000

>been authorized by the domain owner or PSD.   We need a different test.

None of that is at all related to why we added the np= tag.  I'd suggest a review of the WG archives might be useful.

Scott K

---

From: Douglas Foster <[email-masked]>
Date: Wed, 15 Dec 2021 16:55:37 -0500

Scott,  I have many problems with your response.   Was it intended as an ad
hominem?   It certainly came across that way.

If the NP objective can be stated in a sentence or two, you should have
done so, instead of telling me to read years of archive.  An objective that
cannot be explained tersely is not sufficiently defined.

However, I do understand that your original case was to control misuse of
unregistered organization domains.   I have asked repeatedly for an
explanation of why the PSD team believes that MX/A/AAAA is the optimal
algorithm for that problem, since that information was not in your RFC.  I
am still waiting.   If the topic had been thoroughly vetted in the
committee, this should not have been a difficult request.

Nonetheless, this WG is not re-documenting the PSD experiment, we are
generalizing it to all DMARC policies.   I consider NP to be the most
important part of this document, because it is significant new
functionality to protect recipients and name owners.   But because we are
adding it during the standards track phase, we have to get it right on the
first try.   The only way to get it right is to analyze it thoroughly,
which I have been trying to do, with little cooperation, for about a year.

Once we have the correct algorithm for the general case, we can return to
the PSD case to determine if it requires special handling.  My current
opinion is that the optimal rule will work well for all cases.

As you probably suspect, I think the PSD group messed up on the most
important part of its tasking.   Either prove me wrong or let me help you
improve what you started.

Doug Foster

On Tue, Dec 14, 2021 at 11:40 PM Scott Kitterman <[email-masked]>
wrote:

>
>
> On December 15, 2021 4:16:13 AM UTC, Douglas Foster <
> [email-masked]> wrote:
> None of that is at all related to why we added the np= tag.  I'd suggest a
> review of the WG archives might be useful.
>
> Scott K
>
> _______________________________________________
> dmarc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dmarc
>

---

From: Barry Leiba <[email-masked]>
Date: Wed, 15 Dec 2021 17:44:46 -0500

> Scott,  I have many problems with your response.   Was it intended as an ad hominem?
> It certainly came across that way.

It doesn't seem even remotely so to me.  Please be careful with
attributing intent.  No one tried to say that we shouldn't listen to
you.

> If the NP objective can be stated in a sentence or two, you should have done so, instead of
> telling me to read years of archive.  An objective that cannot be explained tersely is not sufficiently
> defined.

It *is* reasonable to expect you to review earlier discussions, rather
than to ask the working group to revisit them without a sense of how
you're adding new information.

Barry

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 15 Dec 2021 18:48:42 -0500

On Wednesday, December 15, 2021 5:44:46 PM EST Barry Leiba wrote:
> > Scott,  I have many problems with your response.   Was it intended as an
> > ad hominem? It certainly came across that way.
> 
> > done so, instead of telling me to read years of archive.  An objective
> > that cannot be explained tersely is not sufficiently defined.
> 

Thanks.  Yes, that was my intent.

To give a short summary, in the interests of moving forward:

The domain owner publishing the DMARC record knows and controls what exists 
and what doesn't.  They don't have to guess.  The question was, particularly 
in the context of PSD, but not exclusively, would record publishers find it 
useful to be able to publish a different (and presumably more strict) policy 
for non-existent domains.  More p=reject equals more bad stuff not getting 
delivered.

I think we can say it's an pretty unqualified yes in the PSD realm:

$ dig +short txt _dmarc.gov
"v=DMARC1; p=reject; sp=none; np=reject; rua=mailto:[email-masked]"

$ dig +short txt _dmarc.mil
"v=DMARC1; p=reject; sp=none; np=reject; rua=mailto:[email-masked]"

$ dig +short txt _dmarc.gov.uk
"v=DMARC1;p=reject;sp=none;np=reject;adkim=s;aspf=s;fo=1;rua=mailto:dmarc-
[email-masked]"

$ dig +short txt _dmarc.police.uk
"v=DMARC1;p=none;sp=none;adkim=s;aspf=s;fo=1;rua=mailto:dmarc-
[email-masked];ruf=mailto:[email-masked]"

All of the current PSDs that have published records with any policy other than 
none have different sp= and np= policies.

Scott K

---

From: Tim Wicinski <[email-masked]>
Date: Wed, 15 Dec 2021 19:27:15 -0500

Thanks Scott.  _dmarc.police.uk doesn't seem to have the 'np' tag.

There are a number of domains with policies that have 'p=quarantine|reject
sp=none' - it would be good to see if 'np=reject' is added to any.

tim

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 15 Dec 2021 21:58:48 -0500

Agreed.  That's the most recently added PSD record and my guess is they are 
early in their journey.  For those of you that haven't done it, managing 
deployment of DMARC (+ DKIM and SPF) across a large number of domains is a 
very time consuming process.  Getting the internal policies right is at least 
as much work as the technical piece.  I don't have any internal insight, but 
given they're new to the game I'm not surprised they are in data collection 
mode.

Scott K

On Wednesday, December 15, 2021 7:27:15 PM EST Tim Wicinski wrote:
> > > > Scott,  I have many problems with your response.   Was it intended as
> > 
> > an
> >

---

From: Douglas Foster <[email-masked]>
Date: Thu, 16 Dec 2021 00:57:45 -0500

Yes, this is important stuff.

This is one of my problem scenarios:

A record arrives at the first hop and obtains DMARC PASS, based on SPF
and/or DKIM interpreted by a DMARC policy.  Based on DMARC PASS, the
RFC5322.From address is confidently judged to be "Honestly identified"
 DMARC checks SPF and DKIM, but not MX or A/AAAA.

But then it is forwarded and loses its credentials during forwarding.

On reception, because of DMARC FAIL, it is tested against NP.    NP checks
MX and A/AAAA but does not check SPF or DKIM.   The message fails this test
and is confidently judged to be "Fraudulently identified".

Which is true?   Was the message From address always fraudulent or always
honest?

To produce consistent results, do we change DMARC to require messages to
evaluate MX/A/AAAA or do we change NP to integrate SPF and DKIM?

Doug

---

From: Tim Wicinski <[email-masked]>
Date: Thu, 16 Dec 2021 01:27:19 -0500

> But then it is forwarded and loses its credentials during forwarding.

It sounds like you're rewriting the message header and sending it back out,
like a mailing list would do.
It's two different mail sessions.

This problem space is what ARC attempts to solve.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 15 Dec 2021 22:45:51 -0800

> and is confidently judged to be "Fraudulently identified".

The nearest thing I can imagine that would cause this is a From of "
[email-masked]" when that domain advertises a public key that verifies the
message, so it has a TXT at "selector._domainkey.example.com", but has no
MX, A, or AAAA for "example.com".  On relay, a mutation causes the
signature validation to fail at the final recipient.

Are you seeing cases like this?

-MSK

---

From: Douglas Foster <[email-masked]>
Date: Thu, 16 Dec 2021 18:52:45 -0500

We both know exactly what causes messages to lose credentials:
- A record that is only SMTP validated, which is then forwarded without
SMTP rewrite
- A message which is forwarded after modifications, such as the ubiquitous
"this message received from an external source".   Of course, it could be a
mailing list modification also.

The point of an NP test was, in my understanding, to identify names that
were never valid in any circumstance, like 'junk.junk.ietf.com", without
any dependencies on message path.    Why would we want to create a
duplicate of the mailing list problem?

However, if MX/A/AAAA is really the right test for fraudulent identifiers,
then we need to open a CVE against all implementations of RFC7489, because
implementations based on that spec have been confidently asserting honest
identifiers without checking the MX/A/AAAA condition.

Why do I need to provide case studies?    Isn't the burden of proof on the
team that told us that MX/A/AAA was absolutely the best possible test to
use?

On Thu, Dec 16, 2021 at 1:46 AM Murray S. Kucherawy <[email-masked]>
wrote:

>> On reception, because of DMARC FAIL, it is tested against NP.    NP
>> checks MX and A/AAAA but does not check SPF or DKIM.   The message fails
>> this test and is confidently judged to be "Fraudulently identified".
>>
>
> [email-masked]" when that domain advertises a public key that verifies
> the message, so it has a TXT at "selector._domainkey.example.com", but
> has no MX, A, or AAAA for "example.com".  On relay, a mutation causes the

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 17 Dec 2021 08:00:34 -0800

> mailing list modification also.

Yes, I'm aware of this aspect of message authentication.  That wasn't my
question.

> duplicate of the mailing list problem?

I understand the first sentence.  I do not see how the second follows.

> identifiers without checking the MX/A/AAAA condition.

I don't follow this either.

> team that told us that MX/A/AAA was absolutely the best possible test to

Because I'm trying to understand your concern.  Sure, it's reasonable for
us to question our assumptions.  But if I don't understand how you get your
premises, or how your premises lead to your conclusions, am I being
unreasonable when I ask for clarification or concrete examples?

-MSK

---

From: Dotzero <[email-masked]>
Date: Fri, 17 Dec 2021 12:29:41 -0500

> Which is true?   Was the message From address always fraudulent or always
<SNIP>

DMARC does not assess "honesty" nor does it assess "fraudulence". It only
determines whether something passes or fails the validation check. You are
apparently trying to overload your value interpretations in a manner that
does not exist in the standard.

Michael Hammer

---

From: Tim Wicinski <[email-masked]>
Date: Fri, 17 Dec 2021 12:38:54 -0500

> does not exist in the standard.
Thank you Michael, for reminding me of this.    DMARC provides a result
based on a collection of tests, and it is up to the receiver of the email
whether they choose to accept the email or to reject it.

tim

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 17 Dec 2021 19:52:14 +0100

> whether they choose to accept the email or to reject it.

Yet, honesty and legitimacy are somewhat similar, and we do foremost consider 
the latter aspect:

    DMARC is designed to prevent bad actors from sending mail that claims
    to come from legitimate senders, particularly senders of
    transactional email (official mail that is about business
    transactions).

Of course, if the From: domain doesn't exist at all, it cannot have a DMARC 
record.  However, according to the formal definition of Section 3.6.2, a 
non-existing domain can pass all DMARC tests.  IMHO, that is a gray area which, 
together with the null MX case, deserves being mentioned somewhere, in the same 
section, in Security Considerations or in an appendix.

Another difficulty of this subject might lay in the distinction between 
non-existing addresses and non-existing domains.  The SPF side of DMARC 
conflates those concept; and indeed "call" tests —part of other legitimacy 
assessments— are usually performed of the envelope address.  No-reply From: 
addresses have now become part of everyday life, but AFAIUI some hold that 
non-existent From: domains are legit too.  Does that such concern touch the 
question too?

Best
Ale
--

---

From: "John Levine" <[email-masked]>
Date: 17 Dec 2021 15:05:39 -0500

It appears that Alessandro Vesely  <[email-masked]> said:
>non-existing domain can pass all DMARC tests. ...

DMARC lets a domain owner say "if mail with my name on it doesn't have
these features, it's not from me." The org domain hack lets it say
similar things about its subdomains which may or may not exist.

But other than that, domains that do not exist do not have DMARC policies,
and DMARC has nothing to say about them.  Please stop trying to make it
into a FUSSP.

There are good reasons that a mail system might refuse mail from domains
that don't have an MX, A, or AAAA, but they have notthing to do with DMARC.

R's,
John

---

From: Douglas Foster <[email-masked]>
Date: Fri, 17 Dec 2021 18:26:38 -0500

The way I evaluate a design is to first look for logical consistency or
inconsistency.    If there is obvious inconsistency, then the design needs
to be re-evaluated to correct the problem.    Once a design appears to be
logically consistent, we do data analysis to validate our design.   The
criteria for DMARC PASS (the identity is authentic) and NP (the identity
cannot be authentic) should be aligned so that they produce results that
are not in direct conflict.   We seem to have an obvious design hole, with
an obvious solution, which is what I proposed.

As to examples, my data set is large enough to be informative but not large
enough to be determinative.   I see very little forwarded mail, so my data
set is not expected to answer your specific question.   I do see legitimate
messages using From domains that are not in DNS.  The existence of
legitimate non-DNS names, and the lack of a suitable compliance mechanism
for those names, is what got me concerned about this test definition.    I
have provided the group with examples, as has someone else.   Yet, the test
specification has not been modified to address that concern.   How do you
think that makes me feel?

A year after raising my concerns, I am still trying to get a collaborative
discussion started about what the optimal test looks like.  In a
collaborative discussion, those who are happy with the status quo convince
the skeptics to come on board, listen to their concerns, acknowledge them,
and do what they can to accommodate those concerns so that consensus can be
achieved.    I am willing to be convinced, but I am skeptical and I am
looking for some collaboration.

Doug

On Fri, Dec 17, 2021 at 11:00 AM Murray S. Kucherawy <[email-masked]>
wrote:

>> - A message which is forwarded after modifications, such as the
>> ubiquitous "this message received from an external source".   Of course, it
>> could be a mailing list modification also.
>>
>

---

From: Tim Wicinski <[email-masked]>
Date: Fri, 17 Dec 2021 18:38:22 -0500

> an obvious solution, which is what I proposed.
DMARC PASS != "this email is legit"

You are looking for a specific test that an email can go through which will
turn a crank and at the end of it says "Yes" or "no".

The text in -04 version is pretty clear:

   A DMARC pass indicates *only* that the RFC5322
<https://datatracker.ietf.org/doc/html/rfc5322>.From domain has been
   authenticated for that message.  *Authentication does not carry an
   explicit or implicit value assertion about that message or about the
   Domain Owner. *

You read "PASS" as "legit email" while I have always read it as "this
variable is true".

What the vendors like Agari, Valimail, Proofpoint, et al, is to
collect all the variables (SPF, DKIM, DMARC, plus organization
specific info) and allow the

organization to tweak as needed.

You are looking to solve the problem of legitimate email. DMARC is not
trying to solve that.

tim

> As to examples, my data set is large enough to be informative but not
> large enough to be determinative.   I see very little forwarded mail, so my
> data set is not expected to answer your specific question.   I do see
> legitimate messages using From domains that are not in DNS.  The existence
> of legitimate non-DNS names, and the lack of a suitable compliance
> mechanism for those names, is what got me concerned about this test
> definition.    I have provided the group with examples, as has someone
> else.   Yet, the test specification has not been modified to address that
> concern.   How do you think that makes me feel?
>
>>> were never valid in any circumstance, like 'junk.junk.ietf.com",
>>> without any dependencies on message path.    Why would we want to create a
>> However, if MX/A/AAAA is really the right test for fraudulent
>>> identifiers, then we need to open a CVE against all implementations of
>>> RFC7489, because implementations based on that spec have been confidently
>>> asserting honest identifiers without checking the MX/A/AAAA condition.
>>>
>>
>> Why do I need to provide case studies?    Isn't the burden of proof on
>>> the team that told us that MX/A/AAA was absolutely the best possible test
>>> to use?
>>>
>>

---

From: Douglas Foster <[email-masked]>
Date: Fri, 17 Dec 2021 18:55:58 -0500

That is not my position, and I don't know how you drew that conclusion from
those words.

I do take the position that DMARC PASS means "This name correctly
represents the stated domain", and NP=TRUE means "This name cannot
represent the stated domain because the domain owner never uses that
name".   I am willing to say that if NP=TRUE produces an accurate result, I
will block the message and I can see no reason why anyone else would do
otherwise.

DMARC FAIL is an ambiguous result, which was your point.  DMARC PASS is not
ambiguous.   NP=TRUE should be ambiguous if at all possible, otherwise it
adds no value.

But back to the actual topic:
- Do you believe the NP test can be useful?  If so, for what purpose?
- What is the optimal test to evaluate NP?  How did you reach that
conclusion?

Doug

On Fri, Dec 17, 2021 at 6:38 PM Tim Wicinski <[email-masked]> wrote:

> You are looking for a specific test that an email can go through which
> will turn a crank and at the end of it says "Yes" or "no".
>
>    A DMARC pass indicates *only* that the RFC5322 <https://datatracker.ietf.org/doc/html/rfc5322>.From domain has been
> You read "PASS" as "legit email" while I have always read it as "this variable is true".
>
>
> What the vendors like Agari, Valimail, Proofpoint, et al, is to collect all the variables (SPF, DKIM, DMARC, plus organization specific info) and allow the
>
> You are looking to solve the problem of legitimate email. DMARC is not trying to solve that.
>
>
> tim
>
>> A year after raising my concerns, I am still trying to get a
>> collaborative discussion started about what the optimal test looks like.
>> In a collaborative discussion, those who are happy with the status quo
>> convince the skeptics to come on board, listen to their concerns,
>> acknowledge them, and do what they can to accommodate those concerns so
>> that consensus can be achieved.    I am willing to be convinced, but I am
>> skeptical and I am looking for some collaboration.
>>
>> Doug
>>
>>> Because I'm trying to understand your concern.  Sure, it's reasonable
>>> for us to question our assumptions.  But if I don't understand how you get
>>> your premises, or how your premises lead to your conclusions, am I being

---

From: Scott Kitterman <[email-masked]>
Date: Sat, 18 Dec 2021 00:00:56 +0000

On December 17, 2021 11:26:38 PM UTC, Douglas Foster <[email-masked]> wrote:
..
>looking for some collaboration.
It may be that this is a cultural issue then.  In the IETF, where there is an established consensus (rough or not), the burden is on those seeking to overturn the consensus to convince people.  If you think about it, if a working group were obligated to rehash things whenever new people show up, it would be difficult to accomplish anything in an open environment where new people can show up anytime.

I suspect you might have more luck if you first consulted Chesterton's Fence.  I think you'd have more luck with questions about why things are the way they are than immediate assertions that they are wrong.

Scott K

P.S. At least as I understand it.  I'm relatively new too.

---

From: Tim Wicinski <[email-masked]>
Date: Fri, 17 Dec 2021 20:43:17 -0500

On Fri, Dec 17, 2021 at 6:56 PM Douglas Foster <
[email-masked]> wrote:

> That is not my position, and I don't know how you drew that
> conclusion from those words.
>

Then my mistake.

> DMARC FAIL is an ambiguous result, which was your point.  DMARC PASS is
> not ambiguous.   NP=TRUE should be ambiguous if at all possible, otherwise
> it adds no value.
>
I see the NP tag being useful for mid to large organizations that have a
regular amount of organizational change (mergers, acquisitions, etc).
A large mostly static organization will not deploy the np tag, because "p
== sp", where the domain tag = the subdomain tag.
Larger organizations deploying DMARC usually run into the problem of not
knowing all the mail flows from all the change, and they end up
with "p != sp" for some period of time.  In these cases, np is really
useful in preventing attack vectors using subdomains (log4j.example.com)
from
being used.

I am sure there are folks who track DMARC record changes over time, but
back in mid 2020 I pulled a bunch of DMARC records of some alexa top 10K,
and noticed the number of domains where "p != sp".  Doing a quick pull of
those domains and checking now I see a number of them now show "p == sp",
which means they feel they have a better understanding of their mail
flows.

I worked for a large organization which had to set "p != sp" for a period
of time as they understood things better. They are now a "p == sp"
organization now, which is good. But having the added bonus of blocking
invalid subdomains during that migration period I am sure would have
made folks feel better.

(the other use case for the np tag is around TLDs and also the SLDs like
co.uk, and Scott is all over that).

tim

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 17 Dec 2021 21:05:45 -0500

> co.uk, and Scott is all over that).

Actually .gov.uk.  I'd expect that .co.uk has similar challenges to what we'd 
expect with .com.  

What you describe is something that was in my mind when we defined np=, despite 
the focus on PSD at the time.  I think that np= has potentially broad utility 
as a gap filler during the deployment process for large organizations.

I think p=reject will be a problem of a lot of organizations for a long time, 
so I think anything we can do to make it easier to have some set of mail 
covered by a reject policy is a good thing.

Scott K

---

From: Douglas Foster <[email-masked]>
Date: Fri, 17 Dec 2021 22:47:45 -0500

This is a useful point for discussion.

MX checks are a valid tool for assessing SMTP MailFrom addresses, since the
sender is supposed to be ready to accept non-delivery reports and other
automated messages.   Of course, this has applicability if (but only if)
the RFC5322.From domain is the same as the RFC5321.MailFrom domain. As far
as I know, DMARC is the only established tool for evaluating the
RFC5322.From address when it differs from the RFC5321.MailFrom address.  So
to my thinking, the NP test is new territory without precedent.

I am looking for a test that says decisively, "The domain owner has never
authorized the use of this name for any mail-related purposes."     There
are many reasons why an acceptable message may fail DMARC verification.
 One of those reasons is that the message is an impersonation, but the
impersonation is acceptable, typically because it is sent for the benefit
of an account user.   But acceptable messages will always use a domain name
that is already in use by the domain owner, never one that they invented.
  The use of a never-authorized name will always be an unacceptable form of
impersonation, and should be blocked.   I thought that the whole WG had
consensus on this point.

The question then becomes, "How do we construct a decisive test?"   We need
a test that has very little possibility of a false positive that would
cause a message to be blocked.   My primary target is the educational
segment, where everyone assumes that they will always use P/SP=NONE.
because they never want their legitimate messages, including mailing list
messages to be blocked.   They should be willing to use NP=Reject if they
are confident that NP will never block a legitimate message.

A weak algorithm is acceptable if it blocks some but not all messages from
never-used domains;   false negatives are undesirable but tolerable.   A
result of DMARC FAIL with NP=FALSE will leave the evaluator in the same
risk posture as if the NP test had not been performed.

The only method we have for evaluating NP is the DNS, so the weakest
possible rule is to block on NXDOMAIN.   Unfortunately, even that test
returns false positives.   They occurs when messages sent by email service
providers use the provider's SMTP address, and this is the norm.  For these
mailings, the only constraints on the RFC5322.From address are the ones
imposed by the provider (you will not impersonate my other clients) and by
the client itself (via DMARC).   With relaxed alignment, the client can
invent and use domain names that are not in DNS.    Consequently, any NP
test will likely require a participating organization to implement some
compliance measures, such as creating DNS entries for those that do not
exist at present.   Those measures should be as simple and error-free as
possible, or the risk-averse organization will simply not participate.

To implement the NP tests, we devise DNS queries which indicate that the
name exists.   When a DATA result is not obtained by any of those queries,
we return NP=TRUE.    To minimize false positives, we want to use an
expansive set of DNS queries.   MX+A+AAAA gives one set of in-use names.
 Adding SPF will add another set of in-use names.   Adding exact-match
DMARC and DKIM will add another set of in-use names.  The only reason to
omit these tests is if you can demonstrate that one of them is always
superfluous because it will only include names obtained from other sources,
or because it misleads.

A+AAAA may belong because it adds to the inclusive list, but it weakens the
rule a lot.  So I ask whether an acceptable result can be produced without
it, a direct challenge to my previous sentence.   This is a tricky one, and
best left for another stage in the analysis.

Doug

On Fri, Dec 17, 2021 at 7:01 PM Scott Kitterman <[email-masked]>
wrote:

>
>
> On December 17, 2021 11:26:38 PM UTC, Douglas Foster <
> It may be that this is a cultural issue then.  In the IETF, where there is
> an established consensus (rough or not), the burden is on those seeking to
> overturn the consensus to convince people.  If you think about it, if a
> working group were obligated to rehash things whenever new people show up,
> it would be difficult to accomplish anything in an open environment where
> new people can show up anytime.
>
> I suspect you might have more luck if you first consulted Chesterton's
> Fence.  I think you'd have more luck with questions about why things are
> the way they are than immediate assertions that they are wrong.
>
> Scott K
>

---

From: Douglas Foster <[email-masked]>
Date: Sat, 18 Dec 2021 07:18:13 -0500

The previous message laid out what we should seek in an NP test, but rushed
through the ending.

We are looking to create a test:

-          Which identifies all names that are used for RFC5322.From
addressing, so that names which are not identified can be classified as
NP=TRUE because the domain owner has never authorized the name for use with
>From addressing.

-          Which produces no false positives after domain owners have taken
necessary compliance measures

-          Which requires a minimum of compliance effort by the domain owner

-          Which minimizes false negatives as much as possible.

The universe of all possible domain names can be partitioned as follows:

-          Names which are used for both SMTP and RFC5322.From.

-          Names which are used for SMTP only, not for RFC5322.From

-          Names which are used for RFC5322.From only, not for SMTP

-          Names which are no used at all for email.

Names used for both SMTP and From

For the first group, assume that this occurs because some outbound messages
have the same domain for RFC5321.MailFrom and RFC5322.From.   How do we
detect names used for RFC5321.MailFrom?

-          SPF is the first indicator, because SPF explicitly affirms that
the domain name is used as a MailFrom address.

-          MX is a secondary indicator, because it explicitly affirms that
the domain name accepts mail.

-          A/AAAA does not explicitly affirm anything about mail usage, but
it may be used for mail without being explicitly identified.   Treating
A/AAAA as equivalent to MX will cause a high number of false negatives,
weakening the effectiveness of the NP test.   So we need to evaluate
whether its inclusion is necessary.

Names used for SMTP only

We currently have no way to detect domain names that are only used for
SMTP.   Unless we add an indicator for this purpose, we are forced to
assume that all SMTP names are also FROM names.   This creates some false
negatives, but we have said that some false negatives are tolerable even
though they are not desirable.

Names used for RFC5322.From only

In the context of DMARC, names that are not used for SMTP Mail should be
protected with a DKIM signature.  Additionally, email is the primary use
case for DKIM signatures, so the existence of a DKIM public key is at least
suggestive that the name is used for email.   DNS does not provide a simple
way to identify all DKIM signatures on a domain name, so we are limited to
checking only those names that are found within DKIM signatures on the
message being evaluated.

If a message has a DKIM signatures, and a public key exists in DNS for the
specified scope, we can reasonably conclude that the domain owner intends
to sign some messages with that signature, and that the most likely domain
name for which it will be used is the exact match name.   Consequently, the
recommended test for this situation is the existence of a DKIM public key
which matches a DKIM signature in the message.   The NP test is only
applicable when the message does not produce DMARC PASS, so the signature
itself has tested as unverified, but the presence of the public key
indicates that the name is used for email sometimes, even if this
particular signature is a forgery.

Of course, a message can be DMARC PASS using relaxed alignment on a DKIM
signature.   This is the reason that an RFC5322.From domain name can be
invisible in DNS.  To allow any unverified signature key to match any
aligned domain name would create too many false negatives.  Domain owners
will need to take compliance action to ensure that names used only for
RFC5322.From are able to comply with the NP test by one of the defined
methods.   To address this need, it seems appropriate to assume that the
existence of an exact-match DMARC policy is also an indicator that the name
is used for RFC5322.From messaging.

Names which are not used for any email purpose

These domain names are identified when all of the other tests have not
produced a match, and these produce NP=TRUE as the result.

Refining the test:  Should A/AAAA be included in the test rule?

The NP test is not applicable to all mail, it is only applicable to domains
that have published a DMARC policy with an NP clause.   Publishing a NP
policy, other than NONE, is an assertion that the domain owner has
voluntarily undertaken compliance with the NP test.   The DMARC policy also
suggests that the domain owner has a general interest in making his
outbound messages DMARC compliant.   The A/AAAA test is needed to avoid
false positives only when:

-          The name is actually used for SMTP mail, and

-          No SPF record has been published, and

-          No MX record has been published.

For domains that are participating in DMARC, the absence of an SPF record
would be surprising, and the absence of both MX and SPF would be even more
surprising.   Treating an implicit MX as if it was an explicit MX
introduces a large number of false positives, significantly weakening the
effectiveness of the NP test.   I propose that requiring MX or SPF or
exact-match DKIM public key or exact-match DMARC policy provides enough
options for compliance.   The A/AAAA test is not necessary, and is actually
counterproductive to the goals of the NP test.

On Fri, Dec 17, 2021 at 9:06 PM Scott Kitterman <[email-masked]>
wrote:

> > > name".   I am willing to say that if NP=TRUE produces an accurate
> result,
> > > I

---

From: Jeremy Harris <[email-masked]>
Date: Sat, 18 Dec 2021 14:37:10 +0000

On 18/12/2021 03:47, Douglas Foster wrote:
> the RFC5322.From domain is the same as the RFC5321.MailFrom domain.

I disagree.  It is well-established practice for a mailing list manager
to accept and process NDRs accepted on the 5321.mailfrom (which differs
from the 5322.from).
-- 
Cheers,
   Jeremy

---

From: "John Levine" <[email-masked]>
Date: 18 Dec 2021 12:15:17 -0500

>to accept and process NDRs accepted on the 5321.mailfrom (which differs

Jeremy is right. Mailing lists always, and I mean always, put their
own 5321 bounce address on the messages so they can do bounce
management. If you look at the mail from this list, the bounce address
is [email-masked].

I have to say I am dismayed that we are spending time dealing with such
utterly basic misconceptions here.

R's,
John

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 18 Dec 2021 12:50:52 -0500

Anyone raising 'Chesterton's Fence' sets my teeth on edge.

There is a longstanding field of political engagement called Agenda Denial
that deals with winning arguments they cannot possibly win by making sure
the argument is never made.

It is never the right time, and never will be.

There will always be something that must be done first.

The people raising the issue have raised it in the wrong way.

The issue needs more study.

G. K. Chesterton was a rather unpleasant chap. Like most of the English
establishment of the time, his goal was to avoid sharing any of the wealth
of Empire with the undeserving masses who created/looted it.

I strongly caution against anyone attempting to raise his standard in a
technical discussion.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 18 Dec 2021 13:17:01 -0500

My analysis is rather different but comes to a stronger statement of John's
point.

Mailing lists are not well supported by SMTP and never will be. Any
discussion of how to make mailing lists work better has to begin with the
acceptance that they will never work very well which is what most people
have been arguing in this thread.

That does not mean that we should not attempt to make changes but it does
mean that any changes should be considered through two lenses, not just
one. First 'will this make mailing lists a bit more tolerable', Second,
'will this help a successor technology work better'.

Rather than seeing mailing lists as they work today as the pinnacle of
evolution, we should see them for what they are, a rather disgusting hack
that we never got round to fixing.

Why is it assumed that the input to a mailing list is an SMTP email? That
seems to me to be a rather narrow assumption.

I am typing this into Gmail, why assume that my HTTP transaction must be
mediated by an SMTP transaction to reach a mailing list user? Why should
'mailing list' necessarily involve SMTP at all?

Once we recognize that the inputs and outputs from 'mailing lists' can be
through other transports than SMTP, all arguments about preserving
'original' headers collapse. This is not an interaction between an SMTP
sender and receiver, the mailing list itself is a principal.

I am of course aware that you can read IETF lists via IMAP, NNTP etc. In
the short term, those efforts are being hobbled by the lack of decent
clients. If the Web 4.0 folk get somewhere with a new generation of user
centric social media, that might change.

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 18 Dec 2021 19:17:11 +0100

> similar things about its subdomains which may or may not exist.

Yup... more or less.  Time gave a good summary of p <=> sp.  np is new and 
there's no data but np=reject.  Almost boolean.  (And I still don't see why a 
legit sender would use a non-existing domain in From:.)

> and DMARC has nothing to say about them.  Please stop trying to make it

Nobody is trying to invent a FUSSP here.  However, DMARC has to coexist with 
different email filters and different aims, so it's useful to give some hint at 
how it relates to them.  Adopters might stare at Section 3.6.2 trying to grasp 
its (lack of) meaning.

> that don't have an MX, A, or AAAA, but they have nothing to do with DMARC.

It's easy to build false syllogisms using those terms, though.

Best
Ale
--

---

From: "John R Levine" <[email-masked]>
Date: 18 Dec 2021 13:39:19 -0500

> different email filters and different aims, so it's useful to give some hint 
> at how it relates to them.  Adopters might stare at Section 3.6.2 trying to 
> grasp its (lack of) meaning.

I would not object to a sentence that clarifies that DMARC does not apply 
to domains that do not publish a DMARC policy, including those that do not 
publish a policy because they do not exist.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Scott Kitterman <[email-masked]>
Date: Sat, 18 Dec 2021 20:44:32 +0000

Okay.  I didn't mention the fence as an endorsement of his personality or politics.  Not sure why that's relevant.

Generally, I think arguing that something is wrong and should be changed without understanding why it's the way it is isn't a great way to go.

Scott K

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 18 Dec 2021 16:06:12 -0500

Accusing other people of disagreeing with you because they fail to
understand your position and that they would agree with you if only they
did their research is a rather aggressive move.

It is the constant refrain of the coinsplainer community.

If your goal is to preserve the status quo without actually defending your
position, telling people that it is incumbent on them to find the arguments
against their proposal and agree with them does the trick.

We do need a better way to track architectural arguments so that they do
not get lost. There is another form of this particular pathology which hit
us with the DNS CAA record which I proposed before the DigiNotar attack,
became an RFC but nobody took any notice and then a few years later, a new
issue came up and we ended up with a CABForum mandate to deploy.

I never considered CAA perfect, it was a compromise between a set of rather
ugly constraints imposed by DNS DNS-ing. By the time we got round to the
BIS, I had forgotten the rationale for the compromises. So we changed them.
And then spent three years in a set of stepwise refinements that brought us
back to the starting point because DNS is still DNS-ing and even harder.

On Sat, Dec 18, 2021 at 3:44 PM Scott Kitterman <[email-masked]>
wrote:

> Generally, I think arguing that something is wrong and should be changed
> without understanding why it's the way it is isn't a great way to go.
>
> Scott K
>
> collaborative
> >> >and do what they can to accommodate those concerns so that consensus
> can
> >> be

---

From: Scott Kitterman <[email-masked]>
Date: Sat, 18 Dec 2021 23:32:24 +0000

That's true.  It's equally true that not every case of suggesting someone needs to do additional research to understand the current state of things is such a strategy.

In this case, at least as I understand it, the OP wants some kind of absolute, guaranteed mechanism for indicating a domain is never legitimately used in email and is complaining that the RFC 9091 definition for a non-existent domain doesn't fulfill the criteria for that.

On this we agree.  It doesn't.  It was never intended to.  I don't understand what you are doing, but I'm sure you are doing it wrong is not a particularly good argumentative approach either.

I completely agree that "we've discussed that before, go read the archives" can and is used to shut down discussion in the IETF, but that doesn't mean it's always the wrong answer.

Scott K

Scott K

---

From: Tim Wicinski <[email-masked]>
Date: Sat, 18 Dec 2021 23:45:58 -0500

> Actually .gov.uk.  I'd expect that .co.uk has similar challenges to what

I knew I'd get something wrong.

> as a gap filler during the deployment process for large organizations.

As an operations person who has worked with a lot of technical debt, when I
saw your idea
for np=, *all* I could think of was large organizational issues.

> covered by a reject policy is a good thing.

Here is a data point on this.  Sometime back in Fall 2020, I was collecting
some DMARC DNS records.
I was using the alexa-top* lists, which are horrible datasets, because
web domains don't always send email
and email domains don't have web properties etc.

Using the top-1000 domains:

1000 domains
 562 domains with DMARC record
 318 domains with DMARC record where p != sp

Rechecking that same list of domains last night:

  371 domains with DMARC record where p != sp

53 domains from this list updated their policies.  I did not dive into the
specifics, but they mostly appear
to have the 'p=' move forward (none->quarantine->reject) while 'sp='
appeared to stay the same.
(I see a few where 'p=' moves to reject and 'sp=' moves to none. I can
figure this out if interested)

To Scotts' point on organizations moving to p=reject will take time, I
would agree.  But allowing a large organization
allow sp=none but np=reject is a huge help.  (This also means organizations
need to cull the DNS data, but some of us
are a bit too obsessive about this)

tim

---

From: Douglas Foster <[email-masked]>
Date: Sun, 19 Dec 2021 14:35:30 -0500

What I said was that reception of NDRmessages is only a requirement for the
SMTP From address, so they are only required for the RFC5322.From address
when the two From addresses match.   For msiling list messages, tbe two do
not match.

My topic was about the ability or inability to detect a never-valid RFC5322
>From address.   I am not engaged in any effort to change mailing lists.
 NP=reject MUST never reject mailing list traffic.  If we cannot do that,
NP is useless.

But we can meet that requirement, if we construct the right test.   I can
support several different variations of the test, which differences in
strictness and complexity.   I just cannot support the MX-A-AAAA test.

Doug

Doug

---

From: Scott Kitterman <[email-masked]>
Date: Sun, 19 Dec 2021 20:42:16 +0000

If the domain owner has suggested that you reject mail from a sub-domain that has none of A, AAAA, or MX records, why would you not do that?  Just as with any DMARC policy it's on the sender to ensure authorized email conforms to the policy.  My impression is that you think that rejecting mail from such sub-domains is inherently risky somehow?

My view is that it's substantially less risky than for more usual sub-domains.  Note that I don't claim it's risk free.  No filtering decision is risk free, so I don't find suggestions that it's not totally free of uncertainty particularly useful.

My sense is that you are still searching for something that the np= tag was never meant to be.  It might be more fruitful to try and specify what problem you are trying to solve and how we might go about it independent of the non-existent domain definition.  Maybe you can propose a totally new tag that addresses the issues you are concerned about.  If this new tag gets support from the group then we could look at if np= is still needed or if it's redundant.

Scott K

---

From: Douglas Foster <[email-masked]>
Date: Sun, 19 Dec 2021 20:17:54 -0500

Yes, I am looking for a high degree of certainty, comparable to what I get
with DMARC PASS.   (For example, the DKIM private key might have been
compromised, but I choose not to worry about that possibility.)

As I have said several times recently, DMARC FAIL is an ambiguous result,
and it is desirable to partition the set of all failures to see if
certainty can be obtained for some of them.   ARC is an attempt to solve
uncertainty for a subset of messages by providing a new authentication
method which says ARC-satisfactory messages are definitely not malicious
impersonations.   NP is, at least to my mind, is an attempt to solve
uncertainty in the opposite direction by identifying a subset of failures
that can only be malicious impersonations.

When acceptable impersonation occurs, it is always on behalf of an actual
user.   If there is no actual user, impersonation can only be malicious.
For simplicity, and of necessity, we ignore risks associated with the
local-part and focus on the domain name.   We ask whether an entire domain
can be categorized as never-valid, so that messages to that domain can be
moved from unverified-with-uncertain-risk to unverified-with-confirmed-risk.

At minimum, to satisfy the PSD goal, we need to be able to say that
"DoesNotExist.tld" is invalid because the organization is not registered
with a PSD.   For that to work, we also need to not block messages from
valid names.   The NXDOMAIN test is sufficient to solve the PSD problem, so
the only issue for registered domains is to ensure that valid domains do
not return NXDOMAIN.  A more complex rule can block more non-mail names
from impersonation attacks, but they increase the compliance issues.

DF

On Sun, Dec 19, 2021 at 3:42 PM Scott Kitterman <[email-masked]>
wrote:

> If the domain owner has suggested that you reject mail from a sub-domain
> that has none of A, AAAA, or MX records, why would you not do that?  Just
> as with any DMARC policy it's on the sender to ensure authorized email
> conforms to the policy.  My impression is that you think that rejecting
> mail from such sub-domains is inherently risky somehow?
>
> My view is that it's substantially less risky than for more usual
> sub-domains.  Note that I don't claim it's risk free.  No filtering
> decision is risk free, so I don't find suggestions that it's not totally
> free of uncertainty particularly useful.
>
> My sense is that you are still searching for something that the np= tag
> was never meant to be.  It might be more fruitful to try and specify what
> problem you are trying to solve and how we might go about it independent of
> the non-existent domain definition.  Maybe you can propose a totally new
> tag that addresses the issues you are concerned about.  If this new tag
> gets support from the group then we could look at if np= is still needed or
> if it's redundant.
>
> Scott K
>
> On December 19, 2021 7:35:30 PM UTC, Douglas Foster <
> >> >> MX checks are a valid tool for assessing SMTP MailFrom addresses,
> since
> >> the

---

From: tjw ietf <[email-masked]>
Date: Sun, 19 Dec 2021 20:56:53 -0500

I have seen many cases of an actual user generating an “acceptable iMpersonation”, but the Organization wanted those ruled Invalid because it was Not approved. 

Also please be more precise when using the term “domain” around DNS people. To us every label separated by a ‘.’ is a domain.  

Tim 

Sent from my iPhone

> On Dec 19, 2021, at 20:18, Douglas Foster <[email-masked]> wrote:
> 
> 
> Yes, I am looking for a high degree of certainty, comparable to what I get with DMARC PASS.   (For example, the DKIM private key might have been compromised, but I choose not to worry about that possibility.)
> 
> As I have said several times recently, DMARC FAIL is an ambiguous result, and it is desirable to partition the set of all failures to see if certainty can be obtained for some of them.   ARC is an attempt to solve uncertainty for a subset of messages by providing a new authentication method which says ARC-satisfactory messages are definitely not malicious impersonations.   NP is, at least to my mind, is an attempt to solve uncertainty in the opposite direction by identifying a subset of failures that can only be malicious impersonations.
> 
> When acceptable impersonation occurs, it is always on behalf of an actual user.   If there is no actual user, impersonation can only be malicious.  For simplicity, and of necessity, we ignore risks associated with the local-part and focus on the domain name.   We ask whether an entire domain can be categorized as never-valid, so that messages to that domain can be moved from unverified-with-uncertain-risk to unverified-with-confirmed-risk.
> 
> At minimum, to satisfy the PSD goal, we need to be able to say that "DoesNotExist.tld" is invalid because the organization is not registered with a PSD.   For that to work, we also need to not block messages from valid names.   The NXDOMAIN test is sufficient to solve the PSD problem, so the only issue for registered domains is to ensure that valid domains do not return NXDOMAIN.  A more complex rule can block more non-mail names from impersonation attacks, but they increase the compliance issues.
> 
> DF
> 
> 
> 
>> On Sun, Dec 19, 2021 at 3:42 PM Scott Kitterman <[email-masked]> wrote:

---

From: "John Levine" <[email-masked]>
Date: 19 Dec 2021 20:57:17 -0500

It appears that Scott Kitterman  <[email-masked]> said:
>If the domain owner has suggested that you reject mail from a sub-domain that has none of A, AAAA, or MX records, why would you not do that?  Just as with any DMARC policy it's on
>the sender to ensure authorized email conforms to the policy.  My impression is that you think that rejecting mail from such sub-domains is inherently risky somehow?

I may not have been clear enough -- I agree that sp= and np= let you say things about non-existent subdomains of a domain that does exist.

But if a domain does not exist, and is not a subdomain of one that publishes a sp= or np= policy, DMARC has nothing to say about it.

As I said, there may be other reasons to reject the mail, but they're not DMARC.

R's,
John

---

From: Douglas Foster <[email-masked]>
Date: Sun, 19 Dec 2021 23:10:31 -0500

Yes, "Acceptable" is in the eye of the evaluator.   I was not taking a
position about what should be acceptable.

I grudgingly accept messages that violate my DMARC policy, from some
sources.   Their messages are simply too important.

My point is that RFC 7960 issues will ONLY occur on actual user accounts,
not for accounts on non-mail domain names.   A domain name that has never
been authorized or used for email purposes will never have users and will
never be "acceptable".   "Never" means that I have opportunity to eliminate
ambiguity.  This is the mail that I want to block.

Doug

On Sun, Dec 19, 2021 at 8:56 PM tjw ietf <[email-masked]> wrote:

> I have seen many cases of an actual user generating an “acceptable
> iMpersonation”, but the Organization wanted those ruled Invalid because it
> was Not approved.
>
> Also please be more precise when using the term “domain” around DNS
> people. To us every label separated by a ‘.’ is a domain.
>
> Tim
>
> Sent from my iPhone
>
> On Dec 19, 2021, at 20:18, Douglas Foster <

---

From: Douglas Foster <[email-masked]>
Date: Mon, 20 Dec 2021 07:12:34 -0500

Scott asked some questions along the lines of "since no test is perfect,
why don't we just run with the easily-understood MX/A/AAA test.   I had not
responded to that part of the question previously.

I have two major problems with the MX/A/AAAA test:

- If we are only testing for "Does this domain send out SMTP mail?", then
SPF must be included in the criteria.  SPF speaks to what the domain sends,
while MX speaks to what the domain receives, and A/AAAA speaks to what the
domain might receive.   If we are testing for send behavior, we need to
examine the send authentication attribute.   Any test which is based on
SMTP and other things should include SPF for the same reason.

- Since we are not testing exclusively for SMTP behavior, we need to think
about how to avoid false positives, and how a domain owner changes a test
result from Fail to Pass.   I have been calling this action "compliance
measures", hoping that the term was clear.    For the MX/A/AAAA test, with
or without SPF, the only way to indicate compliance is to publish false
information about SMTP configurations that do not apply to this domain.
 Such false information has the potential for changing the behavior of SMTP
participants, and cannot be acceptable in a standards-track document.

I have suggested feature-specific DNS flags, to indicate that a particular
domain does or does not participate in FROM addressing, or does not
participate in SMTP addressing.   These have the advantage of being very
specific, but they depend on domain owners publishing new information.
 A test which minimizes publishing new data has a better chance of being
usable, with confidence, by evaluators.   So I suggested an exact-match
DMARC policy as an indicator to mean "used for RC5322.From addressing."
I am not entirely happy with that workaround, as it may have unwanted side
effects about how reporting occurs back to the domain owner.

DF

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 20 Dec 2021 08:15:46 -0500

In an organization of non-trivial size the chance that the person making the 
decision about what to put in DNS can reliably answer the question "used for 
RC5322.From addressing" is vanishingly small.  They can, however, be very 
confident about if they have set up A/AAAA/MX records.  At best you are trading 
one kind of uncertainty for another.

Scott K

On Monday, December 20, 2021 7:12:34 AM EST Douglas Foster wrote:
> > >What I said was that reception of NDRmessages is only a requirement for
> > 
> > the
> > 
> > >My topic was about the ability or inability to detect a never-valid
> > 
> > RFC5322
> > 
> > >> >> sender is supposed to be ready to accept non-delivery reports and
> > 
> > other
> > 
> > >> >> automated messages.   Of course, this has applicability if (but only
> > 
> > if)
> >

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 20 Dec 2021 18:39:15 +0100

On Sun 19/Dec/2021 21:42:16 +0100 Scott Kitterman wrote:
> If the domain owner has suggested that you reject mail from a sub-domain that has none of A, AAAA, or MX records, why would you not do that?

Are we sure that's what the domain owner meant to suggest?

An organization can use sp= on its root domain record, or it can define a DMARC 
record at some subdomains.  So far so good.

Then it turns out that one can also define DMARC record at some non-existing 
sub domains, possibly as an alternative to using np=...  Now this begins to 
look puzzling.

The reason to introduce np= was to select domains that really don't exist.  Why 
don't we stick to that definition?

Best
Ale
--

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 20 Dec 2021 12:41:25 -0500

On Monday, December 20, 2021 12:39:15 PM EST Alessandro Vesely wrote:
> > that has none of A, AAAA, or MX records, why would you not do that?
> An organization can use sp= on its root domain record, or it can define a
> DMARC record at some subdomains.  So far so good.
> 
> The reason to introduce np= was to select domains that really don't exist. 
> Why don't we stick to that definition?

What definition are you wondering why we didn't stick to?

Scott K

---

From: "John Levine" <[email-masked]>
Date: 20 Dec 2021 15:20:45 -0500

>sub domains, possibly as an alternative to using np=...  Now this begins to 

Only if you misunderstand the way the DNS works. If there is a txt
record for _dmarc.foo.bar, that means the name foo.bar exists. It may
not have any records, but it exists. That's the difference between a
DNS NXDOMAIN response for a non-existent domain and NOERROR for one
with no records. The difference is not an accident, not a mistake, and
is not going to change. See for example RFC 8020.

>don't we stick to that definition?

Yes indeed, why don't we, and end this pointless argument?

R's
John

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 21 Dec 2021 13:05:35 +0100

> What definition are you wondering why we didn't stick to?

Real non-existence.  I'm not sure how to define it formally, because it would 
be exorbitant to require to try everything.  On the other hand an application 
probably tried to authenticate SPF, DKIM, and DMARC already.  If any of those 
record was found, then it's silly to even lookup A/ AAAA/ MX for the purpose to 
determine if the domain exists.

Best
Ale
--

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 21 Dec 2021 12:30:38 +0000

>determine if the domain exists.

I don't remember exactly why we settled on A/ AAAA/ MX, but the lack of a clear, actionable definition is why we included one. Lack of DNS records related to email authentication only means lack of email authentication, which is in no way required.  Given the way most systems are typically architected, by the time you are doing email authentication, A or AAAA and MX will already be in the local cache, so this is a pretty inexpensive thing to check.

Scott K

---

From: Miles Fidelman <[email-masked]>
Date: Tue, 21 Dec 2021 12:51:08 -0500

Phillip Hallam-Baker wrote:
> My analysis is rather different but comes to a stronger statement of 
> John's point.
>
> discussion of how to make mailing lists work better has to begin with 
> the acceptance that they will never work very well which is what most 
> people have been arguing in this thread.
Hmm... seems to me that mailing lists have been a staple of Internet 
world, since about a few days after Ray Tomlinson sent the first ARPANET 
email.  The IETF still runs on email lists.  So do most open source 
projects.  Not to mention business teams.

And for most of those years, it's been basic SMTP based email. Nothing 
has yet supplanted it.  Not NNTP.  Not Atom.  Not the myriad of web forums.

One might better argue that SMTP might be due for some enhancemnet - to 
better support email lists.  (And, in some sense, it has - what with the 
various standard headers for propagating list management information.)

The arguments re. DMARC are that it BREAKS stuff that used to work, 
while not doing much to solve the problem it's intended to solve.

Miles Fidelman

-- 
In theory, there is no difference between theory and practice.
In practice, there is.  .... Yogi Berra

Theory is when you know everything but nothing works.
Practice is when everything works but no one knows why.
In our lab, theory and practice are combined:
nothing works and no one knows why.  ... unknown

---

From: "John Levine" <[email-masked]>
Date: 21 Dec 2021 14:32:12 -0500

It appears that Scott Kitterman  <[email-masked]> said:
>>Real non-existence.  I'm not sure how to define it formally, ...

The DNS has had a formal definition of non-existence for over 30
years. You look up a name, if it returns records or NOERROR it exists,
if it returns NXDOMAIN it doesn't. There is no reason for us to invent
something new and incompatible.

>I don't remember exactly why we settled on A/ AAAA/ MX, but the lack of a clear, actionable definition is why we included one.

See above.  I don't remember where the text in A.4 came from, but it is wrong.
If we are telling people to test whether a domain exists, they should do it
the way the DNS does it.  The correct test happens to be cheaper than A.4,
one query rather than three.

R's,
John

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 21 Dec 2021 16:32:23 -0500

On Tuesday, December 21, 2021 2:32:12 PM EST John Levine wrote:
> >I don't remember exactly why we settled on A/ AAAA/ MX, but the lack of a
> >clear, actionable definition is why we included one.
> See above.  I don't remember where the text in A.4 came from, but it is
> wrong. If we are telling people to test whether a domain exists, they
> should do it the way the DNS does it.  The correct test happens to be
> cheaper than A.4, one query rather than three.

Fair enough.  I can't remember why we thought RFC 8020[1] wasn't adequate when 
we did RFC 9091.  I think we should modify the DMARC text to just refer to it.

Scott K

[1] https://www.rfc-editor.org/rfc/rfc8020.txt

---

From: Dave Crocker <[email-masked]>
Date: Tue, 21 Dec 2021 13:50:02 -0800

> people have been arguing in this thread.

SMTP is a transfer protocol.  It does not know about mailing lists and 
doesn't need to.  (There is text in the SMTP spec about mailing lists, 
but really the text has nothing to do with SMTP, per se, and not a lot 
to do with the operation of mailing lists.

Mailing lists are user-level processes that sit at a layer above SMTP.

> evolution, we should see them for what they are, a rather disgusting 
> hack that we never got round to fixing.

Can't say I've ever noticed anyone suggesting that mailing lists are the 
pinnacle of anything.  On the other hand, efforts to produce more 
interesting capabilities that aren't centralized have not gotten much 
traction.  One keeps hoping, but field experience is not encouraging.

>
> Why is it assumed that the input to a mailing list is an SMTP email? 
> That seems to me to be a rather narrow assumption.

Not everyone makes that assumption.

That said, where is the spec for the alternative and who supports that spec?

> Once we recognize that the inputs and outputs from 'mailing lists' can 
> be through other transports than SMTP, all arguments about preserving 
> 'original' headers collapse. This is not an interaction between an 
> SMTP sender and receiver, the mailing list itself is a principal.

When playing in the sandbox of a particular set of specifications, then 
it's fine -- actually it's essential -- to specify requirements such as 
information preservation.

If the point is that mail can transit heterogeneous environments, with 
different semantics, then the fact of that difference ensures 
information loss.  Absent that assurance, why shouldn't the information 
be preserved, no matter how it is transported?  That is, after all, one 
of the benefits of distinguish the message object specification from the 
message transport specification.

Apologies. I've probably entirely missed your point.

d/

-- 
Dave Crocker
[email-masked]
408.329.0791

Volunteer, Silicon Valley Chapter
Information & Planning Coordinator
American Red Cross
[email-masked]

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 21 Dec 2021 23:23:31 -0800

On Tue, Dec 21, 2021 at 4:31 AM Scott Kitterman <[email-masked]>
wrote:

> clear, actionable definition is why we included one. Lack of DNS records
> related to email authentication only means lack of email authentication,
> which is in no way required.  Given the way most systems are typically
> architected, by the time you are doing email authentication, A or AAAA and
> MX will already be in the local cache, so this is a pretty inexpensive
> thing to check.
>

It comes from SMTP itself.  RFC 5321 and its antecedent(s) specify that to
identify the next hop for a message needing routing, you look up the MX for
the recipient's domain.  If that's missing, you try the A/AAAA for that
same name.  If that's also missing, the message is undeliverable.

Since that test has worked well for SMTP for rather a long time, DMARC
adopted it.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 21 Dec 2021 23:29:01 -0800

On Tue, Dec 21, 2021 at 11:32 AM John Levine <[email-masked]> wrote:

> wrong.

We're talking about two different things here, I think.  The DNS definition
of "nonexistent" is as cited above while the DMARC definition matches the
well established SMTP algorithm that figures out where the next hop for a
particular recipient is.  If there is no next hop, then for email purposes,
the domain doesn't "exist".  The logic goes something like: If this message
fails DMARC, and a bounce has nowhere to go, then I'm pretty sure I don't
want to deliver it.

We're free to change our minds about what test is appropriate here, but
that was the genesis of A.4.

-MSK

---

From: Douglas Foster <[email-masked]>
Date: Wed, 22 Dec 2021 08:24:32 -0500

I am ready to embrace NXDOMAIN as the preferred test, based on two
assumptions which I am hoping the group can endorse.

Scope
The PSD case is a special case of not-valid-for-mail names, and I was
interested in trying to address more than that one case.   A complete list
seems to be:
- Fictitious TLDs
- PSD names, because they do not send mail
- Unregistered and therefore fictitious organization domains, and their
descendants
- Fictitious additions to the set of names used for an organization domain
- Use of names that are only used for non-mail purposes
- Use of names that are used for SMTP but not for RFC5322.From

A first question is whether the domain owner is the only arbiter of what
names are real or fictitious within his space.   This is necessary because
DMARC Fail occurs on messages that I want and allow, even though I know
that they did not originate from the domain owner.   Some US Government web
sites and some email security services are important enough to break the
rules and get away with it.   However, these "acceptable impersonations"
are always using an actual email address.   Similarly, mailing list
messages may produce DMARC Fail, but the subscriber is a valid member of a
valid domain.   So my first assumption is, "Yes, the domain owner has
official control over the definition of valid names within his
organization"

Not all RFC5322.From domains exist in DNS, and not all DNS names are valid
as RFC5322.From domains.  "Valid" becomes a nebulous concept based on the
intent of the domain owner.  While NXDOMAIN is certainly a solution to the
PSD problem, it is not obvious that it is a sufficient solution to the
general problem.

My second assumption is that as an evaluator, I am unlikely to gain any
benefit from checking for invalid names underneath an organization domain.
  If a domain owner ensures that both mail-enabled and non-mail names are
protected using SP=REJECT, then an attacker gains no advantage from
choosing an invalid name over a valid one.   Similarly, if the domain owner
leaves both mail-enabled and non-mail names unprotected using SP=NONE, then
the attacker has no advantage from choosing an invalid name over a valid
one.   To the contrary, it seems advantageous for the attacker to use a
valid name when one is available.    Using an invalid name becomes
advantageous only if the domain owner protects all mail-enabled names with
P=REJECT, but leaves non-mail names unprotected using SP=NONE.  This
configuration seems bizarre to me, and therefore unimportant.
Nonetheless, it still might be useful if all non-mail names could be
determined with confidence, but this requires near-perfect information to
be useful, and experience shows that this is unlikely.

If the second assumption is accepted, then the last three scope topics
disappear.   For the remaining three

- For unregistered and therefore fictitious organization domains. NXDOMAIN
is the simplest and most appropriate test.  We block based on NP=REJECT in
the PSD's DMARC policy when a TXT (or any) name lookup returns NXDOMAIN.
 Organizations within that PSD can avoid false positives easily:  either
publish their own DMARC policy without NP=REJECT, or ensure that all of
their RFC5322.From domains do exist in DNS, even if the entry is as trivial
as Type=TXT, DATA="I Exist".

- For PSDs, we block if the PSD has a DMARC record with the PSD flag.   If
that test is considered insufficient, we block by consulting the
publicsuffix.org list and block names from that list that the evaluator
considers applicable.

- For fictitious TLDS, we have an undiscussed topic.   If all valid TLDs
now participate in DNS SEC, then a DNS lookup on valid TLDs will return
DATA or NOERROR, while fictitious TLDS will always return NXDOMAIN.    If
the PSD flag becomes ubiquitous, then a tree walk which does not produce a
DMARC policy could also be evidence of an invalid TLD.   If PSD Flag usage
is not universal, then a DNS query on the TLD name should be used as a
follow-up to the tree walk.   If some valid TLDs do not participate in DNS
SEC and do not publish the PSD flag, then we are back to consulting a list.

Doug

On Wed, Dec 22, 2021 at 2:29 AM Murray S. Kucherawy <[email-masked]>
wrote:

>> >I don't remember exactly why we settled on A/ AAAA/ MX, but the lack of
>> a clear, actionable definition is why we included one.
>>
> We're talking about two different things here, I think.  The DNS
> definition of "nonexistent" is as cited above while the DMARC definition
> matches the well established SMTP algorithm that figures out where the next
> hop for a particular recipient is.  If there is no next hop, then for email
> purposes, the domain doesn't "exist".  The logic goes something like: If
> this message fails DMARC, and a bounce has nowhere to go, then I'm pretty
> sure I don't want to deliver it.
>