Subject: [dmarc-ietf] Ticket #61 - Define and add a simplified (redacted) failure report

From: Alessandro Vesely <[email-masked]>
Date: Wed, 9 Dec 2020 18:07:50 +0100

We would like to close this ticket by Dec 23, two weeks from now, so please get 
on it.

The ticket text is:

     It has been asked for a new report type (perhaps a subset of failure
     reports) that provides minimal data from the email (specifically, the
     initial ask is for the to: and from: email addresses only) in order to aid
     identification of the email's destination (and hence, the owner who can
     help with getting it authenticated) without providing other PII.

     This is a significant use case for large organizations, where the
     departments or other sub-organizations run their own emailing
     infrastructure. This has been specifically requested by multiple
     universities.

DMARC failure reporting is based on Authentication Failure Reporting Using the 
Abuse Reporting Format (RFC 6591), which in turn is based on An Extensible 
Format for Email Feedback Reports (RFC 5965).  DMARC adds five fields for the 
second MIME part of the report.  The third part can be either the full message 
of just the rfc822-headers.  The latter is defined in The Multipart/Report 
Media Type for the Reporting of Mail System Administrative Messages (RFC 6522), 
which mentions that Received: fields can also be useful for diagnosing 
failures.  In any case, private data such as the local part of email addresses 
can be redacted according to Redaction of Potentially Sensitive Data from Mail 
Abuse Reports (RFC 6590).

In order to be useful, reports should contain enough data to discern whether 
the failed message was abusive, and what stream does it belong to if it wasn't. 
  Should DMARC Failure Reporting (our document) include some guidance about 
what parts of the failed message to include and which ones to redact?

Best
Ale
--

---

From: "John Levine" <[email-masked]>
Date: 9 Dec 2020 13:52:45 -0500

>     help with getting it authenticated) without providing other PII.

As always, I would want to see some evidence of an actual problem to
be solved here. In the existing format, reporters can and do redact as
much as they want.  Why isn't that sufficient?

Looking at the actual forensic reports I get, the majority are from
antispamcloud.com which gloms some report info and the failed
message's headers into a text body, ignoring the spec that says it's
supposed to be multipart/report. I presume if we changed the spec they
still wouldn't follow it, so why bother.

The rest of the reports are multipart/report, some with the whole
message, some with just the headers.

I think that if a reporter isn't willing to provide the headers it's
unlikely to provide anything.  If we have a concrete reason to believe
that there are people who would send these proposed super-redacted
failure reports who do not send reports now, I might consider this.
Otherwise, it's not a problem and close the ticket.

R's,
John

---

From: Brandon Long <[email-masked]>
Date: Wed, 9 Dec 2020 13:29:03 -0800

> Otherwise, it's not a problem and close the ticket.

In today's much more privacy conscious world, should we have RUF reports in
DMARC
at all?

Brandon

---

From: "John R Levine" <[email-masked]>
Date: 9 Dec 2020 16:39:41 -0500

yOn Wed, 9 Dec 2020, Brandon Long wrote:
>> unlikely to provide anything.

> DMARC at all?

It's a reasonable question, but a company in Redmond WA is sending me 
unredacted reports every daay so presumably someone thinks they are OK.

I agree they're a lot less popular than aggregate reports, and don't see 
any reason to spend more time on them.  That's why I didn't propose an 
http version of ruf reports.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Jesse Thompson <[email-masked]>
Date: Wed, 9 Dec 2020 17:48:12 -0600

On 12/9/20 11:07 AM, Alessandro Vesely wrote:
> We would like to close this ticket by Dec 23, two weeks from now, so please get on it.
> 
> The ticket text is:
> 
> DMARC failure reporting is based on Authentication Failure Reporting Using the Abuse Reporting Format (RFC 6591), which in turn is based on An Extensible Format for Email Feedback Reports (RFC 5965).  DMARC adds five fields for the second MIME part of the report.  The third part can be either the full message of just the rfc822-headers.  The latter is defined in The Multipart/Report Media Type for the Reporting of Mail System Administrative Messages (RFC 6522), which mentions that Received: fields can also be useful for diagnosing failures.  In any case, private data such as the local part of email addresses can be redacted according to Redaction of Potentially Sensitive Data from Mail Abuse Reports (RFC 6590).
> 
> In order to be useful, reports should contain enough data to discern whether the failed message was abusive, and what stream does it belong to if it wasn't.  Should DMARC Failure Reporting (our document) include some guidance about what parts of the failed message to include and which ones to redact?

During our DMARC deployment (and even today) I was frustrated that I could see from aggregate reports that an ESP (or some other type of hosting provider whereby the IP address alone wasn't identifiable enough) was using our domain, but I didn't know who within our organization was a customer of the ESP.  Ultimately, I wanted to reach out to the customer and set them up with a subdomain to use with the ESP so they could send branded and DMARC SPF&DKIM-aligned email.  Asking the ESP who they're allowing to use our domain was a dead-end because they considered that private customer information (irony!) and they're naturally inclined to make their customers happy by tolerating the use of the domain without complicating matters that may result in possibly losing the customer.

Forensic reports are a useful mechanism to find examples of these messages and subsequently track down the customer on our end.  Usually, seeing the local part of the address used in the From header (maybe coupled with the Subject) was enough information.  I did struggle with the needle-haystack problem with these reports.  It was challenging to find what I was looking for, and it was challenging to create a process for going through all of the reports in any meaningful way.  Perhaps I didn't really find the right tool for the job, or perhaps the problem wasn't large enough for me to justify finding/developing the right tool.

Alternatives:

- Assuming that all ESPs were cooperative to inquiries from domain owners, and there was a consistently-used internal identifier that could be referenced that doesn't directly disclose publicly identifiable information about the customer, then that might be all of the forensic information that I would need.  But fall back to From/Subject, otherwise.  

- SPF macros can be constructed such that the envelope information is sent back to the domain owner via DNS queries.  This tactic is handy only if there is something identifiable in the local part of the return path.  The downside to this tactic is that you're spraying unencrypted identifiable information across the internet.

I guess that I'm leaning towards encouraging redacted reports with some minimally identifiable information.  The reporters still have the option to not send reports (or redact however they see fit), and domain owners have the option to not request reports if they are concerned about privacy.  

The potential downside to not having failure reports at all is that potentially less secure/private mechanisms may be adopted due to demand (such as SPF macros, sketchy data warehousers, etc). 

I did purposefully ignore the use case for using DMARC failure reports to track direct domain abuse, which I assume is a valid use case and should be considered in more depth.  I just don't have a direct experience using that data in any meaningful way.  Perhaps it could be used in a way that articulates the *actual* amount of direct domain spoofing being used by malicious actors, as opposed to aggregate reports which only gives you IP addresses (which may be enough if cross referenced with reputation lists).  Obviously, if you rely on the full details of the failure report to build anti-spam heuristics, then you wouldn't want the reports redacted.  But if you are in that business, I assume that spam traps work better anyway.

>From the original ticket, I am failing to completely understand why "aid identification of the email's destination" is more useful than identifying the [true] origin.  I suppose it could be useful for organizations who want to validate the use case (make sure they aren't sending unsolicited messages, etc) before adding another server to SPF (without talking to them?).  That's just not something we do, since we have a Holy Roman Empire and let people do whatever they want with their own subdomains.

Jesse

---

From: Brandon Long <[email-masked]>
Date: Wed, 9 Dec 2020 16:12:16 -0800

On Wed, Dec 9, 2020 at 3:48 PM Jesse Thompson <jesse.thompson=
[email-masked]> wrote:

> > DMARC failure reporting is based on Authentication Failure Reporting
> Using the Abuse Reporting Format (RFC 6591), which in turn is based on An
> Extensible Format for Email Feedback Reports (RFC 5965).  DMARC adds five
> fields for the second MIME part of the report.  The third part can be
> either the full message of just the rfc822-headers.  The latter is defined
> in The Multipart/Report Media Type for the Reporting of Mail System
> Administrative Messages (RFC 6522), which mentions that Received: fields
> can also be useful for diagnosing failures.  In any case, private data such
> as the local part of email addresses can be redacted according to Redaction
> of Potentially Sensitive Data from Mail Abuse Reports (RFC 6590).
> >
> > In order to be useful, reports should contain enough data to discern
> whether the failed message was abusive, and what stream does it belong to
> if it wasn't.  Should DMARC Failure Reporting (our document) include some
> guidance about what parts of the failed message to include and which ones
> to redact?
>
> During our DMARC deployment (and even today) I was frustrated that I could
> see from aggregate reports that an ESP (or some other type of hosting
> provider whereby the IP address alone wasn't identifiable enough) was using
> our domain, but I didn't know who within our organization was a customer of
> the ESP.  Ultimately, I wanted to reach out to the customer and set them up
> with a subdomain to use with the ESP so they could send branded and DMARC
> SPF&DKIM-aligned email.  Asking the ESP who they're allowing to use our
> domain was a dead-end because they considered that private customer
> information (irony!) and they're naturally inclined to make their customers
> happy by tolerating the use of the domain without complicating matters that
> may result in possibly losing the customer.
>
> Forensic reports are a useful mechanism to find examples of these messages
> and subsequently track down the customer on our end.  Usually, seeing the
> local part of the address used in the From header (maybe coupled with the
> Subject) was enough information.  I did struggle with the needle-haystack
> problem with these reports.  It was challenging to find what I was looking
> for, and it was challenging to create a process for going through all of
> the reports in any meaningful way.  Perhaps I didn't really find the right
> tool for the job, or perhaps the problem wasn't large enough for me to
> justify finding/developing the right tool.
>

Of course they are useful.  There are a lot of potentially privacy invasive
things that are useful.

At best, we considered allowing RUF reports for cases where the dmarc
domain was the receiver, ie if someone had a message that failed dmarc
while sending to the same domain, then presumably the domain admin already
has the power to view the message.

But, if you limit it to that level, then the domain admin could already
theoretically do this themselves by having those messages delivered to some
destination to look at them, or setting
up their own forwarding rule to their dmarc analysts.

Brandon

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 9 Dec 2020 20:18:37 -0800

On Wed, Dec 9, 2020 at 1:29 PM Brandon Long <blong=
[email-masked]> wrote:

> In today's much more privacy conscious world, should we have RUF reports
> in DMARC
> at all?
>

Forensic reports in DMARC are akin to the DKIM failure reporting we added
to ARF back in the MARF working group.  In fact if you go back and read
those RFCs, an ancestor to the "pct=" tag is there. (RFC 6591 and RFC 6651
in particular are what I'm looking at.)

Back in the original DKIM era I always found this kind of reporting to be
really valuable especially since DKIM can fail for a variety of reasons
that are far less obvious than SPF.  Being able to get the verifier to tell
me exactly what it saw and compare it to what I think I sent was key to
getting the implementation right especially in curious corner cases (any of
you that remember the DKIM interop event would know what I mean).

Seems to me that's still a useful thing to have, at least sometimes.  We
might say something like: Include support for this, but don't have it on by
default.  Or even if it's an extension to DMARC and not part of the base
protocol, it might be really helpful in some situations.

-MSK

---

From: Seth Blank <[email-masked]>
Date: Wed, 9 Dec 2020 20:28:55 -0800

> protocol, it might be really helpful in some situations.

Speaking as a report processor, never in all my years of helping domain
owners use dmarc reports to authenticate their mail has this ever been
necessary.

As an individual, I feel extremely strongly that failure reports should go
away in their entirety. Although Jesse Thompson has outlined a use case
that really has no other good solution, and is equally true in other large
complicated organizations. However, this problem is mitigated with a tree
walk, where you can get the reports to the appropriately localized part of
the organization (like, math.wisc.edu, instead of to Jesse at wisc.edu
who'd need to hunt things down).

As Chair, I've seen relatively strong opinions that people wish for failure
reports to remain. More conversation here is useful.

For this ticket in particular-- the simplified failure report with only
from: and to: addresses speaks to Jesse's exact use case, without any of
the other PII that tends to get failure reports in privacy trouble (like
body content and attachments). This approach to Jesse's use case should get
a fair discussion, separate from whether we want failure reports at all.

Seth

-- 

*Seth Blank* | VP, Standards and New Technologies
*e:* [email-masked]
*p:* 415.273.8818

This email and all data transmitted with it contains confidential and/or
proprietary information intended solely for the use of individual(s)
authorized to receive it. If you are not an intended and authorized
recipient you are hereby notified of any use, disclosure, copying or
distribution of the information included in this transmission is prohibited
and may be unlawful. Please immediately notify the sender by replying to
this email and then delete it from your system.

---

From: "John Levine" <[email-masked]>
Date: 10 Dec 2020 00:44:06 -0500

>a fair discussion, separate from whether we want failure reports at all.

Having sat in on far too many GDPR discussions, I'm sure that the To
and From addresses are exactly the kind of PII that makes lawyers
nervous. Keep in mind that there is no guarantee that the entitity
getting the reports has any responsibility for either, particularly if
a third party is collecting the reports.

I don't think this group has any particular expertise in the issues
that are likely to make organizations decide whether there is legal
risk in sending the reports and whether and how much to redact them. I
would leave whole ruf section alone other than perhaps making clearer
that the reports are optional and it may be useful to send even if
they are heavily redacted. As an example, a report with the Message-ID
but no To or From might still be enough for the report recipient to
figure out where a message came from while not disclosing any PII.

R's,
John

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 10 Dec 2020 10:42:44 +0100

On Thu 10/Dec/2020 05:28:55 +0100 Seth Blank wrote:
> On Wed, Dec 9, 2020 at 8:19 PM Murray S. Kucherawy wrote:
>> On Wed, Dec 9, 2020 at 1:29 PM Brandon Long wrote:
>>
>>> in DMARC at all?
>> [...]

Can we be explicit about that?  I mean to suggest to develop but not to enable. 
  Furthermore, I'd recommend to develop options to enable failure reports on a 
per-domain basis.  (We could also mention that admins may contact the email in 
the <report_metadata> section of troublesome aggregate reports to ask for 
failure reports to be enabled for their domain for the time necessary to solve 
the problem at hand.)

> away in their entirety.

Could we at least limit them to a single, must-be-aligned recipient?

> a fair discussion, separate from whether we want failure reports at all.

I would suggest to redact the local parts of To:, Cc: and similar fields 
(X-Original-To:, Received: for), possibly leaving only the From: intact. 
Delivered-To: should be removed.  The rest of the header can be sent safely, 
methinks.

Best
Ale
--

---

From: Jesse Thompson <[email-masked]>
Date: Thu, 10 Dec 2020 10:18:50 -0600

On 12/9/20 6:12 PM, Brandon Long wrote:
> At best, we considered allowing RUF reports for cases where the dmarc domain was the receiver, ie if someone had a message that failed dmarc while sending to the same domain, then presumably the domain admin already has the power to view the message.
> 
> But, if you limit it to that level, then the domain admin could already theoretically do this themselves by having those messages delivered to some destination to look at them, or setting

I agree with this.  With senders that have a relatively large volume, there is a very high probability that at least one of the recipients is local.  Most of the necessary information is probably already available in logs.

Jesse

---

From: Jesse Thompson <[email-masked]>
Date: Thu, 10 Dec 2020 10:59:54 -0600

On 12/9/20 10:28 PM, seth=[email-masked] wrote:
> As an individual, I feel extremely strongly that failure reports should go away in their entirety. Although Jesse Thompson has outlined a use case that really has no other good solution, and is equally true in other large complicated organizations. However, this problem is mitigated with a tree walk, where you can get the reports to the appropriately localized part of the organization (like, math.wisc.edu <http://math.wisc.edu>, instead of to Jesse at wisc.edu <http://wisc.edu> who'd need to hunt things down).

To be clear, I'm not advocating for this From/To use case, but I know of universities who would.  There's at least one who cleverly flattened their SPF record to include every known sender specifically because they had no mission to change the way their institution distributively sends email.  That is the type of organization who *may* want From/To data, assuming they want to do more validation before adding yet another IP to their SPF.  It's a stretch in my mind.

I would only strongly advocate for seeing the unredacted From header, since my primary concern was with identifying a little bit more information about who was using the domain and why (i.e. who is using random ESP). 

The stated purpose of Failure Reports is "for quickly notifying the Domain Owners when there is an authentication failure ... also provide more information about the failed message than is available in an aggregate report".  Since the focus of DMARC is to authorize the use of the domain used in the From header, then the logical next incremental levels of "more information" should be:

1) From header domain (already known)
2) local part of From address
3) Friendly From
4) Subject
5) other parts of the message containing identifiable information

1->5 decreases in usefulness/relevancy to DMARC
1->5 increases in unnecessary information disclosure

Jesse

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 11 Dec 2020 14:48:09 +0100

On Thu 10/Dec/2020 17:59:54 +0100 Jesse Thompson wrote:
> On 12/9/20 10:28 PM, seth wrote:
>> As an individual, I feel extremely strongly that failure reports should go away in their entirety. Although Jesse Thompson has outlined a use case that really has no other good solution, and is equally true in other large complicated organizations.
> 

I'm not clear whether you're talking about sending or receiving reports.  Received: chains are key for evaluating addresses to add to SPF records.  I don't think it makes sense to specify their omission from 

My guess at why an organization may want to send only From/To rather than a possibly redacted text/rfc822-headers are as follows:

* It is too hard for them to asses the risk of including unknown header fields, yet they must do it before enabling ruf,

* the software they use doesn't have an option to redact PII, or (unlikely)

* they try to save bandwidth and disk space by reducing that ghastly pile of freaky fields that infest message headers these days.

> I would only strongly advocate for seeing the unredacted From header, since my primary concern was with identifying a little bit more information about who was using the domain and why (i.e. who is…

Agreed.

> 1->5 increases in unnecessary information disclosure

The mail filter I do sends aggregate reports but not failure reports.  Should I add it?  I'm thinking I could frame it like so:

* Have a global flag to enable or disable ruf's, which can be overridden on a per-domain basis.  Default to disabled.

* The flag can specify three confidentiality levels:
  - full message
  - header only
  - header only + redact.

* Redacting the header would work as follows:
  1. Collect recipients addresses in To:, Cc:, and envelope,
  2. compute the rfc6590-redacted version of each address,
  3. for all fields, replace any occurrence with the redacted version.

* Reports are left in a user-configured directory, assuming that a user supplied script deals with them.

Does that make sense?

Should dmarc-failure-reporting include text with practical suggestions along those lines?

Best
Ale
--

---

From: Jesse Thompson <[email-masked]>
Date: Fri, 11 Dec 2020 11:24:16 -0600

> I'm not clear whether you're talking about sending or receiving reports.  Received: chains are key for evaluating addresses to add to SPF records.  I don't think it makes sense to specify their omi…

Only the last hop from the receiver's viewpoint is really needed for a domain owner to assess gaps in SPF.  That's already addressed with aggregate reports.  What's missing is the additional context that the failure reports are intended to provide, to help domain owners determine if the IP address which sent the message was legitimate.  Of course, received chains are helpful for troubleshooting.  There's no "right" answer regarding the balance of useful information vs privacy, so I'm suggesting that we put things like this on a spectrum of usefulness/privacy-concerning. 

> * they try to save bandwidth and disk space by reducing that ghastly pile of freaky fields that infest message headers these days.

If sending only a limited amount of information is considered an acceptable alternative to full/unredacted information, it might help encourage these organizations to send failure reports, perhaps?

> Should dmarc-failure-reporting include text with practical suggestions along those lines?

Does this redaction logic apply to the From header too?  If so, I would recommend adding a fourth confidentiality flag for reporters who want to redact recipient information but leave the From header unredacted to better aid the domain owner in tracking down the sender.

Jesse

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 12 Dec 2020 11:36:41 +0100

> Only the last hop from the receiver's viewpoint is really needed for a domain owner to assess gaps in SPF.  That's already addressed with aggregate reports.  What's missing is the additional contex…

Thus far, the spec doesn't discuss what to redact, except for mentioning local 
parts of email addresses.

> If sending only a limited amount of information is considered an acceptable alternative to full/unredacted information, it might help encourage these organizations to send failure reports, perhaps?

I'm surprised that the request in this ticket proposes to send only the two 
most sensitive fields.

Best
Ale
--

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 31 Dec 2020 13:29:36 +0100

On Thu 10/Dec/2020 00:48:12 +0100 Jesse Thompson wrote:
> [...snipped Alternatives:...]

I'm a bit lost about where we are at this point.

The possibility to redact parts of the report is already mentioned in the draft.

We could add the generic possibility to select which header fields to include 
in the report.  For example, the draft might suggest to only include known 
header fields.  That way one could drop all X-* fields with unknown binary 
content and any other scaring stuff.  If one wants to only leave From: and To:, 
so be it.  Their reports, their choice.

Otherwise we can just close this ticket with WONTFIX.

Opinions?  Other ideas?

Best
Ale
--

---

From: "John Levine" <[email-masked]>
Date: 31 Dec 2020 11:01:58 -0500

In article <[email-masked]> you write:
>>>     reports) that provides minimal data from the email ...

>Otherwise we can just close this ticket with WONTFIX.

Please do that. It was clear that we have nothing useful to say about
what one might want to redact or be required to redact.

R's,
John

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 31 Dec 2020 18:44:57 +0100

> what one might want to redact or be required to redact.

Done.

Best
Ale
--