Subject: Re: DMARC methods in mailman

From: Philip Homburg <[email-masked]>
Date: Tue, 20 Dec 2016 12:06:01 +0100

Just a random idea, I don't think I have seen it before, but I may have missed
it.

What about adding a checkbox in the mailman profile for subscribers of the
lists that basically says: if you detect that the sender is using DMARC then
please rewrite the From header.

This way everbody who is completely ignoring the silly DMARC stuff sees no
difference. And everybody who uses DMARC for scoring can still see the original
from.

People who receive the list on a yahoo account would have to mark the checkbox
(though maybe a list of default domains can be hacked in such that new
subscribers for those domain would get it set automatically)

It's a bit of coding, but it should result in almost no damage except in the
specific case of somebody on yahoo (or similar) receiving a post from somebody
from a DMARC protected domain.

---

From: "John Levine" <[email-masked]>
Date: 20 Dec 2016 19:56:01 -0000

>please rewrite the From header.

Whether a mail system enforces DMARC policies is up to the system's
manager, not individual users.  Even in the IETF, I expect that many,
perhaps a majority, of users have only the dimmest idea what if
anything their operator does with DMARC.

R's,
John

---

From: Theodore Ts'o <[email-masked]>
Date: Tue, 20 Dec 2016 17:35:44 -0500

> anything their operator does with DMARC.

This starts getting more complicated, but the mailing list could try
intentionally sending a message which fails DMARC (e.g., with a
claimed from address of [email-masked]) to the mailing list
subscriber and see if it gets bounced, and use that to set the default
for that setting automatically?

						- Ted

---

From: Randy Bush <[email-masked]>
Date: Wed, 21 Dec 2016 07:57:41 +0900

has anyone looked at what nanog, who discussed this as interminably as
ietf, is doing?  it mangles the From: only on dmarc mail.  i have
avoided learning the details.

randy

---

From: Philip Homburg <[email-masked]>
Date: Wed, 21 Dec 2016 00:26:55 +0100

>for that setting automatically?

Typically, a user whose mail system rejects too many e-mails is automatically
unsubscribed.

There are a range of possibilities, from just adjusting the boiler plate in
the unsubscribe notice to automatically turning the option on when a
reject involves a DMARC protected sender.

As for Randy's suggestion. That would be equivalent to turning the option
on for everybody. That may also be a sensible default. People who want to
reply directly can turn the option off manually.

Just always rewriting DMARC protected From headers is suboptimal for
recipients that don't want to have anything to do with DMARC. Why subject
them to broken From headers?

---

From: "John R Levine" <[email-masked]>
Date: 20 Dec 2016 18:52:44 -0500

> for that setting automatically?

Without exception, every suggestion made in this discussion has already 
been brought up and addressed on the Mailman developers' list, or the 
DMARC discussion list at dmarc.org.  Perhaps people who find this topic to 
be of interest could learn more faster and save everyone else's time by 
reading the archives.

Among the reasons this particular approach won't work is that at some mail 
providers, notably Gmail, the DMARC treatment varies both by sender and by 
recipient.  Sometimes they reject, sometimes they accept and deliver, 
sometimes they accept and file as spam.  Whatever they do to a test 
message may or may not be what they do to real messages.  Furthermore, 
providers change their DMARC mail handling every day (which as, we may 
recall, is why we have this problem in the first place.)  If an operator 
ignores DMARC failures today, they may do something else next week or next 
year.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: S Moonesamy <[email-masked]>
Date: Wed, 21 Dec 2016 02:57:17 -0800

Hi Philip,
>please rewrite the From header.

There is already code in Mailman, e.g. if not 
(msgdata.get('from_is_list') ... to do the above.  The configuration 
can be set by AMS.

>subscribers for those domain would get it set automatically)

The focus has been on Yahoo because of a message which was posted 
several years ago.  I collected some statistics for an IETF mailing 
list.  Approximately 40% of the subscribers are using [1] DMARC; most 
of then are not using Yahoo.

Regards,
S. Moonesamy

1. This is not the percentage of subscribers using "p=reject".

---

From: Philip Homburg <[email-masked]>
Date: Wed, 21 Dec 2016 13:10:16 +0100

>(msgdata.get('from_is_list') ... to do the above.  The configuration 

Just to be clear, is this an option that individual subscribers of the mailing
list can set?

For me, the current IETF lists work fine. My MTA completely ignores DMARC,
so it would be nice not to be subjected to mangled From headers.

>of then are not using Yahoo.

Are you talking about senders or recipients? The reason yahoo stands out is
because yahoo seems to be one of the few providers that rejects or bounces
mail that fails DMARC checks.

Or are you saying that at the moment 40% of the subscribers of IETF lists reject
or otherwise not receive mail from DMARC protected senders?

---

From: Theodore Ts'o <[email-masked]>
Date: Wed, 21 Dec 2016 10:12:18 -0500

> or otherwise not receive mail from DMARC protected senders?

It's much more likely that this is the percentage of subscribers that
are sending from domains that are claiming a DMARc policy.  What's
interesting is how few of these domains are actually *following* the
DMARC specification in rejecting, unconditionally, e-mails which fail
the DMARC checks.

Which to me shows how defective by design DMARC really is; even many
of the proponents of DMARC are running mail systems which are not
honoring the DMARC "specification".

Given that the DMARC "specifcation" isn't even being treated as a
standard that must be obeyed in all of its particulars by its
proponents --- the fact that this is being used by its propoonents to
twist mailers of the IETF --- a standards body --- into knots because
it is enforcement is random and *not* standardized is, quite frankly,
amazing to me.

Let the employees of those companies which are proponents of DMARC
suffer.  And they might not be suffering that much given that at least
one of those companies isn't really enforcing DMARC on the receiving
end for their users....

					- Ted

---

From: S Moonesamy <[email-masked]>
Date: Wed, 21 Dec 2016 07:22:13 -0800

Hi Philip,
>Just to be clear, is this an option that individual subscribers of the mailing

No.  You can view the options available to individual subscribers at 
http://www.elandsys.com/~sm/test.png

>so it would be nice not to be subjected to mangled From headers.

I have not encountered any issue.  I would have opened a ticket if 
there was an issue.

>mail that fails DMARC checks.

I was talking about recipients.

>or otherwise not receive mail from DMARC protected senders?

The 40% was statistics about one IETF mailing list only.  Messages 
from the mailing list to those (40%) subscribers would not be 
rejected as some of the mail providers are not advertising a 
"p=reject".  A number of the mail providers in that 40% will not 
deliver messages from the mailing list to the subscribers as they are 
advertising a "p=quarantine".

Regards,
S. Moonesamy

---

From: Philip Homburg <[email-masked]>
Date: Wed, 21 Dec 2016 17:09:32 +0100

>http://www.elandsys.com/~sm/test.png

The point of my proposal was to make an option per subscriber.

The most sane way to deal with systems that reject or otherwise discard mail
that fails DMARC checks is to rewrite the From.

However, for other people, is a regression for various reasons.

So the only sensible way forward is to have a per subscriber option. 

>advertising a "p=quarantine".

So is that one mailing list typical, or just a bunch of people who like to 
reject mail?

But apart from that, you are saying that for at least that one list, if
you send a mail to the list with a 'p=reject' then 40% of the subscribers are
not going to see that message.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 21 Dec 2016 16:18:22 +0000

>> At 04:10 21-12-2016, Philip Homburg wrote:
  (snip)
> not going to see that message.
This is not correct, SM said that 40% of subscribers publish DMARC 
policy. See my other email on specific numbers.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 21 Dec 2016 16:19:38 +0000

(snip)
> 1. This is not the percentage of subscribers using "p=reject".

I ran some scripts on a snapshot of list of subscribers a week ago and 
here is some information:

    Total addresses in the whitelist: 83131
    Number of unique domains:         19668

    Total domains which published any valid DMARC policy: 1398

    Total domains with p=reject DMARC policy: 243
    Total domains with p=quarantine DMARC policy: 167

    Email addresses with p=reject DMARC policy: 4773
    Email addresses with p=quarantine DMARC policy: 1370

---

From: S Moonesamy <[email-masked]>
Date: Wed, 21 Dec 2016 09:52:51 -0800

Hi Philip,
>The point of my proposal was to make an option per subscriber.

Ok.

>that fails DMARC checks is to rewrite the From.

Ok.

>So the only sensible way forward is to have a per subscriber option.

A per-subscriber option makes matters easier.

>So is that one mailing list typical, or just a bunch of people who like to

Alexey posted some statistics about all the IETF mailing lists at 
https://www.ietf.org/mail-archive/web/ietf/current/msg100242.html

That one mailing list is not typical.  I don't have access to the 
data at the SMTP level to comment about messages from the mailing 
list which are rejected by receiving mail servers.

>not going to see that message.

No.  I am saying that 40% of the subscribers are receiving messages 
from the mailing list through mail systems which advertise DMARC 
policies for the domain names which those subscribers are using.

Regards,
S. Moonesamy

---

From: "Christian Huitema" <[email-masked]>
Date: Fri, 23 Dec 2016 14:17:04 -0800

On Wednesday, December 21, 2016 7:12 AM, Theodore Ts'o wrote:

>> ...
> are sending from domains that are claiming a DMARC policy.  What's

I think that Ted is making a very strong point here. For the IETF mailing
list, the problem does not start when some company publishes a DMARC
"p=reject" policy. The problem occurs when some mail providers decides to
unconditionally bounce messages. These bounces are causing mail to be lost,
and subscribers to be disconnected from mailing list.

We have at least one big provider, Gmail, implementing a sane compromise:
reject the email if you do not have reasons to otherwise trust it. That is,
use some logic instead of bone headed unconditional behavior. From a user
point of view, that may mean having to fish specify that they "trust
[email-masked]", maybe after fishing some messages out of the spam
folder. Not great, but much more light weight than most of the proposed
alternatives. So, in short, we have a solution with demonstrated working
code.

Maybe the IETF should document this expected behavior as part of the mailing
list submission process. "If you want to participate in the IETF, your mail
provider should allow you to receive messages from the IETF."

-- Christian Huitema

---

From: "John Levine" <[email-masked]>
Date: 23 Dec 2016 17:42:24 -0000

>provider should allow you to receive messages from the IETF."

As I keep asking, how much blood are we willing to lose to cut off our
noses to spite our faces?

Pretty much all of the largest mail providers in the world check DMARC
on inbound mail.  Gmail is somewhat more flexible than others but even
they reject or junk a lot of it.

R's,
John

---

From: Randy Bush <[email-masked]>
Date: Sat, 24 Dec 2016 15:25:38 +0900

> Christian said:
>> Maybe the IETF should document this expected behavior as part of the
>> mailing list submission process. "If you want to participate in the
>> IETF, your mail provider should allow you to receive messages from
>> the IETF."
> 

not much.  on the other hand, we were not discussing noses or faces.  so
i'll pass on the negative hyperbole.

i'm with christian.  we have a lot of qualifications to play, from ipr
to nomcom qualification, to standards-compliant smtp servers to get your
mail.  christian's statement was sufficiently simple that you'll excuse
me for repeating it will modification.

"If you want to participate in IETF mailing lists, you should be able to
receive messages from and send messages to the IETF which follow
Internet standards."

randy

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 24 Dec 2016 13:45:12 -0500

> On Dec 24, 2016, at 1:25 AM, Randy Bush <[email-masked]> wrote:
> 

Sure, but, ultimately, what's the goal of this discussion?  Presumably
some workable configuration of the list managers that serves the needs
of IETF list members (possibly with some adjustment of how participants
interact with the lists).

So, if this discussion is to get anywhere, there needs to emerge a
consensus on the usability requirements for the lists.  At least
the following need to be understood:

   1. Must WG lists add [WG] subject tags?  Or can this be left to
      subscriber etiquette?

   2. Are message footers are a requirement?

   3. Is modification of the "From" header acceptable?

Discussion of whether to exclude subscribers from domains with
DMARC policy are not necessary if some acceptable combination
of the choices in 1, 2, 3 (or perhaps other options I left out)
works with DMARC and retains list usability.

As I mentioned before, for me, the most usable option is avoiding
message modification of any kind, which leaves the origin DKIM
signature valid.  I've not seen anyone comment on whether that's
workable for IETF WG lists (it works well enough for *this* list).

-- 
	Viktor.

---

From: Philip Homburg <[email-masked]>
Date: Sun, 25 Dec 2016 00:14:45 +0100

>workable for IETF WG lists (it works well enough for *this* list).

For me, the current behavior of the ietf lists is perfectly fine.

It is very nice to see the mailing list in the subject. It is also
very nice to have a proper From header.

That's why I proposed a per-subscriber setting such that only those
who are actually stupid enough to reject or otherwise drop mail based
on DMARC have to see mangled mailing list traffic.

There no technical reason for a one-size-fits-all solution. So let's
stop investigating those types of solutions.

---

From: Yoav Nir <[email-masked]>
Date: Sun, 25 Dec 2016 13:11:00 +0200

> On 25 Dec 2016, at 1:14, Philip Homburg <[email-masked]> wrote:
> 

For me it’s not fine. And not just because I use gmail. If someone whose email provider has a DMARC p=reject record sends a message to the list and half the subscribers don’t see that message, the conversation gets disrupted even if the other half sees it. 

Check out the mailing list for Token Binding working group. The most frequent posters are from Microsoft and Google, and I have to fish their messages out of the spam folder every few days to keep up with the conversation. That disrupts the entire working group, not just the people with the DMARC record or the people whose provider respects DMARC headers.

> on DMARC have to see mangled mailing list traffic.

“Stupid”?  Most of us have the choice of a corporate mail account where we have no control over policy, and a public provider mail account where we also have no control over policy. Yes, we can install our own mail server and set whatever policy we would like. That is not a viable option for most people

> stop investigating those types of solutions.

 A per-receiver setting is one extra step for new subscribers. That’s something I’d rather avoid.

Yoav

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 25 Dec 2016 13:05:59 -0500

> On Dec 25, 2016, at 6:11 AM, Yoav Nir <[email-masked]> wrote:
> 
> 

More importantly, recipients don't always know whether they need anti-DMARC
armour or not, and are neither responsible for nor consulted on potential
changes in the receiving domain's policies.  Therefore, per-recipient policy
is unlikely to work well.

If message (subject and/or body) modification is a hard requirement, then
it seems that for now anti-DMARC measures are needed in the "From:" header.
FWIW, my view is that forgoing message modification is better than From
mangling.

The need for email origin authentication to specify that "Sender" preempts
"From" has been well understood for a long time before there there was DMARC.
If there is to be a non-broken replacement, it must correct this design error
and place the "burden" of dealing with that on any MUAs that fail to display
Sender (as e.g. from <sender> on behalf of <author>).

-- 
	Viktor.

---

From: tom p. <[email-masked]>
Date: Mon, 26 Dec 2016 10:31:35 +0000

---- Original Message -----
From: "Viktor Dukhovni" <[email-masked]>
To: "IETF general list" <[email-masked]>
Sent: Saturday, December 24, 2016 6:45 PM

> workable for IETF WG lists (it works well enough for *this* list).

The most attractive mail lists for me are ones that always modify
messages, namely removing the mountain of HTML and leaving just the
text/plain.  I wish that all WG Chairs exercised that option.  Only
today, on another list, I saw a well-respected contributor say
"Oh, how I hate HTML emails!"
Yes.

I note too that e-mails in unusual dialects appear to get footers added
in text/plain, to identify which list they are on.

I believe that this does not invalid the origin DKIM signature, so I see
the criterion as always modifying messages but not in a way that
invalids DKIM signatures.

Tom Petch

---

From: "Patrik Fältström" <[email-masked]>
Date: Mon, 26 Dec 2016 12:10:42 +0100

> invalids DKIM signatures.

Bingo! What should have been some kind of requirement is to define the method of signing the content of portions of a message with some BCP for mailing lists.

The stress, if I may call it that, today for DKIM is that the signing is not in sync with how mailing lists are run.

   paf

---

From: Philip Homburg <[email-masked]>
Date: Mon, 26 Dec 2016 13:33:58 +0100

>FWIW, my view is that forgoing message modification is better than From

If we are to assume that suscribers to technical mailing lists like the IETF
are stupid enough to pick mail providers that drop mail based on
non-standard mail extension like DMARC, without being aware that they
did so, them I'm also fine with a knob that by default mangles the
>From header as long as I can turn it off for myself.

Breaking mailing list for everybody just because as small group is stupid
enough to use mailing breaking mail-extensions is extremely bad.

Note I don't care if your employer uses DMARC, use a personal account then.
I don't care if your free e-mail comes with idiotic filtering, subscribe
to the lists from a sensible mail account then.

In any case, just because there is a small group of people that are
set on using non-standard extenisons, or to ignorant or lazy to avoid them,
that's not a reason to break mailing lists for everybody.

>Sender (as e.g. from <sender> on behalf of <author>).

You are saying that the IETF has any infuence over a specification that
came in through the independent stream editor?

---

From: Ted Lemon <[email-masked]>
Date: Mon, 26 Dec 2016 09:11:50 -0500

On Dec 26, 2016, at 7:33 AM, Philip Homburg <[email-masked]> wrote:
> are stupid enough

Perhaps instead of casting aspersions on our fellows, we should _stop_ making assumptions about why they use email addresses provided by operators that behave in ways we don’t like.

---

From: Theodore Ts'o <[email-masked]>
Date: Mon, 26 Dec 2016 09:49:01 -0500

> Sender (as e.g. from <sender> on behalf of <author>).

But if MUA's do this, then it becomes trivial to phish consumers,
which was the original excuse for DMARC.  So if MUA's do this,
eventually Yahoo and the other big mail providers will promulgate a
non-standard "fix" that will bounce message with Sender lines that
aren't equal to the From field.   And then what will you do?

Hint: stop using mail providers that obey non-standard mail protocols,
because they *will* break you eventually, and/or randomly.

					- Ted

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Dec 2016 11:08:13 -0500

Just another thought along the same lines.  If one is trying to
design an MUA (or web browser) to reduce phishing risk, the
following two design decisions should be high on the list of
things to avoid doing:

(i) If a message arrives with a head field of 
    From: Santa Claus <[email-masked]>
display
    From: Santa Claus 
to the user, presumably on the grounds that actual mailboxes are
ugly and a bad user experience.

(ii) If, in the middle of an HTML page, a construction appears
like:
   <a href="http://evil.example/satanic-being">
https://santa-claus.example.com/ </a>
display
   https://santa-claus.example.com/
to the user without any comment.

As long as those implementing popular MUAs and browsers are
doing both of those things, it is really hard to hear about the
problems DMARC is supposedly solving.

   john

--On Monday, December 26, 2016 9:49 AM -0500 Theodore Ts'o
<[email-masked]> wrote:

>> The need for email origin authentication to specify that
>> "Sender" preempts "From" has been well understood for a long
>> time before there there was DMARC. If there is to be a
>> non-broken replacement, it must correct this design error and
>> place the "burden" of dealing with that on any MUAs that fail
>> to display Sender (as e.g. from <sender> on behalf of
>> <author>).
> 
> But if MUA's do this, then it becomes trivial to phish
> consumers, which was the original excuse for DMARC.  So if
> MUA's do this, eventually Yahoo and the other big mail
> providers will promulgate a non-standard "fix" that will
> bounce message with Sender lines that aren't equal to the From
> field.   And then what will you do?
> 
> Hint: stop using mail providers that obey non-standard mail
> protocols, because they *will* break you eventually, and/or
> randomly.
> 
> 					- Ted
>

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 26 Dec 2016 13:31:28 -0500

> On Dec 26, 2016, at 9:49 AM, Theodore Ts'o <[email-masked]> wrote:
> 

You're still operating under the false assumption that DMARC's purpose
is to solve phishing.  It's real purpose (at Yahoo et. al.) is to reduce
support desk workload at the sending domain.  Any minimal efficacy at
reducing phishing is entirely incidental.

Anyway, there's no additional phishing risk. One of the few things
that Outlook does right is display both Sender and From, as

	<sender> on behalf of <author>.

If the DMARC replacement authentication (via DKIM's d= or similar
is then applied to <sender>, there's no new phishing risk.

Users who don't understand that "on behalf of" means allegedly
from are just as likely to be confused by a myriad other perplexing
signals in email and web content.  The phishing problem has no solution
by way of email authentication for sufficiently naive and gullible users.

-- 
-- 
	Viktor.

---

From: Theodore Ts'o <[email-masked]>
Date: Mon, 26 Dec 2016 15:52:49 -0500

> is then applied to <sender>, there's no new phishing risk.

By that argument, there's no excuse for the big mailer providers for
bouncing List mail because of DMARC.  They could just reference the
List-ID field, and display something like this:

    <From> via mailing list <list-id header contents>

But they don't do this.  Why, pray tell?  And is this a reason for
them not prusueing your suggestion?  Why aren't they doing this
instead of waiting and hoping ARC will solve the problem?  Maybe
because users are clueless enough that they would still be getting
confused?

					- Ted

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 26 Dec 2016 16:20:03 -0500

> On Dec 26, 2016, at 3:52 PM, Theodore Ts'o <[email-masked]> wrote:
> 

Many things continue to exist that should be otherwise...

> But they don't do this.  Why, pray tell?

Inertia, lack of imagination, architectures that enforce simple
policy during the SMTP transaction, that don't leave room for more
nuanced handling in the UI.

> Why aren't they doing this

Someone would need to write the requisite code.  It is easier to
just shift the cost onto the tiny population that uses mailing
lists.

-- 
	Viktor.

---

From: "John Levine" <[email-masked]>
Date: 27 Dec 2016 01:30:16 -0000

>messages, ...

>I believe that this does not invalid the origin DKIM signature, ...

It most certainly does.  

If people want to argue about DKIM and DMARC, could we at least spend
a few minutes reading the specs so we get our basic facts straight?

Here's a reading list:

RFC 5585 - DKIM service overview
RFC 6376 - DKIM spec
RFC 6377 - DKIM andm mailing lists
RCC 7489 - DMARC

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 27 Dec 2016 01:34:01 -0000

>    <From> via mailing list <list-id header contents>

Great idea.  Because there is no possibility whatsoever that if mail
systems did that, bad guys would put in fake List-IDs to get their
phishes delivered.

R's,
John

---

From: "Christian Huitema" <[email-masked]>
Date: Mon, 26 Dec 2016 18:03:07 -0800

> systems did that, bad guys would put in fake List-IDs to get their

Of course, they will. A system like that only works if the MUA is reasonably sure that the mail was in fact sent by the specified "sender", and that the sender was some reputable list forwarder that the user trusts.

But your mail and many comments on this lists point to the huge responsibility of the MUA with respect to phishing. Phishing is about duping the user by displaying misleading information. The effective defenses have to rely on proper user interface design, using all the information in the user context. Attempting to do that by just using network rules makes the network more complex, but cannot solve the problem.

-- Christian Huitema

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 26 Dec 2016 23:13:37 -0500

> On Dec 26, 2016, at 8:34 PM, John Levine <[email-masked]> wrote:
> 

Right and DMARC has put an end to email phishing, and this progress
is in serious jeopardy of being undone if a bad guy can send an email

	From: Bobby Phisher <[email-masked]> on behalf of Joe Banker <[email-masked]>

and is at no risk with:

	From: Joe Banker <[email-masked]>

Reminds me of "Yes minister", we've to do *something* about
phishing and DMARC is something...

-- 
-- 
	Viktor.

---

From: S Moonesamy <[email-masked]>
Date: Mon, 26 Dec 2016 21:48:05 -0800

Hi Tom, Patrik,
>invalids DKIM signatures.

That modification invalidates the DKIM signature.

At 03:10 26-12-2016, Patrik Fältström wrote:
>Bingo! What should have been some kind of 
>requirement is to define the method of signing 
>the content of portions of a message with some BCP for mailing lists.

There is a BCP for mailing lists.  The IETF 
document was not aligned with the Mailman common 
practices at the time the IETF document was written or now.

>The stress, if I may call it that, today for 
>DKIM is that the signing is not in sync with how mailing lists are run.

Yes.

Regards,
S. Moonesamy

---

From: Dave Crocker <[email-masked]>
Date: Tue, 27 Dec 2016 07:13:30 -0800

On 12/26/2016 6:03 PM, Christian Huitema wrote:
> But your mail and many comments on this lists point to the huge responsibility of the MUA with respect to phishing. Phishing is about duping the user by displaying misleading information. The effective defenses have to rely on proper user interface design,

Unfortunately, this is mostly /not/ true.

The actual experience, both in field work and usability research, is 
that UI design does not affect user processing of phishing very much. 
Neither design nor user training have much effect.

Hence most effective phishing protection is in the filtering engine(s) 
below the UI.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Theodore Ts'o <[email-masked]>
Date: Tue, 27 Dec 2016 11:10:45 -0500

> phishing and DMARC is something...

My point was that if Phishing is this unmitigated evil that justifies
doing anything, including inflicting pain on mailing lists, and the
big mail providers will not be moved, then there are only so many
choices available to development groups/organizations which rely on
mailing lists.

(1) Acquiesce and transition to using bulletin board web fora that
don't work when you are reading mail off-line, and require you to
periodically remember to go to a particulra web site to catch up on
the latest mailing list discussion.

(2) Acquiesce and inflict pain on mailing list users:
     (a) Inflicting the pain on everyone, and give people who
         know that they are using mail systems that don't
	 conform to DMARC to opt-out of the pain.
     (b) Require users who know they are using a DMARC-system
     	 (perhaps with hueristics to automatically set the default
	 correctly) to opt-in to DMARC-required from field mangling.

(3) Decide that perhaps mailing lists for developers are incompatible
    with mass-market consumer-grade mail systems.
     (a) Make it the problem of those mail users whose mail system has
         an involuntary DMARC policy to know they need to switch to
         another mail system.  (I'm surprised there are actually that
         many Google employees that are still sending from google.com
         addresses to IETF lists, although admittedly I've been
         working more to make sure that Google engineers participating
         in open source mailing lists use their personal gmail.com
         addresses --- or sending from their kernel.org addresses ---
         instead of their google.com addresses.)
     (b) Make it the problem of those mail users who mail systems
         honor DMARC policies as a receiver to switch to another mail
	 system.
     (c) For those mail systems that don't really honor DMARC
         policies, but rather treat it as a strong SPAM signal, make
         it the problem of the mail users to dig e-mails out of the
         Spam folder, and hopefully this will cause the machine
         learning systems of said mail systems to figure things out
         eventually --- or those mail users can switch to a developer-
	 friendly mail system.
     
All of the various solutions have downsides, or fit into the category
of, "in the long term, it will allow for easier phishing, so the
people who have inflicted DMARC on e-mail will have a some other
non-standard change that will screw over mailing lists *again*" ---
some of the MUA changes proposed fall into this latter category; if
they are done on a wide scale, they *will* inspire the big mail
providers to disallow List-ID: or Sender: headers.

It may be that ARC will help, somewhat, but I suspect it will not be a
silver bullet, and to some extent, we will have to choose one of the
three solutions in the end anyway, because it won't be a complete
solution.  In the best case, ARC may make (3) a more palatable
alternative, in that the failure modes caused by DMARC plus ARC are
small enough that people are willing to live with (c), or where the
failure modes are enough to force users to switch to mail systems
where (c) is practical, and so the IETF might have enough courage in
its own strength and economic importance not to knuckle under to the
big mail providers, because the number of dropped messages to due
DMARC and ARC not being a complete solution is small enough ---
certainly a few messages are lost to SPAM filters today, and we live
with this.

I will again remind people that the incentive to participate in Linux
Kernel development was enough for IBM to allow its engineers to use
IMAP clients instead of Lotus Notes --- and not even
standards-violating Domino servers for which corrupting whitespace in
the body is a mandatory behavior that can't be disabled --- but
instead using honest, standards-compliant open source IMAP servers.

This was a non-trivial achievement, to go up against IBM Software
Group.  Standing up for your principles *does* work as a strategy,
even when the company that needed to be influenced was a big company
like IBM.

If a company really wants to participate in an external standards or
open source activity, because there is a strong business case, it will
find a way.  If it doesn't, then issues relating to the mail system
probably won't make a difference in the long term.  Certainly the IETF
has shown its impotence with respect to mail standards...

						- Ted

---

From: John C Klensin <[email-masked]>
Date: Tue, 27 Dec 2016 13:12:53 -0500

An observation on this one part of Ted's note...

--On Tuesday, December 27, 2016 11:10 AM -0500 Theodore Ts'o
<[email-masked]> wrote:

> All of the various solutions have downsides, or fit into the
> category of, "in the long term, it will allow for easier
> phishing, so the people who have inflicted DMARC on e-mail
> will have a some other non-standard change that will screw
> over mailing lists *again*" --- some of the MUA changes
> proposed fall into this latter category; if they are done on a
> wide scale, they *will* inspire the big mail providers to
> disallow List-ID: or Sender: headers.

I think this is one key issue the community keeps losing sight
of in this discussion.  When a technique is invented that we
know how to break or get around and then use it to attack a
problem in the hope that the bad guys are too dumb (or just
won't bother) to develop and apply the workarounds, we create a
few additional problems.  First, the "lazy and stupid bad guys"
assumption often turns out to be a matter of scale and
economics: as long as enough messages (or other attacks) get
through, they may not care but, if our technique has a real and
significant impact, then, in most cases, the workarounds will be
applied.  Such application will have at least two bad effects:
it will increase the economic and/or operational costs to the
good folks and/or victims and it will, in Paul Vixie's words,
make the bad guys smarter.  Second, even before that transition
occurs, it will have an effect that some of us find
objectionable on moral grounds -- shifting the risks and impacts
to those least able to defend themselves.

Both burden-shifting and creating obstacles that encourage more
sophisticated behavior by attackers are reasons we have given
against weak crypto an ineffective privacy protections, yet we
find ourselves embracing similarly-weak techniques in the hope
that they will help control spam, phishing, etc., for a while.
Sorry, but I don't get the latter as being any more reasonable.

   john

---

From: Dave Crocker <[email-masked]>
Date: Tue, 27 Dec 2016 10:24:04 -0800

> When a technique is invented that we
> few additional problems.

Except that none of the anti-abuse efforts being used at scale are 
predicated on any such assumptions or logic.

That logic is, therefore, a strawman serving to distract from serious 
discussion.

d/
-- 

   Dave Crocker

---

From: "Christian Huitema" <[email-masked]>
Date: Tue, 27 Dec 2016 10:36:28 -0800

> Hence most effective phishing protection is in the filtering engine(s) 

We actually agree. In my mind, I was not thinking of UI as the arrangement of displayed pixels, but rather the intelligent selection of which information to present and what interactions to design. Without this local intelligence, MUA are not likely to handle the example that Viktor gave, "Joe Banker <[email-masked]>". Among other examples. My point is that this intelligent filtering benefits from information about the user context, such as what bank the user normally deals with. That kind of information might be available in the user context, but is normally not available to the mail delivery system.

-- Christian Huitema

---

From: Dave Crocker <[email-masked]>
Date: Tue, 27 Dec 2016 10:46:57 -0800

On 12/27/2016 10:36 AM, Christian Huitema wrote:
>>> But your mail and many comments on this lists point to the huge
>>> responsibility of the MUA with respect to phishing. Phishing is
>>> about duping the user by displaying misleading information. The
>>> effective defenses have to rely on proper user interface design,
>>
>> The actual experience, both in field work and usability research,
>> is that UI design does not affect user processing of phishing very
>> much. Neither design nor user training have much effect.
>>
>> Hence most effective phishing protection is in the filtering
>> engine(s) below the UI.
>
> We actually agree. In my mind, I was not thinking of UI as the
> arrangement of displayed pixels, but rather the intelligent selection
> of which information to present and what interactions to design.
> Without this local intelligence, MUA are not likely to handle the
> example that Viktor gave, "Joe Banker <[email-masked]>".
> Among other examples. My point is that this intelligent filtering
> benefits from information about the user context, such as what bank
> the user normally deals with. That kind of information might be
> available in the user context, but is normally not available to the
> mail delivery system.

To that end, saying "MUA" might have some formal validity, but it does 
not help the discussion.  Too many readers think it refers to something 
having to do with end-user interaction.

Worse, Viktor's line of logic presumes the modified From field somehow 
gets the message past filters better, and that is just plain wrong.

The modifications to the From line are intended for end users, not 
filtering engines.

(Whether they are actually helpful for end-users is a different 
discussion.  cf, my previous note.  To my knowledge, there have been no 
studies to establish that the ad hoc modifications are at all useful.)

d/

-- 

   Dave Crocker

-- 

   Dave Crocker

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 27 Dec 2016 14:02:26 -0500

> On Dec 27, 2016, at 1:46 PM, Dave Crocker <[email-masked]> wrote:
> 
> Worse, Viktor's line of logic presumes the modified From field somehow gets the message past filters better, and that is just plain wrong.

I was not suggesting any modification of the message From: line.  Rather
I was applauding the fact that Outlook (for one) presents a more detailed
view of the message headers than is common practice.  In particular, it
augments the displayed origin information with Sender context when present.

If "Sender + From" are displayed as in Outlook, then it becomes reasonable
to authenticate Sender when present, and not apply authentication policy
to "From", since the message is not in fact *from* the author.  It is from
the sender, (purportedly) on behalf of the author.

It is rather implausible that phishers will want to present their messages
this way (on behalf of), most users don't receive such email, and it will
stand out as unexpected.  And users who still believe such messages to be
legitimately *from* the purported author and fall victim to scams will fall
for a myriad other misdirections.

Breaking legitimate use-cases (lists) in order to fail to "solve phishing"
is counterproductive in my view.  Yahoo's DMARC cost reduction would also
be equally effective if they displayed "on behalf of" given "Sender:"
as in Outlook, and authenticated the Sender domain instead.  This would do
no damage to mailing lists.

-- 
	Viktor.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 27 Dec 2016 11:47:05 -0800

> for a myriad other misdirections.

All of the above is language cast in terms of end-users.  In the absence 
of extremely careful and constrained and informed reference, discussion 
of phishing needs to completely avoid references to end-users.

Really.  Just stop.  Everyone.

End-users are essentially irrelevant to the formalized detection and 
handling of phishing.

d/

-- 

   Dave Crocker

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 27 Dec 2016 15:10:33 -0500

> On Dec 27, 2016, at 2:47 PM, Dave Crocker <[email-masked]> wrote:
> 
> 
> End-users are essentially irrelevant to the formalized detection and handling of phishing.

That's good news.  There's no need to worry about what the From: field says at all,
all that matters is automated detection of scams.

Since phishers don't send messages with "Sender" such a change would have no immediate
negative consequences.  Real lists can DKIM sign their outbound messages and get a
decent reputation, while unsigned or forged "Sender" headers can aggressively filtered.

The user can continue to see the bare "From:" header, or "on behalf of" ala Outlook as
deemed most appropriate by the MUA designer and user preference.

-- 
	Viktor.

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 31 Dec 2016 15:49:13 -0500

[ On-list follow-up to off-list discussion with John R Levine ]

> On Dec 27, 2016, at 12:09 PM, Viktor Dukhovni <[email-masked]> wrote:
> 
> What I'm saying is that phishing protection is not the actual goal
> when publishing or filtering on DMARC.  The metric used is complaints
> about spam, or messages blocked not reduction in monetary loss,
> and this lines up with reducing support costs, ...
> 
> So filtering on DMARC reduces complaints and support costs.  Its
> effect on phishing is a separate issue.
> 
> Many of my inbound 419 scams are sent with Yahoo/Gmail/... "Reply-To"
> addresses (the From address is in some random domain), and DMARC
> does nothing to address the endemic infestation of Yahoo/Gmail/...
> by 419 scammer maildrops.

A perfect illustration of this, in the form of (two copies of) a
vanilla 419 scam sent via Yahoo, claiming to be from Gmail, with
a Gmail From/Reply-To landed in my mailbox today:

Date: Sat, 31 Dec 2016 07:55:04 +0000 (UTC)
From: "Mr.Iheleme Oskama" <[email-masked]>
Reply-To: "Mr.Iheleme Oskama" <[email-masked]>
Subject: Greetings My Dear Friend

My MUA's (Mail.app) filters easily recognized it as Junk email, if
only Yahoo's outbound email filters had been equally effective, but
I don't believe that protecting email users against scams is the
game being played.

Yes, when Paypal publishes DMARC policy, the policy is both reasonable,
and does have positive impact in reducing phishing of Paypal users.
Paypal's DMARC policy also has no negative impact on mailing lists.

When a large consumer email provider publishes p=reject, their
motivation is likely less noble and negative impact on other
legitimate uses of email is not negligible.

-- 
	Viktor.