Subject: We need an architecture, not finger pointing.

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 26 Oct 2015 11:26:26 -0400

I changed the subject as I think that a discussion of architecture
that starts off with an assertion of fault on the part of one party is
probably wrong and bad faith, certainly so.

On Sun, Oct 25, 2015 at 6:09 PM, Brian E Carpenter
<[email-masked]> wrote:

>> My proposal was not to eject them but to mark them as second class citizens.
>
> Indeed. And if it came to that, speaking for myself, I would switch to an address
> not contaminated by dmarc p=reject, unless a good solution for mailing
> lists was found first.

What we lack here is a viable concept of Internet architecture that is
free of the clutter from history.

Most computer systems have historical clutter, it is somewhat
inevitable. For example, why do printer drivers have to be installed
with system privilege in Windows etc? I am a user connecting to a
machine via the network. I don't need system privs to talk to the
network, why do I need them to install a 'driver'? When it would cost
$3500 to replace a 36" plotter which works fine except for the lack of
a signed driver, this is not an abstract question.

When you get into the reasons, it turns out that back when VMS was
designed, the computer was responsible for tracking number of pages
printed and enforcing quotas. An approach that obviously worked a lot
better in the old days when the printer connected to the computer and
not the network.

In the old days people belonged to an institution that issued their
users 'email addresses' and people got to assume that being
[email-masked] meant that Alice was the owner of the email address
and could decide how it was used. Which was a fairly odd assumption to
make since it obviously only ever extended to outbound email. Inbound
email was always going to arrive at example.com

But people could send their mail without any reference to the domain
name owner and so it was assumed that this should be the way the
Internet works.

Let us imagine for the sake of argument that we were designing a whole
new Internet from scratch applying lessons learned. How would we set
about describing a domain name? I think that we would arrive at a set
of rules that include the following:

* Domain names are unique, each name has a single owner.
* Domain names may be subdivided to accommodate an unbounded number of
Internet services and account holders.
* The owner of a domain name has exclusive control over the use of the
name and all subdivisions thereof.
* To ensure the goal of accessibility is met, domain names must be
easy and cheap to obtain and maintain.

We might quibble over the wording but I think that is pretty much
where we would arrive at and that is pretty much where the Internet is
today. If you want to be a first class citizen on the Internet you
need to have your own domain name. Otherwise you are at the mercy of
someone else.

I read all my IETF mail through Gmail but I don't own that domain and
so I don't get to make the rules governing it. Instead I use
[email-masked] which is a domain I own. One of the reasons I
have that domain is so that I have control of my mail and not Google.

The architectural description above is reasonably concise and can be
consistently applied. Now consider what sort of architectural
statement would be required to support the folk who allege that DMARC
is somehow broken. Instead of saying the domain name owner gets to
decide how it is used we would have to make separate statements about
the domain name owner and the users who have accounts in that domain.
And to do that we would have to start talking about specific
applications and specific circumstances.

I think architecture should read like a theory in physics. The simpler
the statement, the more generally it is seen to apply in practice, the
fewer corner cases, the better. If you have an architectural statement
that resorts to special pleading, it is a pretty good sign that it is
wrong.

As a rule, I don't think there is anything wrong in the IETF taking
decisions to make things easy for ourselves. But what is the biggest
problem we face? Is it really our personal ability to exchange email
on mailing lists? I thought there were rather more important
objectives at stake.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 27 Oct 2015 17:44:21 +1300

On 27/10/2015 04:26, Phillip Hallam-Baker wrote:

...
> have that domain is so that I have control of my mail and not Google.

Oh, right, so that's why your mail arrives with Google's dkim signature:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:sender:date:message-id:subject:from:to:cc:content-type;
        bh=IBh5MMCj7Oi9anz1BnRTjSN9rwt0xryWllXeoFm/33w=;
        b=ot71/Et+dU2GgseolztwUgGc8Iv3yAZAIjVvvBb7dRxuLc+1QBRdOIEpVrJehRZ+lP
         /B+2nHxixfQHTtuJttus1RjNTlDtwaHnFeon4NlU5HK8odyO4sFUXY4qy3POv3RRgowu
         pxFe6zuWJNqnXErIWW0zeZf8QVfpDjKiTz0RTiK1BSLecjPIdBnPr7mdip0xDmab4XPC
         SoaV0jX3VlpsNVqX4+i0tpCKBXQ/TnzuU1/wXlkM0abfp0VThi+xb7YmNJHp1eahYVa/
         /LEg7ho/I4gy4S92E6cYJAsM+E05h8uBQULFiObWfjRLrFsKt3pxmSloOvroBoKF7iuo
         Mkig==

I suspect that when they start generating DMARC you will get that too.

> on mailing lists? I thought there were rather more important

Actually, yes, the mailing list model and the fact that we contribute
as individuals are both pretty important objectives, not incidentals,
in the way the IETF works, distinctly different from the model in some
other SDOs. And please, I wasn't finger-pointing when I started this thread:
I was asking how do we fix this, since it apparently concerns 15 to 20%
of IETF participants.

Architecturally, we need to validate that [email-masked] is the
genuine sender even after the message has been relayed. DMARC doesn't do
that in its present form.

    Brian

---

From: <[email-masked]>
Date: Tue, 27 Oct 2015 04:49:39 +0000

> fewer corner cases, the better.

"Does this make money?"

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: ietf <[email-masked]> on behalf of Brian E Carpenter <[email-masked]>
Sent: Tuesday, 27 October 2015 3:44 PM
To: Phillip Hallam-Baker
Cc: Philip Homburg; John Levine; IETF Discussion Mailing List
Subject: Re: We need an architecture, not finger pointing.

...

    Brian

---

From: Christian Huitema <[email-masked]>
Date: Tue, 27 Oct 2015 18:17:11 +0000

> that in its present form.

Actually we need to validate two assertions:

* That the mail came from the stated author, e.g. PHB.
* That the mail was relayed by the IETF mailing list.

Identity checks matter. Lots of the discussion focused on SPAM, but the "acute problem of the day" is actually phishing, and specifically forging a mail that appears to come from someone you trust, to entice you to open a document or visit a URL that you should not. That's a pretty common step in the chain of events that leads to another "42 million user accounts compromised in a breach." 

-- Christian Huitema

---

From: "John R Levine" <[email-masked]>
Date: 27 Oct 2015 18:06:37 -0400

> * That the mail was relayed by the IETF mailing list.

Well, yes, but those are easy.  What's hard is demonstrating that the 
message that the list relayed is the same in a semantic sense as the 
message that Phill sent, even though it has the kind of changes that lists 
make, a tag in the subject line, a footer at the bottom, and attachments 
stripped.

See the last decade or so on the DKIM and now DMARC mailing lists for 
endless not very productive discussions about ways to describe permitted 
changes without also allowing vast amounts of spam and phishing, leavened 
by blithe assertions that mailing lists have been doing the wrong thing 
for 40 years and should never make any changes to messages at all.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

---

From: Mark Andrews <[email-masked]>
Date: Wed, 28 Oct 2015 10:28:02 +1100

In message <[email-masked]>, "John R Levine" write
> make, a tag in the subject line, a footer at the bottom, and attachments 

Perhaps we should not be stripping attachments but encapsulating
the whole message with enough DKIM signed meta data to enable DKIM
processing to work a the far end after DKIM verifying the mailing
list input first.  This gives you a trust chain. 

Add in List-Label: <string> header to allow the MUA to insert it
into the displayed Subject: and a footer after the encapsulated
message.  The latter should work immediately.

If list policy is not to have attachements then reject the
submission rather than strip the attachements.

If list policy is no text/html then reject messages with text/html.

Mark

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: "John R Levine" <[email-masked]>
Date: 27 Oct 2015 19:58:39 -0400

> list input first.  This gives you a trust chain.

Those are separate issues.  We strip attachments because people send 
messages that say "my kid's school play is tomorrow at 7 PM, tickets $5, 
and here's a poster" and attach a 20MB powerpoint horror.

Wrapping messages to make them a one-message digest is 100% standards 
compliant, and is one of the options in Mailman.  The only problem is that 
most MUAs render them unspeakably badly.

This is very well trodden ground.  Look at the Mailman developers' list 
where this has all been discussed to death.

Regards,

---

From: Ted Lemon <[email-masked]>
Date: Wed, 28 Oct 2015 09:33:05 -0400

On Oct 27, 2015, at 7:58 PM, John R Levine <[email-masked]> wrote:
> Those are separate issues.  We strip attachments because people send messages that say "my kid's school play is tomorrow at 7 PM, tickets $5, and here's a poster" and attach a 20MB powerpoint horror.

Seems like you could just bounce messages with attachments larger than X, where X is O(1k).   We have to allow for MUA-specific attachments, but we could even handle those by whitelisting them.   "Fixing" messages with attachments is a convenience, not a requirement.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 28 Oct 2015 11:19:52 -0400

> Identity checks matter. Lots of the discussion focused on SPAM, but the "acute problem of the day" is actually phishing, and specifically forging a mail that appears to come from someone you trust,…

This is correct.   If I seem a bit strident on this issue, it’s because I’ve actually had a family member personally affected by this problem, in a rather severe way.   The ability to verify that mail actually came from whom it claims to have come is quite important particularly for aging family members who might not not be as good at detecting scams as they once were.   And for that matter, I’ve been momentarily fooled once or twice in recent years—the amateurs give us a false sense of security, but some phishers are _very_ skilled.

Of course, part of this is a UI issue, which is out of scope, but perhaps worth mentioning: MUAs should never present a blinded URL.   If the HTML looks like this:

<a href="http://nefarious.example.org/QOJWEOJOWJCJ#UR1OJOJFOIJ?hack=yes <http://nefarious.example.org/QOJWEOJOWJCJ#UR1OJOJFOIJ?hack=yes>">http://www.example.com/ <http://www.example.com/></a>

The user should see this:

http://nefarious.example.org/QOJWEOJOWJCJ#UR1OJOJFOIJ?hack=yes <http://nefarious.example.org/QOJWEOJOWJCJ#UR1OJOJFOIJ?hack=yes>

I mention this only because PHB brought up architecture and Christian mentioned enticing people to visit URLs.   The definition of "working" has to include not being subject to obvious UI vulnerabilities that are only safe if the end user is Bruce Schneier.   Perhaps we need a Consumer Reports for commonly-used software.

---

From: "John R Levine" <[email-masked]>
Date: 28 Oct 2015 14:02:50 -0400

> Seems like you could just bounce messages with attachments larger than 
> X, where X is O(1k). ....

You could, but that now requires more manual work as people ask why didn't 
my message go through, blah blah blah.  Having run lists for my church and 
other civilians for a long time, I speak from direct experience here.

The point of mailing lists is to automate stuff, and I really don't want 
to break stuff that works now just to circumvent the limitations of the 
FUSSP/FUSPP du jour.

Regards,

---

From: t.p. <[email-masked]>
Date: Wed, 28 Oct 2015 17:57:41 +0000

----- Original Message -----
From: "Mark Andrews" <[email-masked]>
To: "John R Levine" <[email-masked]>
Cc: "Christian Huitema" <[email-masked]>; "IETF Discussion
Mailing List" <[email-masked]>
Sent: Tuesday, October 27, 2015 11:28 PM

Mark

Now that is a magnificant idea.  E-mails are ten times the size they
used to be and, despite Moore's Law and variations thereof, this gives
me problems.  Get rid of html and that would reduce the size of many by
a factor of three or more.

And while we are at it, get rid of all the X- e-mail headers that seem
to have no impact on e-mail and that would save a fortune.

This, for me, is the problem with e-mail.

Tom Petch

---

From: Ted Lemon <[email-masked]>
Date: Wed, 28 Oct 2015 14:14:23 -0400

On Oct 28, 2015, at 2:02 PM, John R Levine <[email-masked]> wrote:
> You could, but that now requires more manual work as people ask why didn't my message go through, blah blah blah.  Having run lists for my church and other civilians for a long time, I speak from direct experience here.

And this is because the SMTP store-and-forward model requires that error messages be delivered asynchronously, which is another architectural problem with SMTP.   If error messages could be delivered at the moment of transmission rather than later, at least most of the time, this wouldn’t be an issue.   It’s entirely possible to do it, but requires a little more sophistication than Postfix or sendmail have.

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 28 Oct 2015 18:34:40 +0000

On Wed, Oct 28, 2015 at 02:14:23PM -0400, Ted Lemon wrote:

> And this is because the SMTP store-and-forward model requires that error
> messages be delivered asynchronously, which is another architectural
> problem with SMTP.

    1.  There is no requirement to always send asynchronous bounces,
	most properly operated mail systems strive to reject
	synchronously rather than accept and then bounce.  Avoidable
	backscatter is frowned upon.

    2.  It is not possible to *guarantee* delivery of all mail accepted
        for onward relaying.  *Some* asynchronous errors are unavoidable.

> If error messages could be delivered at the moment of transmission rather
> than later, at least most of the time, this wouldn't be an issue.

Most of the time, on properly configured receiving systems, errors
are already synchronous.

> It's entirely possible to do it, but requires a little more
> sophistication than Postfix or Sendmail have.

My future-gazing crystal ball is out of stock on Amazon, delivery
estimate not available.  With no crystal ball to predict the future,
mail queued for onward relaying might later bounce.

-- 
	Viktor.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 28 Oct 2015 14:40:57 -0400

> Most of the time, on properly configured receiving systems, errors

This is almost never the case.   Sure, if you send mail to a RCPT TO: that fails, then you can get immediate notification, but rejection of attachments generally happens after the mail has been accepted and queued.   In order to _bounce_ a message based on content, you need to evaluate it before sending the "250 Message Accepted response."   There is a valid code for that, which I don’t remember off the top of my head, but I don’t know of any MTAs that use it.

The point is that it is possible following the current specs to deliver an immediate response in the majority of cases, but that isn’t being done, and furthermore MUAs aren’t expecting it, and so probably won’t give the user a message that explains to them why the message was rejected.   This is an entirely solvable problem, but it is not a solved problem.

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 28 Oct 2015 19:27:45 +0000

On Wed, Oct 28, 2015 at 02:40:57PM -0400, Ted Lemon wrote:

> This is almost never the case.   Sure, if you send mail to a RCPT TO: that
> fails, then you can get immediate notification, but rejection of attachments
> generally happens after the mail has been accepted and queued.

You're clearly sending mail to poorly operated systems, likely to
systems where the attachments are rejected somewhere other than at
the edge MX host.  Such systems should not be rejecting the
attachments, it is too late for that, once it crossed the line from
"outbound" to "inbound".  The systems in question need to apply
their attachment policy closer to the "edge", i.e. at the first
inbound hop.

> In order to _bounce_ a message based on content, you need to evaluate it
> before sending the "250 Message Accepted response."   There is a valid
> code for that, which I don�t remember off the top of my head, but I don't
> know of any MTAs that use it.

The SMTP reply code and DSN code SHOULD be correct, but are not
especially critical.  Any 5XX 5.N.M code will have the effect of
synchronously rejecting the message.

Postfix has header/body checks that reject based on content, and
supports milters and proxy filter that can do the job out of process.
The reply for bad content in Postfix defaults to "550 5.7.1 message
content rejected".

> The point is that it is possible following the current specs to deliver
> an immediate response in the majority of cases, but that isn�t being done,

Well it is being done, and Postfix users are strongly encouraged
to do so whenever backscatter is discussed on the users list.

Systems where content inspection happens after mail is queued, and
unwanted content triggers bounces are misconfigured.

> and furthermore MUAs aren't expecting it, and so probably won't give the
> user a message that explains to them why the message was rejected.   This
> is an entirely solvable problem, but it is not a solved problem.

I await patches for Postfix and your favourite MUA that solve this
"entirely solvable" problem.

MUAs are a very different issue.  The MSA should generally accept
and bounce back to the authenticated user, this is fine, because
the MSA authenticates the client.  MUAs are often not able to deal
with rejection at submission time, especially rejection for a subset
of the recipients.  Nor is it possible for the MSA to synchronously
report remote rejects from (some subset of) the receiving domains.

-- 
	Viktor.

---

From: John C Klensin <[email-masked]>
Date: Wed, 28 Oct 2015 16:36:29 -0400

Folks, please tone down the rhetoric, read the specs, try to
understand the comments Dave Crocker and I made, and at least be
clear about the difference between your taste and what SMTP
requires and allows.  There actually are reasons for most or all
of SMTP.   If you think those specifications are incorrect, I
look forward to seeing the well-reasoned I-D, one that starts
from the Internet email environment as it actually exists
outside the giant providers (and as several people, notably
including John Levine) keep trying to explain).

Some particular comments below.

--On Wednesday, October 28, 2015 19:27 +0000 Viktor Dukhovni
<[email-masked]> wrote:

>> > Most of the time, on properly configured receiving systems,
>> > errors are already synchronous.
>> 
>> This is almost never the case.   Sure, if you send mail to a
>> RCPT TO: that fails, then you can get immediate notification,
>> but rejection of attachments generally happens after the mail
>> has been accepted and queued.

> You're clearly sending mail to poorly operated systems, likely
> to systems where the attachments are rejected somewhere other
> than at the edge MX host.  Such systems should not be
> rejecting the attachments, it is too late for that, once it
> crossed the line from "outbound" to "inbound".  The systems in
> question need to apply their attachment policy closer to the
> "edge", i.e. at the first inbound hop.

If you are describing what you consider good practice, then
fine.  But SMTP requires no such thing and there are perfectly
reasonable configurations that make it impossible without
rejecting messages that should not be rejected (see John
Levine's comments about complaints about why mail was not
delivered, but that is only part of the issue).

And please don't talk about "edge MX host" when you probably
mean "final delivery MTA" or equivalent.  They are different and
the spec very deliberately allows some cases that "edge MX host"
probably doesn't cover.

>> In order to _bounce_ a message based on content, you need to
>> evaluate it before sending the "250 Message Accepted
>> response."   There is a valid code for that, which I don�t
>> remember off the top of my head, but I don't know of any MTAs
>> that use it.

No, in order to _reject_ a message as part of the SMTP dialog,
you clearly need to evaluate it before that dialog is complete.
Depending on how the delivery SMTP server is configured, that
may or may not be possible and it is, in general, neither a
violation of the standard nor of good sense to configure it that
way (optimality for that case involves a lot of factors even
though operational conditions these days, especially
"backscatter" risks, usually favor rejection over bouncing. You
can _bounce_ a message at any point prior to delivery into a
mail store (and maybe after that).

A "250 Message Accepted response" does not mean the message must
be delivered.  It means that the MTA has taken responsibility
for it and, barring very unusual circumstances (e.g.,
"operational necessity") will either deliver it as intended or
generate a bounce message (aka NDN).

> especially critical.  Any 5XX 5.N.M code will have the effect
> of synchronously rejecting the message.

Yep.  SMTP is very clear about that and has been since RFC 821
-- especially if there is any doubt, only the first digit of the
reply is actionable.
 
> Postfix has header/body checks that reject based on content,
> and supports milters and proxy filter that can do the job out
> of process. The reply for bad content in Postfix defaults to
> "550 5.7.1 message content rejected".

Good for Postfix.  The last I heard, it was not the standard.

>> The point is that it is possible following the current specs
>> to deliver an immediate response in the majority of cases,
>> but that isn�t being done,
> 
> Well it is being done, and Postfix users are strongly
> encouraged to do so whenever backscatter is discussed on the
> users list.
> 
> Systems where content inspection happens after mail is queued,
> and unwanted content triggers bounces are misconfigured.

Thank you for your opinion.    While I also generally prefer to
have them configured some other way when that is feasible,
"misconfigured" goes too far.  As a particular example, it is
often desirable to configure one or more lower-preference MTA to
collect mail traffic when the preferred delivery server is not
available.  It is also common, but not an SMTP requirement, that
the lower-preference MTAs have enough information about the
delivery environment to be able to reject messages for some
reasons, e.g., "no such mailbox" conditions as well as, or
instead of, message header or content conditions.  However, such
arrangements can easily lead to race conditions such that, if
the lower-preference servers are configured _really_ carefully,
they should not return "5yz no such mailbox" code unless they
know that from, e.g., syntax rather than by absence from a list
of mailboxes but should probably return a 4yz code or accept the
message and then let the final delivery server make the
deliverability determination and generate an NDN if needed
(because, other than just discarding the message, that it the
only thing it usefully can do).

>> and furthermore MUAs aren't expecting it, and so probably
>> won't give the user a message that explains to them why the
>> message was rejected.   This is an entirely solvable problem,
>> but it is not a solved problem.
> 
> I await patches for Postfix and your favourite MUA that solve
> this "entirely solvable" problem.
> 
> MUAs are a very different issue.  The MSA should generally
> accept and bounce back to the authenticated user, this is
> fine, because the MSA authenticates the client.  MUAs are
> often not able to deal with rejection at submission time,
> especially rejection for a subset of the recipients.  Nor is
> it possible for the MSA to synchronously report remote rejects
> from (some subset of) the receiving domains.

Some of that may be an argument for MUA using proprietary and/or
out-of-band means, rather than slightly modified SMTP, to
communicate with the submission server.  But that has been
allowed since the beginning of SMTP and was normal practice for
many years.    But expecting such rejections to always be
detected and reported in real time requires that the submission
server attempt mail delivery when the MUA connection to it is
still open _and_ that it be able to make a direct connection to
the final delivery MTA and do on its first attempt.  See Dave
Crocker's notes about those assumptions.

best,
    john

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 28 Oct 2015 16:57:06 -0400

On Wed, Oct 28, 2015 at 9:33 AM, Ted Lemon <[email-masked]> wrote:
> messages that say "my kid's school play is tomorrow at 7 PM, tickets $5, and
> here's a poster" and attach a 20MB powerpoint horror.
>
>
> Seems like you could just bounce messages with attachments larger than X,
> where X is O(1k).   We have to allow for MUA-specific attachments, but we
> could even handle those by whitelisting them.   "Fixing" messages with
> attachments is a convenience, not a requirement.

Here we are going beyond Internet architecture and into the real of
messaging application architecture. I think it is very clear that the
owner of example.com can decide to publish DMARC records that prevent
use of mail users with accounts in that domain sending to a mailing
list and that flows from a correct understanding of the Internet
architecture.

If you want to fix messaging, it is harder. You essentially have two choices:

1) Attempt to work within the existing deployed infrastructures.
2) Attempt to supersede the existing infrastructure with a new one.

Both give abundant opportunity for despair. Deploying a new messaging
system is next to impossible. There will always be some parts of the
legacy infrastructure that resist incremental change.

We have enough trouble trying to change the WebPKI and that is despite
the system being actually engineered for things like algorithm updates
and having a coordinating body (CABForum) that simply doesn't have an
equivalent for mail.

Things are not totally hopeless however. I think that there is a good
chance we could start to fix this in 2018 when a particular piece of
IPR expires.

We are not going to persuade people to change their email systems,
that is hopeless. That can't happen any more than people would have
changed their FTP servers back in 1993. What is quite achievable
however is to get people to adopt a system that offers a new set of
functionality that incidentally makes the legacy infrastructure
obsolete. The Web was not adopted because people were looking for a
replacement for FTP but it has effectively replaced FTP.

What I would urge people to think about is the need for a standards
based infrastructure with 'dropbox' like capability. Further, one that
allows for end-to-end encryption of the message contents and for
messages to be distributed to groups of users defined by an offline
administrator.

So for example, I am about to release some code to my engineering
team. The code is large (about 200 MB including the test vectors). I
do not know the names of the engineers who are going to be working on
the systems, they don't report to me.

What I want to be able to do is to wrap up the data in a ZIP file, and
encrypt it to a security label ( [email-masked] ) that is
administered by the Director of Engineering.

The only way to do that with S/MIME and OpenPGP right now is that
either I have to encrypt the message for each recipient or some easily
breached server has to.

If we apply Matt Blaze's proxy re-encryption, 'recryption', my
director of engineering can do the administration of the security
label using an offline client and upload data to the recryption
service. the service can then recrypt the information required for any
authorized user to decrypt any of the data I upload.

Recryption is a powerful tool. So powerful that I think it is properly
regarded as the extension of public key cryptography from bilateral
communications to multi-lateral communications.

While we all know and understand that email users don't see the need
for E2E encryption, this is not the case with pull-model data
distribution. People get really worried about putting large chunks of
data onto a file service.

I think that getting people excited about a re-engineering of SMTP is
going to be very hard. But there is a real possibility that people
might be interested in a unified messaging infrastructure that merges
email, news, dropbox, chat and VOIP and has an integrated security
infrastructure built in from day 1.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 28 Oct 2015 17:14:53 -0400

> to do so whenever backscatter is discussed on the users list.

Sorry, I left out one critical point.   Consider what happens when I want to send mail to [email-masked] <mailto:[email-masked]>.   I write a message in my MUA.   It connects to my maildrop at nominum.com <http://nominum.com/>.   Nominum.com <http://nominum.com/> accepts and queues the mail.   Then it establishes a connection to ietf.org <http://ietf.org/>.   IETF.org <http://ietf.org/> mail well send a 5xx status code at the end of the DATA transaction, but it’s too late: my MUA has already disconnected, and there’s no way to send a synchronous status update.

In order for this to work, when I connect to my maildrop, my maildrop has to immediately connect to ietf.org <http://ietf.org/> when it sees RCPT TO: [email-masked] <mailto:[email-masked]>. It has to then tunnel the message through, applying any local policy in the process and aborting the connection to ietf.org <http://ietf.org/> if the local policy detects a violation while _it_ is scanning the message body that’s being dumped on the maildrop.   If local policy allows the process to get to the end of the DATA transaction, and the response from ietf.org <http://ietf.org/> is a 5xx response, then the maildrop server at nominum.com <http://nominum.com/> has to still have the connection open, and has to respond with the same 5xx response.

If Postfix is able to do this, that is news to me, but I will admit that I gave up on being a Postfix expert a long time ago—like most MTAs, it addresses too many use cases, and so it’s difficult to configure.   Sendmail, even farther in the past for me, is even worse.   If there is some new MTA out there that does a better job of addressing modern use cases, I am interested to hear about it.   I get the impression that most of the interesting MTA research nowadays is being done by the big mail processors, and isn’t trickling down, but I may just not have looked in the right place.

Of course, what I am describing is made more painful by various anti-spam tactics like greylisting.

---

From: [email-masked]
Date: Wed, 28 Oct 2015 13:28:24 -0700 (PDT)

> This is almost never the case.

Actually, in the SMTP world, it's almost always the case. The majority of email
systems go to *tremendous* lengths to perform all SMTP rejections at the point
of ingress to the administrative domain, because doing anything else
unavoidably creates large amounts of blowback spam. And that's not acceptable
for a bunch of different reasons.

In the very rare cases where a check ended up having to be done somewhere past
the ingress MTA at a site that uses our software, a bug filing was always the
immediate result. And I can assure you we accord such bugs the highest possible
priority, because we're well aware of how serious their effect can be on a
site's reputation.

> Sure, if you send mail to a RCPT TO: that fails, then you can get immediate
> notification, but rejection of attachments generally happens after the mail has
> been accepted and queued.

Interestingly, you're wrong on both counts. SUBMIT-time recipient address
validation is generally only achievable for addresses within an administative
domain. The minute you send to someone outside your ADMD, such validation gets
to be much more difficult for a variety of reasons.

And the same rules apply regardless of your rejection criteria. Intra-ADMD
Messages are rejected based on content analysis at submission time all the
time. Messages destined for other ADMDs are not, because the rejection criteria
are implemented by the other ADMD's servers and are not available to the SUBMIT
server. But in such cases the rejection does happen at the ADMD ingress point.

To put this another way, is it's not the nature of the check, it's who is
performing the check, that determines whether or not it can be done immediately
from the MUA's perspective.

> In order to _bounce_ a message based on content,
> you need to evaluate it before sending the "250 Message Accepted response."  

Yep, and that's exactly what is done. Routinely and at enormous scale.

> There is a valid code for that, which I don’t remember off the top of my
> head, but I don’t know of any MTAs that use it.

For better or worse, the code for the MTAs handling the majority of the world's
email is no longer something you can inspect. But even looking at the open
source world, there are a number of milters with this capability, and there's a
selection of MTAs that support milter.

> The point is that it is possible following the current specs to deliver an
> immediate response in the majority of cases, but that isn’t being done, and
> furthermore MUAs aren’t expecting it, and so probably won’t give the user a
> message that explains to them why the message was rejected.   This is an
> entirely solvable problem, but it is not a solved problem.

Your problem is you're confusing and conflating message submission (SUBMIT) and
relay (SMTP). We now talk about these as separate protocols for a reason.

And as far as MUA handling of various sorts of errors, it really depends on the
mail client. Some clients handle immediate errors well, others do it poorly,
and some are abysmal. By the same token some clients handle nondelivery report
messages well and some do not.

MUAs aren't really my thing, so I can't really talk about what's common
and what isn't in that space.

In any case, the bottom line is you're jumping to conclusions about a vast
infrastructure that's currently handling in excess of a trillion messages a day
based on what appears to be little more than your personal experience. And I
must say if your experience is what you say it is, it's more than a little
atypical.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Wed, 28 Oct 2015 17:22:41 -0400

> must say if your experience is what you say it is, it's more than a little

Er, the big mail services mostly just silently drop mail they don’t like, as far as I can tell, and they don’t tend to do policies like we’re discussing here: they just evaluate how likely the mail is to be something the reader wants to see, and deliver it or junk it on that basis.   You are right that they do a very good job of it, but it’s quite an expensive job, and that’s part of my criticism: it need not be nearly so expensive, and it need not be something only big companies can do.

This conversation is specifically about a problem with IETF mailing lists, so the points I was making do apply here.   I am in fact curious to hear if the big mail providers are doing things like what I described in the previous message—if so, that’s cool in principle.   But if it’s not described in the RFCs, you haven’t actually contradicted the point I was making, which is that it ought to be reflected in the standards.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 28 Oct 2015 17:30:38 -0400

> including John Levine) keep trying to explain).

I don't think the problem here is a lack of understanding. Rather it
is a difference of opinion as to what the approach should be.

As a meta-point (since we are on the IETF list, that is most
relevant), I think one of the things we lack is a protocol lifecycle
model and the determination to apply it.

Yes, I know we have 'HISTORIC' but when are we going to call it a day
on BEEP let alone FTP?

If we were managing our portfolio rationally, we would put FTP and
BEEP in a bucket marked 'No further effort'. Getting FTP up to modern
security standards would be a huge amount of work and it would be
wasted effort as its feature set has been subsumed by SSH and HTTP
long ago.

The problem is that it is difficult to let go.

SMTP email has had a 40 year run. Like the car I own of that vintage,
it is perfectly capable of functioning (albeit with some assembly in
the case of the car) but it isn't going to approach the speed, safety
or reliability of a new model.

There comes a time when getting 1970s technology to meet 2015 needs is
a case of diminishing returns. And adapting to changing needs
frequently means losing functionality.

In short, expecting every feature of SMTP that could be used in 1980
to be supported today is unreasonable. While it is in theory possible
to drive a 1977 MGB at 100mph one would be most ill advised to try.

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 28 Oct 2015 22:18:30 +0000

On Wed, Oct 28, 2015 at 04:36:29PM -0400, John C Klensin wrote:

[ I've been co-maintaining a widely used MTA (one that for example
delivers all ietf.org list mail) for ~15 years now and was postmaster
for ~60k users for 10 of those years. Any apparent ignorance of
the architecture, specifications and current practice of SMTP on
my part is a failure to communicate, rather than a lack of competence.
I am also quite aware of your history in this space and have no
material disagreement with you on substance.  We differ on style. ]

> fine.  

Yes.  Of course a best practice, but over time as standards age,
they need to be supplemented with a decent dose of BCP.

> mean "final delivery MTA" or equivalent.

No, I mean the "edge MX host".  Or whatever you might call the
first machine on the forward path that is accepting inbound mail
rather than sending outbound mail.

> They are different and

I am well aware of the distinctions, and chose my words with care,
though in this case not a standard term of art.

> Good for Postfix.  The last I heard, it was not the standard.

See BCP comment above.

> "misconfigured" goes too far.

Yes, this explicitly goes much farther than justified by standards,
but in practice backscatter-generating systems are problematic in
a world where a lot of the mail they reject has a forged return
path.

Therefore, in practice on today's Internet, (IMNSHO, and representing
apparent consensus in at least the Postfix community) the only
right thing to do is to not generate backscatter (especially when
the content is suspect).

> As a particular example, it is often desirable to configure one or more
> lower-preference MTA to collect mail traffic when the preferred delivery
> server is not available.

Yes, backup MX hosts are a prime source of backscatter when operated
by third-parties with different content inspection policies than
the destination domain.  They've mostly outlived their usefulness.
I advise Postfix users to avoid arms-length backup MX services.

But we digress.  The message I responded to claims that backscatter
is the norm, and that it can be eliminated.  Both are wrong.
Backscatter is no longer the norm, but (agreeing with you) it cannot
always be eliminated, rather we can make it increasingly rare with
"proper" configuration.

> Crocker's notes about those assumptions.

I agree, those assumptions are unreasonable.

-- 
	Viktor.

---

From: "John Levine" <[email-masked]>
Date: 30 Oct 2015 07:08:22 -0000

>Er, the big mail services mostly just silently drop mail they don’t like, as far as I can tell, ...

Gmail and AOL reject at the end of data.  I have lots of locally
generated bounces, generally for forwarded stuff that their spam
filters liked less than mine did.  It is my impression that with the
possible exception of stuff with malware attachments, everything gmail
accepts ends up somewhere in your inbox, possibly the spam folder.

What leads you to believe otherwise?

R's,
John

---

From: [email-masked]
Date: Wed, 28 Oct 2015 17:39:07 -0700 (PDT)

> Er, the big mail services mostly just silently drop mail they don’t like,
> as far as I can tell,

Actually, most of the big providers return a 5yz error, and they do it
as early in the SMTP dialogue as possible. If the message is accepted it
generally gets delivered, although it may be marked as spam.

A few providers seem to think that its a bad idea to say that a message
has been dropped as spam, but this tends to be more of a low end thing.

Things like size limit or bad content are almost always handled with an
outright rejection.

> and they don’t tend to do policies like we’re
> discussing here:

That depends on the provider. Some offer essentially no filtering capability;
others offer fairly extensive configurable server side filtering.

> they just evaluate how likely the mail is to be something the
> reader wants to see, and deliver it or junk it on that basis.   You are right
> that they do a very good job of it, but it’s quite an expensive job, and
> that’s part of my criticism: it need not be nearly so expensive, and it need
> not be something only big companies can do.

Actually, there are any number of reasonably priced AS/AV solutions for MS
Exchange class mail systems. (Admittedly this may change as more and more stuff
shifts to the cloud.)

The selection for non-Exchange systems is smaller, but there are still options
available. For my small SOHO-class mail system, I pay around $100 a year. This
gets me service for a handful of accounts that's roughly comparable in quality
to what the big players offer. (It took a couple of hours to get it set up and
configured, but most of that had to do with getting things running on a Mac
rather than a PC or Linux machine than it did with any mail system issues.)

I don't think that's terribly expensive, but YMMV.

> This conversation is specifically about a problem with IETF mailing lists, so
> the points I was making do apply here.   I am in fact curious to hear if the
> big mail providers are doing things like what I described in the previous
> message—if so, that’s cool in principle.   But if it’s not described in
> the RFCs, you haven’t actually contradicted the point I was making, which is
> that it ought to be reflected in the standards.

AFAIK everything I've talked about is described in an RFC somewhere. Indeed,
the problem isn't that it's not there, it's that there's too much and it's
difficult for inexerienced people to match up the right capabilities and
architecture to solve a given problem.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Fri, 30 Oct 2015 20:42:27 +0000

> architecture to solve a given problem.

That may well be so.   It may be that all that's needed (by me anyway) is a document of some sort that points out where all this stuff is.  Having read the SMTP documents pretty closely recently, there is definitely a substantial amount of stuff in there that I don't see implemented in systems that are available to me.   However, I am fairly sure that there are in fact gaps in the docs in terms of the features i want to see.   Even if I'm wrong, I think it's worth going through the exercise of figuring that out.   By which I do not mean "go do some work for me."   I'm interested in this and willing to do work, but as I'm sure you've gathered, I do not have a complete picture of what's been documented and what hasn't been.

---

From: John C Klensin <[email-masked]>
Date: Fri, 30 Oct 2015 18:23:47 -0400

--On Friday, October 30, 2015 20:42 +0000 Ted Lemon
<[email-masked]> wrote:

>> AFAIK everything I've talked about is described in an RFC
>> somewhere. Indeed, the problem isn't that it's not there,
>> it's that there's too much and it's difficult for
>> inexerienced people to match up the right capabilities and
> That may well be so.   It may be that all that's needed (by me
> anyway) is a document of some sort that points out where all
> this stuff is.  Having read the SMTP documents pretty closely
> recently, there is definitely a substantial amount of stuff in
> there that I don't see implemented in systems that are
> available to me.   However, I am fairly sure that there are in
> fact gaps in the docs in terms of the features i want to see.
> Even if I'm wrong, I think it's worth going through the
> exercise of figuring that out.   By which I do not mean "go do
> some work for me."   I'm interested in this and willing to do
> work, but as I'm sure you've gathered, I do not have a
> complete picture of what's been documented and what hasn't
> been.

Ted,

As the guy who is sitting on, rather than holding and moving,
the pen on some of the critical documents (notable 5321bis), let
me explain why because it is an important part of the picture.

First, the path from 821 and 974 and through 1123 plus 1869 and
assorted extensions and small modifications to 2821, then from
2821 to 5321, became largely an exercise in merging text.  The
result contains a number of artifacts, including in how the
document is organized, and is consequently redundant in some
places and _very_ hard to read in others.  I can't keep track of
all of it: if someone asks me a question, I typically have to go
back and check.  When I don't, I make mistakes more than half
the time.  Others may have better memories (some probably as a
consequence of working with the specs every day) but, if I can't
keep track, expecting casual readers to do so is probably
unrealistic.  

I wanted to do a complete rewrite in the 5321 timeframe and even
engaged a very good technical editor to help with reorganizing
and structuring.  I eventually gave up because, especially in
the absence of a forcused WG, several members of the community
(including a large subset of the obvious experts) were concerned
about the risk of inadvertent changes accompanying changes in
organization and some needed changes in terminology.  Much as I
might wish otherwise, I don't see any 5321bis being any
different.  We could fix or patch known issues (of which the
current errata list is a subset), fold in some of the EAI
material if it were ready and make informative references to it
is it were not, but what is needed to make things clear and
coherent to you and others is unlikely to happen in 5321bis for
the same reason it didn't happen in 5321.  That reason might be
summarized as a hypothesis that the relevant community just
doesn't have the energy to deal with a very careful review of a
significantly rewritten and restructured document.

The second issue is that I believe that we have ample evidence
that any attempt to open 5321bis would generate a tremendous
amount of discussion (some would say "noise") from people who
believe that email is obsolete, people who believe that multihop
email models are unnecessary and harmful, people who believe
that hop-by-hop in-transit (TLS-based or otherwise) encryption
is an important security measure and those who don't, and people
who have better ideas than SMTP or even MIME.  Think "final
ultimate solution to the email problem".  It is hard to figure
out a model that would prevent those people from being heard and
it would probably not be desirable if we could figure it out.
But if resolving even a large fraction of those arguments
becomes prerequisite to 5321bis, then I suggest that the
community will run out of energy before we have an approved
spec, even if it is the minimum update to 5321 with none of the
structural changes suggested above.

So we are stuck, have been stuck for several years, and I have
no idea how to get unstuck.   

In addition, there are some areas where the IETF could do useful
architectural work.  For example, there is a common trend these
days (one referred to indirectly in several recent notes) to
view email addresses as good personal identifiers.  They
probably aren't, especially in a world where the mailboxes are
not really under the control of the user being identified.  But,
while it is mostly not a technical problem (certainly one that
can't be "solved" by protocol changes) no one has ever explained
the issues in terms that can be understood by those who are
making the decisions, much less publicly recommended against it.

That work, and similar things, may be more important than
"fixing" SMTP... and new SMTP documents would be easier to write
and get agreement about if those issues were under control.

     best,
      john

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 Oct 2015 15:34:58 -0700

On 10/26/2015 8:26 AM, Phillip Hallam-Baker wrote:
> If you want to be a first class citizen on the Internet you

http://www.isi.edu/touch/internet-rights/

It's not so much that you need to own a DNS name as you need to be able
to control what resolves to your name AND what it resolves to. And you
need a public IP address. See the list above for the full set.

Joe

---

From: Dave Crocker <[email-masked]>
Date: Fri, 30 Oct 2015 16:07:17 -0700

> amount of discussion (some would say "noise")

I'll suggest a very simple and basic point that makes most other issues
largely irrelevant:

     There is no community consensus about the changes that are needed
     to improve Internet Mail.

Many individuals are convinced they know Truth in that regard, but they
have little basis for their beliefs, little understanding of the
implications for the installed base or for the range of email scenarios
that are legitimate and useful and/or are ignoring quite a bit of
experience; but in all cases haven't bothered to garner significant
community support.

Many other individuals believe that the way to get improvement is to
throw out the current mail service and start over, but that is, really,
nothing but a preference for anarchy, since they don't even have a set
of specifics they are selling.

Get the community to agree on that changes in functionality or
operations.  Then we can try to figure out how to provide them.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Ted Lemon <[email-masked]>
Date: Fri, 30 Oct 2015 23:27:12 +0000

>     to improve Internet Mail.

The way you get consensus is through discussion.

> experience; but in all cases haven't bothered to garner significant

On the contrary, Dave, I think you have every reason to think you have a good knowledge of the truth.

> of specifics they are selling.

I don't know of any such individuals.   I happen to believe that IMAP is not salvageable, but I'm willing to be talked around on that point.   I've just never seen an implementation that behaved well, and with my understanding of the IMAP data model, really don't see any way to fix it.   But my IMAP fu is weak, and as I say I'm willing to be talked into changing my mind by someone with a better understanding of the data model who is willing to address my complaints.   I am not willing to simply take it on faith that there's nothing to be fixed, however.

> operations.  Then we can try to figure out how to provide them.

What community?   gmail users?   gmail operators?   IETF participants?   Corporate mail users?   Individuals?   This is a serious question--I'm not trying to be facetious.   I don't expect that all of the communities I've mentioned will have the same motivations or wishes, and I don't think that means there is no work to do.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 30 Oct 2015 23:35:46 +0000

Hm.   I certainly think what you would like to do, but believe impossible, about 5321bis sounds good in principle.   Pete and I had breakfast this morning and he talked me around a bit on the asynchronous vs. synchronous thing.   I still think there's a pony in there, but he had some good points, and I can see how to get what I want using his methodology, so I don't think I need to push the point.

One of the things I learned rather late in my short tenure as AD is that sometimes you need someone to evaluate rough consensus who is willing to make some participants unhappy, and that that willingness is actually rare among people who call consensus.   That is not necessarily because they wouldn't be willing in principle, but that they don't actually understand how to call consensus.  I think a lot of the non-terminal discussions we have in the IETF actually fall into that category, and your 5231bis discussion may qualify.

Failing to call consensus in this way can result in the community losing interest, because they rightly perceive that they are wasting their time.

I'm sure you've read RFC 7272, and may or may not agree with me on the point I am making... :)

---

From: Doug Royer <[email-masked]>
Date: Fri, 30 Oct 2015 17:58:19 -0600

On 10/30/2015 05:07 PM, Dave Crocker wrote:

> .....

It is getting hard to figure out which issue is being discussed.
-- 

Doug Royer - (http://K7DMR.us / http://DougRoyer.US)
[email-masked]
714-989-6135

---

From: Doug Royer <[email-masked]>
Date: Fri, 30 Oct 2015 18:08:29 -0600

On 10/28/2015 03:14 PM, Ted Lemon wrote:
> On Oct 28, 2015, at 3:27 PM, Viktor Dukhovni <[email-masked]
> ....
> In order for this to work, when I connect to my maildrop, my maildrop
> has to immediately connect to ietf.org <http://ietf.org> when it sees
> RCPT TO: [email-masked] <mailto:[email-masked]>. ....

Which can no happen in some sites with firewalls or ISPs. The ISP I use
at home decided to block all access in or out to port 25. Did not tell
me. I had to figure it out. That's how they decided to slow down
outgoing spam. I changed my outgoing system configuration. And some
sites use an smtp proxy to send out email. You will not have a clue if
they queue before sending. No direct connection allowed and they do not
tell you.

Some sites translate internal email addresses to/from internal ones as
they traverse the firewall. This happened at Sun, Inside Sun I was
[email-masked], to the outside world I was [email-masked].

Whatever the solution it has to be some kind of identity proof without
regard to routing.

--