Subject: Re: Help test IMAP access to the IETF archives

From: Robert Sparks <[email-masked]>
Date: Wed, 21 Oct 2015 09:55:02 -0500

Testers: Thank you for your help!

Please move to the actual server now.
See <http://trac.tools.ietf.org/group/tools/trac/wiki/Imap>
(Be sure to delete and create a new account in your client - simply 
changing the hostname will not work well).

RjS

On 8/27/15 3:37 PM, Robert Sparks wrote:
> Folks -
>
> We have been testing IMAP access to the IETF archives for several days 
> now, with volunteers from the tools-discuss and ietf chairs lists, and 
> it's been going well. Thank you if you've been helping test already 
> (and please continue to do so)!
>
> Now we need more testers. We'd like to get a simple measure of load, 
> so we'd like to have as many folks as are willing accessing the test 
> instance at the same time with various clients during the next week. 
> It would be particularly helpful to have several people using the 
> system next Thursday.
>
> If you're willing to take the time to do this, please drop me a note 
> directly (I've set reply-to on this message) so I have a feel for how 
> many people are participating.
>
> Instructions for accessing the test instance are below.
>
> Please report any issues you find directly to me. Do not open tickets 
> with the secretariat. Be aware that there are known issues with the 
> data on the test instance (some lists have blank or truncated 
> messages) that have not yet been addressed - it would be good to 
> report any more you find to me, but they won't get fixed immediately.
>
> Please use [email-masked] for general conversation. If you 
> have feature requests, that's a good place to send them. One request 
> we've had from a few testers so far is to provide an additional shared 
> folder with a mailbox for each list that has only the "recent" 
> messages from that list - perhaps the last 6 months. It would be good 
> to hear from folks, after they've tried what's there now, whether that 
> would be useful to spend development time on, or if that time would be 
> better spent elsewhere.
>
> The test instance is getting mail with only a slight delay after it 
> comes through the lists - you should see current traffic.
>
> Please resist the temptation to download the entire archive of every 
> list. I've done it - it's huge (~28G when MailMate does it) and it 
> it's a severe torture test for clients. Interrupt your client if it 
> starts trying to do that to you. Caching a copy of this test instance 
> will not be useful when we deploy the production instance.
>
> Instead, what would help us the most is to configure your client to 
> access those lists you normally subscribe to, and to spend some time 
> next week browsing and searching those lists and exploring a few lists 
> that are new to you.
>
> You can find the details for where the test instance is listening, and 
> rudimentary instructions for setting up a few clients, at 
> <http://trac.tools.ietf.org/group/tools/trac/wiki/ImapTesting>. Please 
> improve that page as you see the opportunity.
>
> Thanks in advance for any time you can give this.
>
> RjS
>
>
>
>
>
>

---

From: Ralph Droms <[email-masked]>
Date: Wed, 21 Oct 2015 12:27:05 -0400

Robert - I followed the directions for Mail.app and I'm able to access the archives from the lists I subscribed to.

I have one question: I now have an "IETF Archive" subfolder in my Inbox folder, which shows a warning icon because the mailbox name is invalid.  Is there any way to remove that subfolder?

- Ralph

> On Oct 21, 2015, at 10:55 AM 10/21/15, Robert Sparks <[email-masked]> wrote:
> 
> (Be sure to delete and create a new account in your client - simply changing the hostname will not work well).
> 
> RjS
> 
>> We have been testing IMAP access to the IETF archives for several days now, with volunteers from the tools-discuss and ietf chairs lists, and it's been going well. Thank you if you've been helping test already (and please continue to do so)!
>> 
>> Now we need more testers. We'd like to get a simple measure of load, so we'd like to have as many folks as are willing accessing the test instance at the same time with various clients during the next week. It would be particularly helpful to have several people using the system next Thursday.
>> 
>> If you're willing to take the time to do this, please drop me a note directly (I've set reply-to on this message) so I have a feel for how many people are participating.
>> 
>> Please report any issues you find directly to me. Do not open tickets with the secretariat. Be aware that there are known issues with the data on the test instance (some lists have blank or truncated messages) that have not yet been addressed - it would be good to report any more you find to me, but they won't get fixed immediately.
>> 
>> Please use [email-masked] for general conversation. If you have feature requests, that's a good place to send them. One request we've had from a few testers so far is to provide an additional shared folder with a mailbox for each list that has only the "recent" messages from that list - perhaps the last 6 months. It would be good to hear from folks, after they've tried what's there now, whether that would be useful to spend development time on, or if that time would be better spent elsewhere.
>> 
>> The test instance is getting mail with only a slight delay after it comes through the lists - you should see current traffic.
>> 
>> Please resist the temptation to download the entire archive of every list. I've done it - it's huge (~28G when MailMate does it) and it it's a severe torture test for clients. Interrupt your client if it starts trying to do that to you. Caching a copy of this test instance will not be useful when we deploy the production instance.
>> 
>> Instead, what would help us the most is to configure your client to access those lists you normally subscribe to, and to spend some time next week browsing and searching those lists and exploring a few lists that are new to you.
>> 
>> You can find the details for where the test instance is listening, and rudimentary instructions for setting up a few clients, at <http://trac.tools.ietf.org/group/tools/trac/wiki/ImapTesting>. Please improve that page as you see the opportunity.
>>

---

From: Tim Wicinski <[email-masked]>
Date: Wed, 21 Oct 2015 19:03:59 +0100

I've run into the same problem with Thunderbird (Earlybird actually as 
I'm running the latest builds) as Ralph.   I assume it's not possible to 
remove regardless if I log in anonymously or with my account.

tim

---

From: Joseph Lorenzo Hall <[email-masked]>
Date: Wed, 21 Oct 2015 14:57:51 -0400

Totally the wrong thread, so I'll apologize profusely now.

GMail is set to turn on DMARC reject strict by sometime early next
year [1] being one of the last big web mail providers to do so; so,
Mailman "From:" rewriting (or turning on the ability for IETF list
subscribers to toggle this for their sends) might need to be done for
IETF lists.

best, Joe

[1]: https://threatpost.com/google-moving-gmail-to-strict-dmarc-implementation/115125/

On Thu, Aug 27, 2015 at 4:37 PM, Robert Sparks <[email-masked]> wrote:
> Folks -
>
> We have been testing IMAP access to the IETF archives for several days now,
> with volunteers from the tools-discuss and ietf chairs lists, and it's been
> going well. Thank you if you've been helping test already (and please
> continue to do so)!
>
> Now we need more testers. We'd like to get a simple measure of load, so we'd
> like to have as many folks as are willing accessing the test instance at the
> same time with various clients during the next week. It would be
> particularly helpful to have several people using the system next Thursday.
>
> directly (I've set reply-to on this message) so I have a feel for how many
> people are participating.
>
> Please report any issues you find directly to me. Do not open tickets with
> the secretariat. Be aware that there are known issues with the data on the
> test instance (some lists have blank or truncated messages) that have not
> yet been addressed - it would be good to report any more you find to me, but
> they won't get fixed immediately.
>
> Please use [email-masked] for general conversation. If you have
> feature requests, that's a good place to send them. One request we've had
> from a few testers so far is to provide an additional shared folder with a
> mailbox for each list that has only the "recent" messages from that list -
> perhaps the last 6 months. It would be good to hear from folks, after
> they've tried what's there now, whether that would be useful to spend
> development time on, or if that time would be better spent elsewhere.
>
> The test instance is getting mail with only a slight delay after it comes
> through the lists - you should see current traffic.
>
> Please resist the temptation to download the entire archive of every list.
> I've done it - it's huge (~28G when MailMate does it) and it it's a severe
> torture test for clients. Interrupt your client if it starts trying to do
> that to you. Caching a copy of this test instance will not be useful when we
> deploy the production instance.
>
> Instead, what would help us the most is to configure your client to access
> those lists you normally subscribe to, and to spend some time next week
> browsing and searching those lists and exploring a few lists that are new to
> you.
>

-- 
Joseph Lorenzo Hall
Chief Technologist
Center for Democracy & Technology
1634 I ST NW STE 1100
Washington DC 20006-4011
(p) 202-407-8825
(f) 202-637-0968
[email-masked]
PGP: https://josephhall.org/gpg-key
fingerprint: 3CA2 8D7B 9F6D DBD3 4B10  1607 5F86 6987 40A9 A871

---

From: Robert Sparks <[email-masked]>
Date: Wed, 21 Oct 2015 15:41:59 -0500

So, what's showing is the "Inbox", which some clients assume always 
exists, but doesn't at this server.

We've made some tweaks during the testing phase to not irritate those 
clients - we'll look at some more while in Yokohama. If you'll be there, 
please show me the result you're getting. (FWIW, I'm not getting any 
error badges on the clients I've set up - it would be interesting to see 
how we're configuring differently, and update the wiki).

It is safe to ignore. I understand it's annoying, and we will work to 
help get rid of at least the warning badge = but you might have to talk 
to your client vendor before the mailbox will go completely away.

RjS

On 10/21/15 1:03 PM, Tim Wicinski wrote:
> I'm running the latest builds) as Ralph.   I assume it's not possible 
> to remove regardless if I log in anonymously or with my account.
>
> tim
>
>
>> Robert - I followed the directions for Mail.app and I'm able to 
>> access the archives from the lists I subscribed to.
>>
>> I have one question: I now have an "IETF Archive" subfolder in my 
>> Inbox folder, which shows a warning icon because the mailbox name is 
>> invalid.  Is there any way to remove that subfolder?
>>
>> - Ralph
>>
>>>> We have been testing IMAP access to the IETF archives for several 
>>>> days now, with volunteers from the tools-discuss and ietf chairs 
>>>> lists, and it's been going well. Thank you if you've been helping 
>>>> test already (and please continue to do so)!
>>>>
>>>> Now we need more testers. We'd like to get a simple measure of 
>>>> load, so we'd like to have as many folks as are willing accessing 
>>>> the test instance at the same time with various clients during the 
>>>> next week. It would be particularly helpful to have several people 
>>>> using the system next Thursday.
>>>>
>>>> If you're willing to take the time to do this, please drop me a 
>>>> note directly (I've set reply-to on this message) so I have a feel 
>>>> for how many people are participating.
>>>>
>>>> Please report any issues you find directly to me. Do not open 
>>>> tickets with the secretariat. Be aware that there are known issues 
>>>> with the data on the test instance (some lists have blank or 
>>>> truncated messages) that have not yet been addressed - it would be 
>>>> good to report any more you find to me, but they won't get fixed 
>>>> immediately.
>>>>
>>>> have feature requests, that's a good place to send them. One 
>>>> request we've had from a few testers so far is to provide an 
>>>> additional shared folder with a mailbox for each list that has only 
>>>> the "recent" messages from that list - perhaps the last 6 months. 
>>>> It would be good to hear from folks, after they've tried what's 
>>>> there now, whether that would be useful to spend development time 
>>>> on, or if that time would be better spent elsewhere.
>>>>
>>>> Please resist the temptation to download the entire archive of 
>>>> every list. I've done it - it's huge (~28G when MailMate does it) 
>>>> and it it's a severe torture test for clients. Interrupt your 
>>>> client if it starts trying to do that to you. Caching a copy of 
>>>> this test instance will not be useful when we deploy the production 
>>>> instance.
>>>>
>>>> access those lists you normally subscribe to, and to spend some 
>>>> time next week browsing and searching those lists and exploring a 
>>>> few lists that are new to you.
>>>>
>>>> You can find the details for where the test instance is listening, 
>>>> and rudimentary instructions for setting up a few clients, at 
>>>> <http://trac.tools.ietf.org/group/tools/trac/wiki/ImapTesting>. 
>>>> Please improve that page as you see the opportunity.
>>>>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 22 Oct 2015 13:05:33 +1300

Joe,

It's easier to change the subject field than to apologise.

Experts,

Will the problem with Mailman distribution of DMARC-contaminated messages
be fixed in time for this,or must we gmail users start to look for a
more reasonable provider?

Regards
   Brian

---

From: Ross Finlayson <[email-masked]>
Date: Wed, 21 Oct 2015 18:59:45 -0700

> more reasonable provider?

Or perhaps people who subscribe to IETF mailing lists can start doing so using more professional-looking email addresses :-)

(Note that it’s possible, I think, to continue to use 3rd-party email services like ‘Gmail’ with email addresses that use your own domain name rather than “@gmail.com" (and, as a beneficial side effect, you won’t be subject to DMARC).)

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 22 Oct 2015 15:21:06 +1300

> Or perhaps people who subscribe to IETF mailing lists can start doing so using more professional-looking email addresses :-)

Um, why? We don't represent our employers here, and many people change
jobs or even countries but prefer their IETF email to continue uninterrupted.
I started using gmail precisely to avoid the problems and stupidities of corporate
email, and of access across travel and relocation, and I know I'm not alone.

> (Note that it’s possible, I think, to continue to use 3rd-party email services like ‘Gmail’ with email addresses that use your own domain name rather than “@gmail.com" (and, as a beneficial side ef…

Only, I think, if your corporate provider has made a suitable deal with G.

    Brian

---

From: "Andrew G. Malis" <[email-masked]>
Date: Thu, 22 Oct 2015 06:04:56 -0400

There is an easy answer to this, that we discussed on this list over a year
ago - just bring mailman up to a more recent revision. It can handle DMARC
re-writing just fine.

Cheers,
Andy

On Wed, Oct 21, 2015 at 10:21 PM, Brian E Carpenter <
[email-masked]> wrote:

> > Or perhaps people who subscribe to IETF mailing lists can start doing so
> using more professional-looking email addresses :-)
>
> > (Note that it’s possible, I think, to continue to use 3rd-party email
> services like ‘Gmail’ with email addresses that use your own domain name
> rather than “@gmail.com" (and, as a beneficial side effect, you won’t be
> subject to DMARC).)
>

---

From: "John Dickinson" <[email-masked]>
Date: Thu, 22 Oct 2015 12:17:46 +0100

> (Be sure to delete and create a new account in your client - simply changing the hostname will not work well).

Thanks for this it works very well with MailMate. Maybe I am asking a stupid question but in order to post to a list do I need to stay subscribed to it in the traditional manner or does the imap subscription do something clever?

regards
John

---

From: Rich Kulawiec <[email-masked]>
Date: Thu, 22 Oct 2015 09:24:56 -0400

1. I'm planning on testing this with fetchmail, which is a command line
POP/IMAP/etc. client.  I'd like to minimize the load that I impose on
your side and on the bandwidth that I chew up, so could you point me
to a mailbox that's (relatively) small?

2. Mailman stores list traffic in standard Unix mbox format.  I think
it would make sense to export those mboxes (via rsync) on a daily basis
to somewhere where they can be accessed via either https or rsync.
(Why?  Because that way someone attempting to retrieve an entire archive
doesn't need to do so one message at a time.  And anyone trying to
maintain their own local archive can just update it periodically by
re-rsync'ing.)

---rsk

---

From: Russ Housley <[email-masked]>
Date: Thu, 22 Oct 2015 09:57:58 -0400

> There is an easy answer to this, that we discussed on this list over a year ago - just bring mailman up to a more recent revision. It can handle DMARC re-writing just fine.

My understanding is that we are using Mailman 2.1.15.  Mailman 2.1.20 and 3.0.0 are available.  We have not asked the Secretariat to reviewed every change, but I am told that they are comfortable that Mailman 2.1.20 is fine.  We have not looked at Mailman 3.0.0 at all.

Versions of Mailman beyond 2.1.15 have no capacity for sending out password reminders; it was removed.  Brian, you have loudly argued for this capability to be retained.  

It seems to me that DMARC re-writing is a more important feature for this community.  I think we should drop support for the password messages and move to a newer version.  I'd like the tools team to check this out, and then if the newer version will not introduce other surprises, move to the newer version.

Russ

---

From: "Andrew G. Malis" <[email-masked]>
Date: Thu, 22 Oct 2015 10:09:44 -0400

Russ,

Thanks, I heartily agree.

Cheers,
Andy

On Thu, Oct 22, 2015 at 9:57 AM, Russ Housley <[email-masked]> wrote:

>
> > There is an easy answer to this, that we discussed on this list over a
> year ago - just bring mailman up to a more recent revision. It can handle
> DMARC re-writing just fine.
>
>
> My understanding is that we are using Mailman 2.1.15.  Mailman 2.1.20 and
> 3.0.0 are available.  We have not asked the Secretariat to reviewed every
> change, but I am told that they are comfortable that Mailman 2.1.20 is
> fine.  We have not looked at Mailman 3.0.0 at all.
>
> Versions of Mailman beyond 2.1.15 have no capacity for sending out
> password reminders; it was removed.  Brian, you have loudly argued for this
> capability to be retained.
>
> It seems to me that DMARC re-writing is a more important feature for this
> community.  I think we should drop support for the password messages and
> move to a newer version.  I'd like the tools team to check this out, and
> then if the newer version will not introduce other surprises, move to the
> newer version.
>
> Russ
>
>

---

From: Dave Crocker <[email-masked]>
Date: Thu, 22 Oct 2015 08:56:09 -0400

On 10/21/2015 2:03 PM, Tim Wicinski wrote:
> I've run into the same problem with Thunderbird (Earlybird actually

It's been a continuing issue for regular' Thunderbird...

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Paul Wouters <[email-masked]>
Date: Thu, 22 Oct 2015 10:38:51 -0400 (EDT)

> Versions of Mailman beyond 2.1.15 have no capacity for sending out password reminders; it was removed.  Brian, you have loudly argued for this capability to be retained.

I must have missed it between all the email reminders with passwords :)

> It seems to me that DMARC re-writing is a more important feature for this community.  I think we should drop support for the password messages and move to a newer version.  I'd like the tools team…

It is Oct 21 2015. While I am not insisting we have flying cars or
hooverboards, I think the IETF not sending plaintext passwords through
unencrypted email is highly overdue. It would be great if on Nov 1st
when my plane lands in Yokohama, I am not treated by a number of IETF
password emails.

Paul

---

From: Robert Sparks <[email-masked]>
Date: Thu, 22 Oct 2015 09:51:12 -0500

> Thanks for this it works very well with MailMate. Maybe I am asking a stupid question but in order to post to a list do I need to stay subscribed to it in the traditional manner or does the imap su…
If the list has a subscribers-only posting filter, you will need to stay 
subscribed to the list - nothing clever, as you suggest, is happening 
when you subscribe to the mailbox in IMAP.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 22 Oct 2015 11:02:00 -0400

> when my plane lands in Yokohama, I am not treated by a number of IETF

+1

The point of eating the dog food is to make better dog food, not get
used to the taste.

Right now we have a broken system of email of which the passwords are
a symptom. In general any time you have a protocol that involves a
password, it is a broken, obsolete system. Unfortunately we seem to
keep coming up with new ways to support passwords rather than an
approach that makes public key based auth as easy to use.

Mailing lists are a broken technology. Currently the IETF is running a
mail server that typically sends several hundred copies of each
message to gmail. Some of those can be batched, but not all of them.
Mine can't be batched because there is no way for mailman to know that
[email-masked] is actually an alias forwarded to a gmail
account. I have something like 5 gig of mail in my gmail account that
is from various mailing lists I subscribe to.

The way the system should work is through a variation of IMAP. Instead
of mailing lists being a layered application that is poorly supported
by and constantly fighting the SMTP infrastructure, mailing list
subscriptions should be handled in the mail client and by the server.
Essentially a mailing list is an IMAP folder that has multiple people
reading it, each of which need to track what has been read separately.

We used to have a system of that type, it was called NNTP.

---

From: "John Levine" <[email-masked]>
Date: 22 Oct 2015 15:42:48 -0000

>Totally the wrong thread, so I'll apologize profusely now.

Subject line changed.

>subscribers to toggle this for their sends) might need to be done for

Members of the DMARC group have submitted two drafts
draft-andersen-arc-00 and draft-jones-arc-usage-00 which describe a
mutation of DKIM intended to let mailing lists coexist with DMARC
without having to do ugly hacks like rewriting the From: line.  I have
reason to believe that several large mail systems intend to implement
this reasonably soon.

The obvious place to discuss this is the DMARC WG, not here.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 22 Oct 2015 15:44:25 -0000

In article <[email-masked]> you write:
>
>> There is an easy answer to this, ...

It's not an answer, it's an ugly hack, and turns out not to be needed,

Please see the two ARC drafts that I mentioned in a recent message.

R's,
John

---

From: Michael Richardson <[email-masked]>
Date: Thu, 22 Oct 2015 12:11:43 -0400

> reason to believe that several large mail systems intend to implement

Can you at least summarize where the drafts are in the process, and if they
require changes to senders, mailing lists, receivers, or all of the above?

Are the proposals (which I guess are not yet adopted) complementary, or will
a beauty contest be required?

I think that [email-masked] needs to know if we'll have a solution in time,
or if the secretariat needs to be asked to deploy mailman 3 sooner.

The very rough consensus a year ago was that we should wait for DMARC to do
something, and here is something.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Miles Fidelman <[email-masked]>
Date: Thu, 22 Oct 2015 12:23:42 -0400

> It seems to me that DMARC re-writing is a more important feature for this community.  I think we should drop support for the password messages and move to a newer version.  I'd like the tools team…

Or, one might consider a switch to a different list manager entirely.  
Sympa is really excellent for handling large numbers of lists, handles 
DMARC issues, doesn't send cleartext password reminders, is incredibly 
well supported.

Just a thought,

Miles Fidelman

-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Thu, 22 Oct 2015 18:41:54 +0200

There is a much bigger problem with the IETF mail servers, I’ve reported it several times to AMS and other folks from the IAOC have been informed as I understand, but nobody is providing any solution, and as a consequence, we are subjected to receive SPAM, as we are accepting emails from subscribers which are in spam black-lists.

This problem also means that if you own mail server (which many of us don’t control), is configured to reject messages from black-listed IPs, then when you reject emails from such folks, the IETF considers that your email address is bouncing, and unsubscribes you. I’ve this problem every other week, in the general area list and also in some others related to DNS (which probably means that the black-listed servers are from people contributing mainly to DNS exploders).

This is my view about the problem, which has been ignored up to now:

1) IETF is accepting messages from subscribers with MX which are Blacklisted.

2) IETF send those emails also with the original headers.
—> that’s why when AMS staff send the logs that I requested, it appears
like the IETF is black-listed, which is not the case, but a couple of 
senders to IETF exploders.

3) Mail servers inspect the IETF and original headers looking for 
black-listed servers.

4) If they are strict, bounce emails from IETF even if the IETF records 
aren’t black-listed but the original sender MX is black-listed

So, we need a policy here. Either:

A) IETF doesn’t accept messages from black-listed servers.
——> if we don’t do so, we may be affecting ALL the subscribers, and even
worst, a SPAMER could subscribe to IETF and send SPAM to ALL our 
subscribers. An even worst, once IETF mail es bounced because the 
black-list, some antispam systems increase the IETF rating as “possible 
spammer”, which is not good.

Or

B) IETF clean-up the original sender header (before sending to the mail 
exploder) to avoid bounces.

Alternatively, not sure how this will work, we could just allow “bounces” 
from subscribers (may be even a big number such as 40-50), instead of 
disabling the membership.

We may need to retry more times sending emails to subscribers, increasing the time between retries, and after no bouncing for “x” many days, resetting counters for this subscriber.

When I got the logs from AMS (august 2015), the offenders where:

149.171.193.32 - Australia Sydney University Of New South Wales, black-listed by sorbs

And

198.57.247.250 - United States New York City Unified Layer, black-listed by spamhaus

Regards,
Jordi

---

From: Rich Kulawiec <[email-masked]>
Date: Thu, 22 Oct 2015 13:20:18 -0400

On Thu, Oct 22, 2015 at 09:57:58AM -0400, Russ Housley wrote:
> My understanding is that we are using Mailman 2.1.15.  Mailman 2.1.20
> and 3.0.0 are available.  We have not asked the Secretariat to reviewed
> every change, but I am told that they are comfortable that Mailman 2.1.20
> is fine.

I have been doing extensive testing of 2.1.20 and my impression is that
a migration from 2.1.1X (including 2.1.15) should not present any major
issues.  This may not be the best long-term fix (although I remain
convinced that Mailman is best-available mailing list manager software)
but in the short-term, I think it will solve some of the problems
with a low investment in effort.

---rsk

---

From: "John R Levine" <[email-masked]>
Date: 22 Oct 2015 13:33:39 -0400

> require changes to senders, mailing lists, receivers, or all of the above?

ARC adds some new trace headers that provide a signed chain of custody. 
If you're already doing DKIM signing or verification, it should be 
straightforward to use upgraded DKIM libraries to apply them.  Receivers 
look at the ARC headers to decide when to accept a message despite DMARC 
failure.  As I said, big mail systems plan to do this.  There aren't any 
mailing list changes other than adding ARC headers which would be 
invisible to users.

> a beauty contest be required?

The work is outside the IETF, no beauty contest.

> or if the secretariat needs to be asked to deploy mailman 3 sooner.

I'd be pretty surprised if Gmail started publishing a DMARC policy before 
lists were able to deploy ARC.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

---

From: Rich Kulawiec <[email-masked]>
Date: Thu, 22 Oct 2015 13:36:33 -0400

On Thu, Oct 22, 2015 at 11:02:00AM -0400, Phillip Hallam-Baker wrote:
> Mailing lists are a broken technology. [big snip]

I disagree in part and agree in part.  Mailing lists are still massively
useful and *clearly* superior, by a huge margin, to things like web-based
message boards.  I'll go so far as to say that no organization or project
can be taken seriously unless it supports at least one mailing list
for announcements and/or communication.  On the other hand, mailing lists
have (recently) taken a beating thanks to the ill-advised deployment
of incompatible technology (like DMARC) by some major email providers.

Ironically, the very same email providers which have allegedly done
this to prevent email abuse are systemic and chronic *sources* of
email abuse AND are among the very worst about fielding complaints
and acting on them in a timely, efficient, comprehensive manner.

Thus while I agree in part with you, I would argue that it's not
mailing lists which are broken, per se, it's some very large email
operations that are broken, thanks to incompetence and negligence,
salted liberally with arrogance.

On the other hand, while mailing lists scale well -- to a point -- they

> We used to have a system of that type, it was called NNTP.

There are methods (supported by Mailman, incidentally) that allow
bidirectional gatewaying between mailing lists and newsgroups.
The Python language folks have been using that for years, and while
it's not without its occasional issues, it seems to work quite well.
(I've been monitoring both the SMTP- and NNTP-delivered versions
and comparing them.)

Is that kind of gatewaying something that might be appropriate
for the IETF's lists?

---rsk

---

From: Dave Crocker <[email-masked]>
Date: Thu, 22 Oct 2015 15:19:21 -0400

> 1) IETF is accepting messages from subscribers with MX which are Blacklisted.

There are many blacklists.  Some are well run.  Some less so.

I'll suggest that any effort that goes down the path of providing
abuse-related directions to IETF ops staff needs to develop a detailed
policy specification -- including citing the abuse information lists to
be consulted -- that there can be review and consensus on exactly what
is to be done.

d/

---

From: Dave Crocker <[email-masked]>
Date: Thu, 22 Oct 2015 15:23:08 -0400

> require changes to senders, mailing lists, receivers, or all of the above?

the effort is nascent.  the specifications have only just been released.

while promising, there is still quite a bit more hope than hard
knowledge about its efficacy.

also, while john's 'soon' is not unreasonable, we need to be careful
about making near-term operational decisions based on brand new
specifications that have no implementations, nevermind no operational
experience.

d/

---

From: David Morris <[email-masked]>
Date: Thu, 22 Oct 2015 12:30:35 -0700 (PDT)

> be consulted -- that there can be review and consensus on exactly what

+1 on Dave's suggestion. I've had to fight through issues caused by
poorly run blacklists. I'd be very concerned about any proposal to
make binary reject decisions based on any blacklist not controlled
by the IETF.

Dave Morris

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Thu, 22 Oct 2015 21:32:35 +0200

Hi Dave,

I understand that, but because we can’t “set” the policy of the companies that employee our subscribers, we need to be able to work with as many email providers as possible, which means being VERY strict.

There are two ways to achieve that, I think:

1) Using a comprehensive set of DNS-BL
	http://www.dnsbl.info/dnsbl-database-check.php

2) Not using any DNS-BL but then we need to clear the original headers before re-sending the email to the IETF exploders.

Regards,
Jordi

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 23 Oct 2015 08:40:58 +1300

> Versions of Mailman beyond 2.1.15 have no capacity for sending out password reminders; it was removed.  Brian, you have loudly argued for this capability to be retained. 

Yes, but amazingly enough I lost that battle. It's a PITA, but one that can be lived with.

> It seems to me that DMARC re-writing is a more important feature for this community.  I think we should drop support for the password messages and move to a newer version.  I'd like the tools team…

The primitive rewriting of the From is a bug in itself, because it destroys
important information (who sent the message, even if they are a non-subscriber).
What John Levine describes is hopeful, but it would be nice to have some assurance
from Google that they will actually wait until it's available before changing
their DMARC policy.

    Brian

---

From: Rich Kulawiec <[email-masked]>
Date: Thu, 22 Oct 2015 15:41:00 -0400

First, the correct slang term for unsolicited bulk email is "spam",
never "SPAM".  The latter is a product of the Hormel Corporation
and has nothing to do with email; the former is not an acronym and
thus should never be rendered in all-caps.

> 1) IETF is accepting messages from subscribers with MX which are Blacklisted.

Second, blacklisted by *whom*?  There are hundreds of public blacklists
and no doubt several orders of magnitude more private ones.  Some of
these list enormous swaths of network space.  I wouldn't be surprised
if *most* of the messages accepted by IETF mail servers originate from
hosts that are in some blacklist, somewhere.

(Also, I don't think you mean their MX's, necessarily.  You mean
their outbound mail servers, which may or may not be the same as
the hosts listed in their MX records.)

> aren???t black-listed but the original sender MX is black-listed

Third, there's your problem.  Don't do that.

If you subscribe to a mailing list, IETF or otherwise, you should accept
all traffic from that mailing list.  It's up to the keepers of the mailing
list to keep it from of spam, which these days isn't all that hard modulo
the occasional compromised list-member account.  And it's up to you to
accept all messages from the mailing list *because you subscribed to it*.

If you don't want to cede authority over list traffic decision-making
to the keepers-of-the-list, then don't subscribe to it.

---rsk

---

From: Christian Huitema <[email-masked]>
Date: Fri, 23 Oct 2015 00:07:39 +0000

On Thursday, October 22, 2015 12:41 PM, Brian E Carpenter wrote:
> > ...
> move to a newer version.  I'd like the tools team to check this out, and then
> if the newer version will not introduce other surprises, move to the newer
> version.
> 

+1.

Rewriting the "From:" header trains users to only look at the user friendly name, and to overlook the rewritten address. The potential for phishing is interesting.

> What John Levine describes is hopeful, but it would be nice to have some
> assurance from Google that they will actually wait until it's available before
> changing their DMARC policy.

Yes. But we may want to look a little bit at privacy issues. The privacy problems with disclosing the current IP address of the user in the Received field are flagged in RFC 7624. We need to make sure that the new ARC field does not amplify this issue.

-- Christian Huitema

---

From: "John R Levine" <[email-masked]>
Date: 22 Oct 2015 20:27:14 -0400

> the effort is nascent.  the specifications have only just been released.

That's certainly true, but it's pretty clear that the DMARC crowd wants to 
make it work so we mailing list users stop hating on them.

Regards,

---

From: Dave Crocker <[email-masked]>
Date: Fri, 23 Oct 2015 03:50:57 -0400

On 10/22/2015 8:27 PM, John R Levine wrote:
> That's certainly true, but it's pretty clear that the DMARC crowd wants
> to make it work so we mailing list users stop hating on them.

That is nice, but does not affect my recommendation, which is predicated
on a simple rule:

     When seeking to solve immediate issues, use existing, tested tools.
 /Do not rely on anything that is in the pipeline because you have
cannot have any reliable knowledge of when it will be operationally viable./

d/

---

From: Dave Crocker <[email-masked]>
Date: Fri, 23 Oct 2015 04:33:57 -0400

> Rewriting the "From:" header trains users to only look at the user friendly name, and to overlook the rewritten address. The potential for phishing is interesting.

Christian,

I don't like the re-writing either, mostly because it causes email
software to think that one person is (at least) two, when doing sorting
and searching, and therefore causes it to have some new semantic failure
scenarios.

But your premise that users get trained by any of this mostly goes
against research and experience:  Users mostly don't notice nuance in
the information in the message header and mostly don't notice anything
reliably and mostly can't be trained.

And no, that's not a slam at users, it's a reality of human factors
design and the body of interactive computer use research.

d/

---

From: [email-masked] (Martin Rex)
Date: Fri, 23 Oct 2015 20:36:31 +0200 (CEST)

Christian Huitema wrote:
> Brian E Carpenter wrote:
> Rewriting the "From:" header trains users to only look at the user
> friendly name, and to overlook the rewritten address.
> The potential for phishing is interesting.

I do not see any increased potential for phishing
Rather the opposite -- DMARC could be abused to give users a false
sense of security and fall to the flawed assumption that it would
authenticate the EMail author (which it doesn't).

Interestingly, there essentially is a legal requirement for rewriting
the From: header field for addresses that publish DMARC records when
delivering EMail to places where DMARC processing isn't outright illegal
(DMARC processing is clearly illegal in the EU), because handing an
EMail to an MTA that might be processing DMARC will be illegal in the EU
just as processing DMARC oneself.  So one of the alternatives to
unconditionally bouncing a DMARC-impaired EMail, will be rewriting the From:.

-Martin

---

From: Joseph Lorenzo Hall <[email-masked]>
Date: Fri, 23 Oct 2015 15:21:14 -0400

+1

Rewriting From: at the mailing list level seems to be a simple fix (so
not DMARC checks will fire or they'll fire against ietf.org's DNS
records and presumably go through fine) and writing a Reply-To: from
the original sender should avoid default reply-all nastiness.

This may be in fact what newer versions of Mailman do... best, Joe

---

From: "John Levine" <[email-masked]>
Date: 23 Oct 2015 19:47:00 -0000

>DMARC issues, doesn't send cleartext password reminders, is incredibly 

I think Sympa is just dandy, and am in the process of switching to it
from mj2, but its DMARC hackery isn't much different from Mailman's.
It does have remarkably good DKIM and S/MIME handling.

I have a back end shim that rewrites mail addresses as required to
avoid DMARC while remaining deliverable, e.g., if a message is from
[email-masked], it rewrites it to [email-masked] and
sets up a temporary local forward.

It's grody but it works remarkably well.  Anyone wants it, just ask.
It's written in perl and the full forwarding groditude uses a litttle
daemon (also written in perl) that will need some rewriting if your
MTA isn't qmail.

R's,
John

---

From: Rich Kulawiec <[email-masked]>
Date: Sat, 24 Oct 2015 09:33:48 -0400

> authenticate the EMail author (which it doesn't).

Precisely.  My spamtraps observe messages all day, every day that pass
whatever validation happens to be in play -- but are clearly forgeries.
And it's a VERY rare end user who is capable of making that same
determination.  Thus the warm fuzzies provided by mail clients that mark
messages as "validated" or "authenticated" or whatever term is used are
going to make these problems worse, not better.

Until the underlying security issues are fixed -- and I see absolutely
no signs that any of the 500-pound gorillas even *intend* to address
those at scale, let alone are actively engaged in doing so -- this (DMARC
and related) just wallpapers over the problem.

---rsk

---

From: Rich Kulawiec <[email-masked]>
Date: Sat, 24 Oct 2015 09:37:04 -0400

> reliably and mostly can't be trained.

+1.  I don't always agree with Dave about everything, but this is one
case where I'll echo his thoughts, with amplification.   "Attempting
to train users and/or use them as part of security/anti-abuse efforts"
is a known-failed approach, with precious few isolated and rare exceptions.

It would be nice if it were otherwise.  It's not.  It's never going to be.
So when evaluating various bits of technology like this, "can users be
trained?" is not a question that needs to be on table.  We already know
the answer.  We've known for decades.

---rsk

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 25 Oct 2015 08:49:34 +1300

>> authenticate the EMail author (which it doesn't).

Just for fun, I looked at a small sample of spam: the most recent 24
messages that gmail itself tagged as junk.

No false positives.
4 tagged as DMARC pass.
5 tagged as DMARC fail (gmail does not currently obey p=discard)
15 with no DMARC status.

Which suggests that DMARC status is pretty much orthogonal to spam detection,
on this small sample.

   Brian

> .

---

From: "John Levine" <[email-masked]>
Date: 24 Oct 2015 21:54:46 -0000

>Which suggests that DMARC status is pretty much orthogonal to spam detection,

DMARC has very little to do with spam detection.  Its original purpose
is to deter phishing of famous brands, of which paypal is the poster
child.  It works pretty well for that, since those organizations tend
to send all of their mail from a few places they control and (other
than Linkedin which is no great loss) the staff members who are on
mailing lists use addresses in other domains.

Last year AOL and Yahoo repurposed it after they each separately
allowed crooks to steal their users' address books, so AOL and Yahoo
users were getting spam from the AOL and Yahoo addresses of people
they knew, sent from outside AOL and Yahoo.  So AOL and then Yahoo
turned DMARC on, which was quite effective at stopping that particular
flavor of spam, but in the process forcing the costs of their security
failures on everyone else.

R's,
John

PS: To the obvious question of why don't crooks phish paypal from
lookalike domains, they do, but a remarkable number of them still use
the exact domain.  Partly that's because it's easier, partly because
if the exact address gets through, it can match entries in the
recipients' address books and get displayed in ways that makes it look
more credible.

---

From: Randy Bush <[email-masked]>
Date: Sun, 25 Oct 2015 08:40:09 +0900

< rant >

it really makes little difference.  face it, email standards are now
made by google, yahoo, aol, ... and shoved down our throats.  have you
seen the hoops one has to jump through to smtp over v6 to google?  they
outsource their alleged pain to the rest of the internet.

my paranoid brother wonders if this is intentionally pushing small smtp
sites to give up and use the mail cartel.  but i think that is just
collateral damage.

randy

---

From: Pierre-Elliott Bécue <[email-masked]>
Date: Sun, 25 Oct 2015 02:43:55 +0100

On jeu. 22 oct. 2015 à 12:23:42, Miles Fidelman wrote:
> Or, one might consider a switch to a different list manager entirely.  Sympa
> is really excellent for handling large numbers of lists, handles DMARC
> issues, doesn't send cleartext password reminders, is incredibly well
> supported.
> 
> Just a thought,

Actually, Mailman 3 got released during this year and seems to be really
good at fixing some great issues from previous versions.

But sure, the migration won't be painless.

-- 
PEB

---

From: Ted Lemon <[email-masked]>
Date: Sat, 24 Oct 2015 21:58:17 -0400

On Oct 24, 2015, at 7:40 PM, Randy Bush <[email-masked]> wrote:
> have you

No, actually I haven’t.   It Just Works for me.   I was surprised, but pleased.

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 25 Oct 2015 03:40:45 +0100

Den 25. okt. 2015 02:58, skrev Ted Lemon:
> On Oct 24, 2015, at 7:40 PM, Randy Bush <[email-masked]
> <mailto:[email-masked]>> wrote:
>> have you

The requirement in v6 that didn't exist in v4 is that you have to have a
PTR record for your MTA's IPv6 address before Google wants to let you
send SMTP over IPv6.

In IPv4, the absence of a PTR record was taken as a spam signal,
together with all the other spam signals; in IPv6, they decided to just
say "no".

---

From: Randy Bush <[email-masked]>
Date: Sun, 25 Oct 2015 12:02:12 +0900

> PTR record for your MTA's IPv6 address before Google wants to let you

and spf

randy

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 25 Oct 2015 04:04:31 +0100

>> PTR record for your MTA's IPv6 address before Google wants to let you

I don't have it. They let me (mork.alvestrand.no) in with only the PTR.

---

From: Ted Lemon <[email-masked]>
Date: Sat, 24 Oct 2015 23:34:30 -0400

>> PTR record for your MTA's IPv6 address before Google wants to let you

If you don’t have spf your mail is going to get scored way down my my mailer too.   SPF is an IETF standard, so you really can’t complain that Google are scary renegades for using it.

---

From: Paul Wouters <[email-masked]>
Date: Sun, 25 Oct 2015 12:33:46 +0100

> together with all the other spam signals; in IPv6, they decided to just

Which is stupid because many ISPs didn't run proper reverse for v6 and couldn't delegate to their customers.

Which broke many small email servers

Upon which the ISPs started auto generating PTR records.

Upon which the anti spammers rejected auto generated PTR as valid and rebroke the small email servers.

Upon which those mail systems just stopped using IPv6.

Upon which we all lost

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 25 Oct 2015 11:30:09 -0400

> Upon which those mail systems just stopped using IPv6.

What you are demonstrating is the need for a coordination body that:

1) Listens to all the stakeholders
2) Takes account of all the stakeholder concerns
3) Arrives at a consensus based on those concerns
4) Publishes them

The problem with the approach here that a lot of people take here is
that they are not interested in step 1 or 2. They treat the IETF
process as a means to get the outcome THEY desire and tough luck to
anyone else who isn't there.

If you want to arrive at an approach that sticks, you have to look at
the interests of all the stakeholders, even the ones that don't bother
to show up.

---

From: John C Klensin <[email-masked]>
Date: Sun, 25 Oct 2015 13:49:02 -0400

--On Saturday, October 24, 2015 09:33 -0400 Rich Kulawiec
<[email-masked]> wrote:

>...
> Precisely.  My spamtraps observe messages all day, every day
> that pass whatever validation happens to be in play -- but are
> clearly forgeries. And it's a VERY rare end user who is
> capable of making that same determination.  Thus the warm
> fuzzies provided by mail clients that mark messages as
> "validated" or "authenticated" or whatever term is used are

> Until the underlying security issues are fixed -- and I see
> absolutely no signs that any of the 500-pound gorillas even
> *intend* to address those at scale, let alone are actively
> engaged in doing so -- this (DMARC and related) just
> wallpapers over the problem.

Rich, Martin, John L. and others,

For several reasons, I've been try to stay out of this since a
few comments when DMARC first came up, but I think some
significant issues deserve command and IETF actions different
from "patch Mailmain".

First of all, this is, intentionally or not, more than wallpaper
for the 500-pound gorilla community.  By making mailing lists
less functional, it drives people, not to NNTP, but to their
proprietary "social"/ "collaborative" systems.  Those systems
are enhanced version of what we used to call "bulletin boards"
and allow them to do a much better job of identifying user
interests and selling ads than mailing lists that they don't
control.  As an alternative, it helps drive people to _their_
mailing lists, which obviously don't have problems with incoming
DMARC (or whatever) and which have the same effects relative to
information-collection.   I don't have any information that any
of those actors deployed DMARC for those commercial reasons, but
would be astonished if no one at any of them had not noticed the
side-effects by now.

Second, even though many of us know how to subvert or bypass
DMARC, etc., and I agree with Dave's comments about user
education, a commercial mail provider is in much the same
position as a financial institution: for the latter, it can be
important to claim that they are applying supposedly state of
the art protection mechanisms even if what that means if SSL 1.0
with certs there is no hope of being able to verify.

If one thing or another is changed to the point that the
validation helps, the methods are, as you have pointed out,
easily subverted.  That would fall into the fairly nasty
category known as "making the bad guys smarter".

--On Sunday, October 25, 2015 08:40 +0900 Randy Bush
<[email-masked]> wrote:

> < rant >
> 
> it really makes little difference.  face it, email standards
> are now made by google, yahoo, aol, ... and shoved down our
> throats.  have you seen the hoops one has to jump through to
> smtp over v6 to google?  they outsource their alleged pain to
> the rest of the internet.

Ok, it seems to me that the above --Rich's, John's, and Randy's
comments included along with others-- means the IETF has three
choices (each of which has small variations):

Option 1: Either what those providers has decided to do is
actually the right option or, because they are a family of 500
pound gorillas, they are going to get their way and we need to
go along.  If that is the correct or preferred way of looking at
this, then we should modify our servers to do what they like or
to work around the problems they create.   We can have a WG try
to make minor improvements in (in this case) DMARC, but with the
understanding that anything that modifies the contents of
"From:" violates the definition and semantics of that field as
identifying the human message originator and noting that we've
got "Sender:" and "Resent-*" fields that are intended for
situations in which the last entity to inject the message into
Internet mail is not the same as the human message originator
and that we, like a few centuries of postal services before us,
made a distinction between envelope and message header
information for a reason.  This option can be accompanied by
periodic whining and/or ranting, but with the understanding that
it is unlikely to accomplish anything.  I hope that, if the
effect of anything that comes out of the WG actually does modify
long-established semantics, it will be identified as "updating"
the relevant specs and subjected to the level of Last Call
scrutiny associated with such things.

(2) We decide that the long-established semantics are the right
ones, that we have made reasonable decisions about how email
works, and the DMARC (at least is strictly applied) encourages
bad behavior that threatens the mail infrastructure and/or the
idea of small or individual mail providers (noting the
significant privacy advantages of such servers).  On that basis,
we refuse to make changes simply to accommodate breakage by one
or more vendors.  We encourage the IAB or IESG to issue a
statement explaining why a collection of vendors introducing and
then enforcing de-facto standards, especially ones that have
nasty side-effects, is a bad idea and precisely the reason IETF,
ISOC, and others have endorsed the Open Stand principles.   If
IETF participants are hurt as a result of their choices of email
providers, too bad -- if they vote with their feet and maybe
complaints to local regulators (see below) perhaps the relevant
vendors will notice.  This may be the right approach whether one
believes that expected output from the DMARC WG will eventually
fix the problem but that large mail vendors will then need to be
persuaded to adopt it or if one believes DMARC is more
fundamentally flawed.

(3) We decide that this is really all ok, that nothing of
significance (other than perhaps upgrading Mailman) needs to be
done, that we are ok with whatever distortions of semantics
occur, and that the many comments of the last week or so just
constitute an instance of the IETF community's periodic need to
whine or rant and this situation just provided a good
opportunity.

> my paranoid brother wonders if this is intentionally pushing
> small smtp sites to give up and use the mail cartel.  but i
> think that is just collateral damage.

I do too.  However your paranoid brother has a point in terms of
what this might look like to a suspicious antitrust or
competitiveness regulator, especially if we made the sort of
statement suggested in (2) above.  Because we have friends at
the relevant vendors, especially Google, if we were going to go
ahead with (2), I think it would be important to give them a
strong heads-up and opportunity to work out joint statements.
But, we we believe in our standards and principles, I don't
think (1) and (3) really ought to be options.

I would hope this could be a discussion for the plenary in
Yokohama and that, if people believe the leadership is not
paying appropriate attention to the issues (either too much or
too little), they will discuss that problem with the Nomcom.

   best,
    john

---

From: [email-masked]
Date: Sun, 25 Oct 2015 07:38:40 -0700 (PDT)

> Which suggests that DMARC status is pretty much orthogonal to spam detection,

There's a certain domain associated with one of the largest ISP/MSPs, where
it's apparently very easy to create a bogus account and spam the world. So lots
of people do just that. Everything from that domain is signed with DKIM and the
domain has DMARC records.

I don't know anyone legitimate who uses that domain, but I do occasionally get
legitimate mail from that domain via a mailing list, which invariably breaks
the DKIM signature and hence fails DMARC checks.

So in this specific case DMARC is a 100% reliable indicator of spam: That
is, if the signature validates it's spam, if it doesn't it's not.

It's been this way for years.

But more generally, over years of testing millions of messages, I see DMARC
failure correlates positively with a message being spam. Enough that I use it
that way in my spam scoring.

You can find examples of practically any behavior you want if the sample
size is small enough.

				Ned

---

From: Philip Homburg <[email-masked]>
Date: Sun, 25 Oct 2015 20:44:50 +0100

>information for a reason.  

No idea how flexible mailman is, but how hard would it be to write some
code that
- detects if DMARC is in use
- if so copies the original From header and other headers worth preserving
  to something like X-Original-From, etc.
- and rewrites the From to a constant string:
  "This user mistakenly uses a DMARC protected system, see <some page that describes what is wrong>" <[email-masked]>

Using a constant string maximizes the incentive to use a proper mail server.

---

From: "John Levine" <[email-masked]>
Date: 25 Oct 2015 20:24:55 -0000

>code that ...

Instead of guessing, five seconds of googlage found this informative page:

http://wiki.list.org/DEV/DMARC

Re your suggestion, how many subscribers to IETF lists currently use
gmail?  What would happen if they were all summarily ejected?

R's,
John

---

From: Philip Homburg <[email-masked]>
Date: Sun, 25 Oct 2015 21:41:52 +0100

>http://wiki.list.org/DEV/DMARC

I guess this is a complex way of saying, no mailman doesn't do that?

>gmail?  What would happen if they were all summarily ejected?

My proposal was not to eject them but to mark them as second class citizens.

People who use unresponsive big e-mail providers off load a lot of problems
to the community. 

Using a constant from should be a good incentive to look for alternatives
but does not provent people from contributing.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 26 Oct 2015 11:09:16 +1300

On 26/10/2015 09:41, Philip Homburg wrote:
>> gmail?  

Only a kind person at the Secretariat could answer that definitively.
Based on a couple of lists I happen to administer, I estimate that
it's 15 to 20%.

Based on recent editions of the Narten posting summary, a similar
fraction of the traffic on this list is from gmail addresses.

>> What would happen if they were all summarily ejected?

In fact, today's problem (from my observations) is the small proportion
of IETF folk who post from yahoo.com addresses. Their traffic is simply
lost as far as any dmarc-obeying receiver is concerned.

Gmail won't be a problem until June 2016, as indicated in the story that
prompted this thread.

> My proposal was not to eject them but to mark them as second class citizens.

Indeed. And if it came to that, speaking for myself, I would switch to an address
not contaminated by dmarc p=reject, unless a good solution for mailing
lists was found first.

   Brian

---

From: Jelte Jansen <[email-masked]>
Date: Mon, 26 Oct 2015 16:54:35 +0100

>From what I read on that page, that is essentially what mailman does (at
least as one of the options), but in a slightly less confrontational way
(i.e. not From: 'bad mail config at sender', but simply From: 'replaced
by list'). It even notes that it breaks reply-to and search, though it
certainly doesn't make an exhaustive list of UI/client processing problems.

Jelte

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Oct 2015 12:26:20 -0400

--On Monday, October 26, 2015 16:54 +0100 Jelte Jansen
<[email-masked]> wrote:

> least as one of the options), but in a slightly less
> confrontational way (i.e. not From: 'bad mail config at
> sender', but simply From: 'replaced by list'). It even notes
> that it breaks reply-to and search, though it certainly
> doesn't make an exhaustive list of UI/client processing
> problems.

For better or worse, we've got a perfectly good, standard, way
of doing that when the list is to be treated as the sender.
That would be to ask developers of list software to use
Resent-From.  It has its own set of issues and the various bits
of software that try to tie keys and DNS records to mail
originators would need to be upgraded to understand the
semantics of "Resent-*" (and, btw, Sender) fields, but it would
conform to the standards, not loose information, and not closely
resemble a cheap hack.

     john

---

From: "John R Levine" <[email-masked]>
Date: 26 Oct 2015 12:31:29 -0400

> Resent-From.

ARC adds a cryptographic chain of signatures adopted from DKIM.  I suppose 
you could use an extended version of Resent-From but it'd amount to the 
same thing.

Regards,

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 26 Oct 2015 13:06:47 -0400

> conform to the standards, not loose information, and not closely

Clearly some folk are going to have to change their behavior.

What is the more practical approach, to ask a small group of
technically literate people to modify their behavior in a modest
fashion or to demand updates to every mail client and server on the
Internet?

Of course the response might be different depending on whether the
objective is to produce a document or arrive at a deployed solution.

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Oct 2015 13:07:04 -0400

--On Monday, October 26, 2015 12:31 -0400 John R Levine
<[email-masked]> wrote:

> ARC adds a cryptographic chain of signatures adopted from
> DKIM.  I suppose you could use an extended version of
> Resent-From but it'd amount to the same thing.

There are two differences, for whose importance I appeal to Dave
Crocker's comments about educating users.  

If I'm a casual (aka "normal") user
    From: Joe Bloggs <[email-masked]>
    Resent-from: fubar mailing list <[email-masked]>

Makes, or appears to make, good intuitive sense and involves no
information loss.  If that user's MUA hides the Resent-*
field(s), not much is lost either, indeed it looks like status
quo with that hidden and some additional documentation
(supplementing the List-* fields) if it is displayed.

By contract, the DKIM fields are clearly designed for processing
by computers, not people.  If they were routinely displayed to
those end users, I'd expect protests to MUA authors/vendors
asking that they be hidden as incomprehensible and annoying
noise.

That doesn't mean that an ARC signature chain is a bad idea.  I
have doubts about its effectiveness in practice, but, if it
help, that is great.   But changing the intuitive (and
business-letter-based) semantics of the "From:" field strikes me
as a bad idea that requires far more consideration than some
mail vendor (or cluster of such vendors) deciding it might make
their lives easier.   Of course, a combination of
address/signature chaining with intelligent use of "Sender:"
and/or "Resent-*" might bring benefits with few or note of the
obvious disadvantages.

    john

---

From: Ted Lemon <[email-masked]>
Date: Mon, 26 Oct 2015 17:08:17 +0000

If we decide that the long-established semantics are the right ones, then I think our email standards deserve to die, because they don't currently work.   I share your concern about email turning into closed bulletin boards, but the way to fix that is to accept that email as it is is badly broken, and try to fix it, not to get mad at Google et al. for refusing to continue to suffer the brokenness.

---

From: "John R Levine" <[email-masked]>
Date: 26 Oct 2015 13:10:52 -0400

> as a bad idea ...

It strikes pretty much everyone as a bad idea other than a handful who've 
drunk too much DMARC koolaid, a group that does *not* include the ARC 
authors.

The whole point of ARC is that lists do whatever they do now, and add ARC 
headers for automated parocessing like they added DKIM headers a few years 
back.  None of the headers visible to users are supposed to change.

Regards,

---

From: Mark Rousell <[email-masked]>
Date: Mon, 26 Oct 2015 17:23:17 +0000

On 23/10/2015 19:36, Martin Rex wrote:
> delivering EMail to places where DMARC processing isn't outright
> illegal (DMARC processing is clearly illegal in the EU), because
> handing an EMail to an MTA that might be processing DMARC will be
> illegal in the EU just as processing DMARC oneself. So one of the
> alternatives to unconditionally bouncing a DMARC-impaired EMail, will
> be rewriting the From:. -Martin 

I've been following the whole DMARC saga with some interest but I've not
previously noticed any suggestion that processing DMARC might be illegal
in the EU. Can you explain why you take the view that processing DMARC
should be illegal in the EU? For the avoidance of doubt, when you refer
to "DMARC processing" to what exactly are you referring?

-- 
Mark Rousell

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Oct 2015 14:45:15 -0400

--On Monday, October 26, 2015 17:08 +0000 Ted Lemon
<[email-masked]> wrote:

> If we decide that the long-established semantics are the right
> ones, then I think our email standards deserve to die, because
> they don't currently work.

Ted, I think millions of users, passing around tens or hundreds
of millions of messages around a day, would probably disagree
with "don't currently work" or at least dismiss it as rather
extreme hyperbole.  Now I can probably think of at least as many
ways in which I think the functionality would be different in a
more perfect world and the ability to positively identify a
sender (or identify spoofed messages or message components) and
to verify that what is received is what was sent, and to do both
without complex arrangements (private key management by end
users as just one example) are high on my list.   I would, of
course, like that done in a fashion that is completely
consistent with privacy, just as I count on the privacy of a
postal message sent in a sealed envelope (and may be deluded
about both).  I note, fwiw, that, to the extent to which I have
non-spoofing or non-tampering expectations of postal mail, those
expectations are largely based on assumptions built into
statutes, not because the protocols are particularly clever
about such things.

> I share your concern about email
> turning into closed bulletin boards, but the way to fix that
> is to accept that email as it is is badly broken, and try to
> fix it, not to get mad at Google et al. for refusing to
> continue to suffer the brokenness.

Perhaps I haven't been looking in the right places, but I
haven't heard Google claim that email is "badly broken", much
less "doesn't work".   What I have heard is some claims about
blocking of some messages originating from bogus or unauthorized
senders.  That is a sender authentication problem, not a "broken
email protocol" one.  As a solution to that problem, there there
have been several comments on this list to the effect that
DMARC, as Google apparently plans to use it, will not be
particularly effective.  Equally important, if Google really
cares about either sender authentication or verification that a
sender who uses a particular backward-pointing address today is
the same entity who used it yesterday, we know of a large
variety of ways to approximate at least the latter.  The
observation that Google isn't doing any of those things, even
the ones they could support with a very large fraction of their
users and without protocol changes, suggests that isn't the
issue.

So, if you are going to claim that our existing standards don't
work, I think it would be good to have a clear explanation of
what you mean and what, precisely, doesn't work.  Of course, I
can only hope that, contrary to your apparent claim, this
message will reach you in spite of non-working protocols and you
will be able to reply.

    john

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 26 Oct 2015 19:04:03 +0000

On Mon, Oct 26, 2015 at 02:45:15PM -0400, John C Klensin wrote:

> extreme hyperbole.

I'll heartily second that.  Email is not broken, rather it is
working surprisingly well, under requirements that make some
desirable security properties quite difficult to deliver.

    * Ubiquitous reach
    * Decentralized provisioning
    * Asynchronous one to many communication.
    * One to one and one to many forwarding

Various "non-broken" IM systems reduce the volume of abuse by
sacrificing one of more of the key features of email.

> Now I can probably think of at least as many
> users as just one example) are high on my list.

And this too, though positive identification is a rather slippery
fish in a world where domains cost close enough to $1 in bulk, and
reputable organizations continue find novel ways to muddle their
identity through various outsourcing arrangements and rebranding
initiatives.

The bad guys can acquire a constant stream of new identities, and
the good guys shoot themselves in the foot by periodically emulating
the bad guys.  The brokenness is not a feature of email, it is
rather a feature of scale.  I correspond by email with an order of
magnitude or two more people than I contact by IM or similar.

> message will reach you in spite of non-working protocols and you

Twisting the knife may be overkill. :-)

-- 
	Viktor.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 26 Oct 2015 16:01:12 -0400

> with "don't currently work" or at least dismiss it as rather

I will admit to using an extreme form of the term "work," which is "work without massive difficulties."   Most of the massive difficulties happen behind the scenes, so we fortunates do not have to deal with them, but the lengths mail providers like Google have to go to to keep spam out of our inboxes cast their shadow over the experience of every email user.   Web registration systems now tell users to whitelist mail from their domains, which would be cool if it could be done automatically rather than through manual intervention, and users are now accustomed to searching for important mail that hasn’t arrived in their junk folders.   I suspect the annual number of person-hours spent on these two tasks would humble us.

My experience of "badly broken": pre-filter, for my email address ([email-masked] <mailto:[email-masked]>) alone, spam is currently arriving at a rate of about three messages per minute.   Why?   Because the design assumptions for email did not account for the fact that in the wild, email is an ecosystem, not a cooperative venture, and there is money to be made with scattershot spam, and money to be made filtering it.

I’ve attempted to implement a whitelist on my server to make sure that mail from people I already know will get through.   What I’ve discovered is that aside from the big providers, nobody sets their SPF up right, so I can’t rely on it to validate whitelisted senders: I either have to just hope for the best, and accept the occasional joe job attack, or else I whitelist IP addresses, or come up with heuristics like "if the MX for a domain points to google, pretend that the domain owner set up SPF according to Google’s docs."

The amount of brainpower that’s required to keep this rickety train on the rails is astonishing.   It is no longer the case that someone like you or I with the resources of an individual can have a reasonably painless experience of operating an SMTP server.   To my mind, this means that SMTP does not "work."   There is no dependable method by which I can ask the question "did this email message come from the source that it claims to have come from" and get an answer.

That’s what I mean by "works."   And that’s why I have every sympathy with mail providers who are throwing up their hands in disgust and saying I’m going to be draconian, even though the specs don’t technically allow it.   If we want something different to happen, we have to figure out a way to allow it to happen, and not just say there’s no problem and Google ought to follow the specs.

> email protocol" one.

It’s true that if we had a reliable way of validating senders, SMTP could continue to operate.   And it might even be that if we had this mechanism in place, spammers would stop dumping crap on my mail server, so I wouldn’t have to pay for gigabytes per day of useless traffic to my server.   But in practice, it would be much better to use a protocol that didn’t even trigger a data transfer until the sender had been validated.

> Equally important, if Google really

Yes, we do know ways to do this.   Most of them are too expensive to be practical.

> message will reach you in spite of non-working protocols and you

I have all the IETF mailing lists whitelisted, and IETF has a correct SPF setup (and, actually, I get my IETF mail through nominum’s servers, and Nominum pays Google a lot of money to filter spam, and they use their big data analytics to do a good job of that, so in that sense it does indeed "work," but I shudder to think what the carbon footprint of each valid delivered message is if you amortize the cost over the total number of messages that had to be transmitted and examined.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 27 Oct 2015 07:56:01 -0700

> Web registration systems now tell users to whitelist mail from their
> domains, which would be cool if it could be done automatically rather
> than through manual intervention, and users are now accustomed to
> searching for important mail that hasn’t arrived in their junk folders.  

Consider that example as a touchstone to the larger set of issues, here:

   Anti-abuse mechanisms (and many other 'fixes') for email tend to be
added piecemeal and locally, rather than considered as end-to-end
systems issues.

   On the average, the focus of concern is local operations, rather than
user interaction and user experience.  It's simpler and quicker to work
that way, but it winds up turning email into a technical and functional
Tower of Babel. This approach to dealing with email problems has been an
issue for at least 40 years.  Recipients complained to local operators
who put some local hack in place to deal with the symptoms.  Often the
originator of the problem never hears about it.

   Email is by far the largest and most complex distributed system on
the Internet.  Having complex objects travel multiple hops along many
independent administrations is a pain.  So let's minimize hops and
administrations and eliminate the cases that need to greater flexibility
email gave with the more highly distributed administrative and
functional design.

   We need to properly distinguish identities from identifiers and we
need to properly distinguish among the many independent actors in the
handling sequence and we need to distinguish trust from mistrust.  Each
of these distinctions can produce very different approaches for
proposals.  For example, trying to look for bad actors is fundamentally
different from looking for good actors.  So is evaluating an operator
versus an author.

   What most folk have moved to is a model in which a user's identity,
identifier and operational support are all tightly integrated and
controlled by a single administration.  For simple scenarios, that works
well enough to cause people to treat more varied scenarios as 'problems'
rather than 'userful requirements'.

   The current evolution of systems changes for anti-abuse are largely
reactive -- the abusers set the agenda -- and largely turn email into a
Procrustean bed:  Only usage scenarios that are supported and protected
cleanly are the easiest and most restrictive on users.  Since that
satisfies the vast majority of current use, the disenfranchisement of
the much smaller minority of other, legitimate scenarios can be ignored,
independent of the problems caused to those who are disenfranchised.

Producing general, end-to-end changes that afford necessary protections
and support the widest possible range of functionality is much more work
than doing localized, incremental hacks.  (Not that the localized,
incremental hacks are cheap or quick either.)

We need to stop thinking we can eliminate abuse.  We need to start
thinking in terms of end-to-end trust models that scale and support
varied usage scenarios.  We need to focus on being able to use trust of
operators, more than on trust of authors.

And mostly, we need to stop creating one hack to fill in the gaps for
another hack.  Or rather -- since the hacks often are the result of an
emergency -- we need to stop thinking the hacks actually 'solve'
problems rather than merely creating new ones.

d/

---

From: John C Klensin <[email-masked]>
Date: Tue, 27 Oct 2015 13:47:46 -0400

For whatever it is worth (and the rarity in email protocol and
architecture discussions is, IMO, worth noting), I completely
agree with Dave's comments and analysis below.  

Let me add one observation about scope: to a considerable
extent, "solving" abuse by mechanisms close to the target
recipient also misses a lot what should be the point because the
damage -- measured in added hacks or additional processing and
bandwidth requirements-- is done almost as soon as the problem
message is sent.  By ignoring it and applying patches and
workarounds at the recipient end (whether those are to
accommodate badly-formatted messages or to mitigate deliberate
abuse) reduces or eliminates pressure on originating systems to
get things right, prevent the abusive behavior, or create legal
deterrents to that behavior that are enforced sufficiently to
act as a deterrent.  As Dave says, it has been that way for
40-plus years.  However, if we want fixes that keep the system
working, rather than more and more hacks that drive us toward a
small and homogeneous set of "acceptable" mail systems, we'd
better start looking at the whole system, figuring out what we
want, and making it work.

    john

--On Tuesday, October 27, 2015 07:56 -0700 Dave Crocker
<[email-masked]> wrote:

> 
>> Web registration systems now tell users to whitelist mail
>> from their domains, which would be cool if it could be done
>> automatically rather than through manual intervention, and
>> users are now accustomed to searching for important mail that
>> hasn't arrived in their junk folders.  
> 
> 
>    Anti-abuse mechanisms (and many other 'fixes') for email
> tend to be added piecemeal and locally, rather than considered
> as end-to-end systems issues.
> 
>    On the average, the focus of concern is local operations,
> rather than user interaction and user experience.  It's
> simpler and quicker to work that way, but it winds up turning
> email into a technical and functional Tower of Babel. This
> approach to dealing with email problems has been an issue for
> at least 40 years.  Recipients complained to local operators
> who put some local hack in place to deal with the symptoms.
> Often the originator of the problem never hears about it.
> 
>    Email is by far the largest and most complex distributed
> system on the Internet.  Having complex objects travel
> multiple hops along many independent administrations is a
> pain.  So let's minimize hops and administrations and
> eliminate the cases that need to greater flexibility email
> gave with the more highly distributed administrative and
> functional design.
> 
>    We need to properly distinguish identities from identifiers
> and we need to properly distinguish among the many independent
> actors in the handling sequence and we need to distinguish
> trust from mistrust.  Each of these distinctions can produce
> very different approaches for proposals.  For example, trying
> to look for bad actors is fundamentally different from looking
> for good actors.  So is evaluating an operator versus an
> author.
> 
>    What most folk have moved to is a model in which a user's
> identity, identifier and operational support are all tightly
> integrated and controlled by a single administration.  For
> simple scenarios, that works well enough to cause people to
> treat more varied scenarios as 'problems' rather than 'userful
> requirements'.
> 
>    The current evolution of systems changes for anti-abuse are
> largely reactive -- the abusers set the agenda -- and largely
> turn email into a Procrustean bed:  Only usage scenarios that
> are supported and protected cleanly are the easiest and most
> restrictive on users.  Since that satisfies the vast majority
> of current use, the disenfranchisement of the much smaller
> minority of other, legitimate scenarios can be ignored,
> Producing general, end-to-end changes that afford necessary
> protections and support the widest possible range of
> functionality is much more work than doing localized,
> incremental hacks.  (Not that the localized, incremental hacks
> are cheap or quick either.)
> 
> We need to stop thinking we can eliminate abuse.  We need to
> start thinking in terms of end-to-end trust models that scale
> and support varied usage scenarios.  We need to focus on being
> able to use trust of operators, more than on trust of authors.
> 
> And mostly, we need to stop creating one hack to fill in the
> gaps for another hack.  Or rather -- since the hacks often are
> the result of an emergency -- we need to stop thinking the
> hacks actually 'solve' problems rather than merely creating
> new ones.
> 
> d/

---

From: [email-masked] (Martin Rex)
Date: Tue, 27 Oct 2015 23:34:05 +0100 (CET)

Mark Rousell wrote:
>Martin Rex wrote:

I have been elaborating on this issue here before

http://www.ietf.org/mail-archive/web/ietf/current/msg87704.html

The issue is that an MTA operated by an ISP is a telecommunications
provider and subject to the telecommunication secrecy requirements
based on EU directive 2002/58/EC.

The sender of the telecommunication is the peer that connects the
MTA and sends the MAIL FROM ... of the SMTP transaction.

The receiver of the telecommunication is the identity (or multiple identities)
specified in the RCPT TO  of the SMTP transaction.

The publisher of a DNS DMARC policy record that happens to match
the domain part from the rfc822-From: from contents of an rfc5322
message that is conveyed within the SMTP transaction is a legal
third party to the telecommunication.

It is a criminal offence for the telecommunications provider to
tell legal third parties about telecommunications between senders
and receivers (such as what a DMARC policy p=report tries to achieve).

It is a criminal offence for the telecommunications provider to
take notice of the *Contents* of the telecommunication except for an
extremely narrow set of probable cause circumstances.  rfc822-From:
is part of the communication contents, not part of the "metadata".
(only the SMTP envelope is the communication metadata).

It is a criminal offence for the telecommunications provider to
suppress telecommunication for non-technical reasons of the
communication link from sender to receipient (such as processing
a DMARC p=reject policy).

Since these are all legal prohibitions, it is impossible to circumvent
these prohibitions through contract clauses.  Any contract clause that
tries to circumvent a legal prohibition is Null and Void from the start
(certainly here in Germany, Para. 134 BGB), and probably in several
other EU member states as well.

-Martin

---

From: "John Levine" <[email-masked]>
Date: 28 Oct 2015 00:02:46 -0000

>based on EU directive 2002/58/EC.

By that theory, spam filtering would be illegal, too, yet all the
German ISPs I know manage to do it.

R's,
John

---

From: "Niels Dettenbach (Syndicat.com)" <[email-masked]>
Date: Wed, 28 Oct 2015 08:48:27 +0100

>German ISPs I know manage to do it.

If i understand  correctly, it *is* illegal, except if the ISP / mail service customer allows it (i.e. by contract). Where spam fighting is getting "intrusive" be the meaning of that law (i.e. using Headers, subject, body etc.), is a bit unclear afaik.

So the most Herman ISPs (should) have a corresponding part in their contracts...

best regards,

Niels.
- ---
Niels Dettenbach
Syndicat IT & Internet
http://www.syndicat.com

---

From: Dave Crocker <[email-masked]>
Date: Wed, 28 Oct 2015 08:22:03 -0700

> So the most Herman ISPs (should) have a corresponding part in their contracts...

On the average over the years, substantive discussion of legal details,
relative to work in the IETF, has proved unhelpful to that work.

Sometimes the entertainment value has been significant, however, but not
any intended engineering guidance.

d/

---

From: Ted Lemon <[email-masked]>
Date: Wed, 28 Oct 2015 11:27:52 -0400

John, Dave, this is exactly the conversation I want to have.   Is this conversation happening somewhere in the IETF?   I confess that I haven’t put a ton of effort into figuring that out—I’ve been studying the current documents and have done some test implementations to eliminate some of my ignorance on this topic, but I haven’t quite gotten to the point where I know more of what to say than random ideas like "how do we close the loop between the browser and the MUA so that wanted notifications can be securely marked for reliable delivery" and that sort of thing.

> On Oct 27, 2015, at 1:47 PM, John C Klensin <[email-masked]> wrote:
>

---

From: Dave Crocker <[email-masked]>
Date: Wed, 28 Oct 2015 08:34:50 -0700

On 10/26/2015 1:01 PM, Ted Lemon wrote:
> My experience of "badly broken": pre-filter, for my email address
> ([email-masked] <mailto:[email-masked]>) alone, spam is currently
> arriving at a rate of about three messages per minute.   Why?   Because
> the design assumptions for email did not account for the fact that in
> the wild, email is an ecosystem, not a cooperative venture, and there is
> money to be made with scattershot spam, and money to be made filtering it.

For 25-30 years, the email system we use now wasn't "in the wild".  The
service was developed, operated and used in a more constrained
environment.  Failure to anticipate massive and fundamental changes in a
computing and communication environment, decades hence, is not exactly a
design failure.  A problem, of course.  But that's different from a failure.

On the other hand, assessing it as a failure fits into an increasingly
common template of changing the email operating environment's rules and
then suddenly reclassifying usage which worked fine in the past as now
being a 'violation'.

> The amount of brainpower that’s required to keep this rickety train on
> the rails is astonishing.   It is no longer the case that someone like
> you or I with the resources of an individual can have a reasonably
> painless experience of operating an SMTP server.   To my mind, this
> means that SMTP does not "work."  

If I'm reading your text correctly, you are saying that the fact that a
service that was once simple to operate but that now requires extensive
staffing and expertise subjects the service to an assessment of 'does
not work'?

Yet I'm pretty sure that that kind of transition from simplicity to
complexity that requires staffing and expertise is the hallmark of many
(all?) infrastructure services in all technologically developed cultures.

Few consumers operate their own telephone exchange or their own air
traffic control center or their own water purification center or...

d/

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 28 Oct 2015 17:11:22 +0000

> traffic control center or their own water purification center or...

And how many of those other services are "free", subsidized by
a surveillance opt-in?

Almost all my spam is of the 419 variety,  It is economical for
the scammer in large part because account creation at the large
providers has no cost, and because the 419 scammers don't use those
accounts to send email, rather the gmail, yahoo, ... mailboxes are
often just "Reply-To" mailboxes, and it is exceedingly difficult
to report that type of abuse to the very same reply mailbox providers.
(The abuse reporting web forms are atrocious and too tedious to
bother).

So, from where I site, the real problem is that mailboxes are cost
free, and the large providers have it uneconomical to run a mail
service that is accountable for abuse by its users, and yet are
"too large to block".

So the similarity to other services that require specialized skills
is real, but does not tell the whole story.  The email ecosystem
has rather peculiar economic externalities.

Imposing the costs on the sources of the costs might only be possible
with disruptive regulation that I don't see happening any time
soon.  (Crazy rules like mandatory account creation fees for
publically provided email?  Mandatory abuse desk SLAs? ...)

There are of course other abuse vectors, the above is not the whole
story, but it should be clear that the problems are very far removed
from protocols, they are mostly problems of economic externalities.

Should the makers of products hawked by affiliate marketing networks
be liable for spam?  Should banks be in part liable for the losses
of customers scammed by 419 victims (which might quickly lead to
banks requiring branch manager risk approvals to wire large sums
of money to Lagos).  

The financial system resorts to "hacks" to contain the costs of
fraud, because the costs of fraud management are lower than the
costs of imposing tighter controls.

Why should we expect the email network to be more secure than the
credit card networks, the cheque clearing networks, ...

Many large networks have externalities, fraud and abuse.  Radical
redesigns to address the externalities are rare, in most cases once
the network is established, it responds to attacks with workarounds,
not disruptive fundamental design changes.

-- 
	Viktor.

---

From: [email-masked] (Martin Rex)
Date: Thu, 29 Oct 2015 01:48:11 +0100 (CET)

> German ISPs I know manage to do it.

I agree with Dave Crocker, that discussing legal issues on IETF mailing
lists usually doesn't work well at all.  I would really prefer to stop
this particular sub-discussion here.

The problem with your "by that theory" is that this legal theory doesn't
work the way you think it does.

An anti-spam filtering scheme that happens before right before an EMail
is delivered into a personal mailbox, and which happens on a purely
opt-in basis by each individual EMail receipient is compatible with
the EU laws.

The EMail recipient as specified by the EMail sender is a rightful
participant of the communication, and the processing happens
"at the communication endpoint", and not "in transit".
The receipient is allowed to decide which data to "receive" and
which data to ignore or delete, and the receipient is also allowed
to voluntarily(!!) outsource spam filtering or pre-sorting to
his Mail provider.

Maybe a brief description of a decision from the German Federal Labour court
from 2009 helps understanding (1 AZR 515/08 from 20-Jan-2009).

A labour union sent (unsolicited) EMail advertisements to employees
at their work Email accounts, and the employer sued them to stop
sending such EMails to work email accounts of employees.

The German Federal Court (the highest appelate court on that matter)
ruled, that the employer must tolerate such EMails.  Only employees
themselves, individually and voluntarily, can send "opt-out" requests
for such EMails to unions, which the union will then have to respect.
Since labour unions are privileged by the german constitution, they
even enjoy a "cold call" privilege that is an exception to the cold
call prohibition for other businesses.

If an employer would try to filter out such union Emails by technical
means, it will be a criminal offense, and no amount of contracts and
contract clauses can make such action non-criminal.

-Martin

---

From: Niels Dettenbach <[email-masked]>
Date: Thu, 29 Oct 2015 10:42:49 +0100

> service that is accountable for abuse by its users, and yet are

Setting up an an email service is typically easy, but running an reliable 
email service is a much harder job. This is where many MTA admins or companies 
on the road of implementing their own MTA / internet email service fail over 
the time. Around 2-5 years ago there was an hype of having your own MTA / 
internet mailer system even as the smallest company - leading to many "crap" 
within the net, because of many different standard violations, bugs and 
security holes in the setups, made it very difficult to fight against abusive 
smtp usage. 

Anyhow: These times seems to be a bit over now... ß)

There are professional operating MTA / Relay and/or mail providers which offer 
email services on any level and for any expectations of user audiences - 
including very effective anti spam tech and active anti abuse management. 

Ii someeone is happy with some kind of "cost free" service at all, who usually 
has to save bucks on some points or generate revenue over other vectors (i.e. 
selling user profiles, sell ads, sell reputation of their users traffic 
etc.pp.), this is OK, but this is his decision. 

I know that there are companies relying on email and gelegated this to any 
lowest entry service - but users who really do expect a really (!) cost free 
high reliable / high volume, high quality email service are just out of 
reality. This is like on any service on the market.

(just my two cents)

best regards,

Niels.

-- 
 ---
 Niels Dettenbach
 Syndicat IT & Internet
 http://www.syndicat.com
 PGP: https://syndicat.com/pub_key.asc
 ---

---

From: Ted Lemon <[email-masked]>
Date: Thu, 29 Oct 2015 16:06:30 +0000

Neils, the question is, why is it so hard to set up your own mail server and have it basically do what you need?   Is it because it's really that hard, or is it because the right tools don't exist, or the problem hasn't been properly reduced to practice?   I think it's more the latter than the former.   It sounds like Postfix has improved since I last did a Postfix configuration, but it's still got too many knobs, because it addresses too many disjoint use cases.   And it looks like mailpile is starting to wrestle with the identity problem in a useful way, and there is other interesting work going on in the usability/identity/security areas, but I'm not hearing people here talk about how that's being worked on in the IETF.   I would hate to see a bunch of ad-hoc protocols with no spec and no interop.

---

From: [email-masked]
Date: Thu, 29 Oct 2015 11:08:41 -0700 (PDT)

> Neils, the question is, why is it so hard to set up your own mail server and
> have it basically do what you need? 

Ted, you keep saying things like this as if they are true. Simply put, they're
not. The obvious counterexample is MS Exchange, which, my personal dislike for
it notwithstanding, is hardly rocket science to install and configure. Lots of
people with very modest technical skills do it routinely. (The place where they
get into trouble is when they outgrow what they originally did, have to upgrade
the hardware, etc. This is where things can get to be a bit of a PITA with MS
Exchange.)

In fact it's the existence of MS Exchange that keeps a lot of vendors from
bothering with this space. (It's absolutely why we started moving away from it
~15 years ago.) Even so, there are other offerings, such as the server IETF
participant Hector Santos works on. Perhaps he can comment on how
straightforward his product is to set up.

Now, of course if you insist whatever obscure flavor of Linux floats your boat,
and then insist on using pure open source and compiling it from scratch, which
then more or less means different software for the store, MTA,  and AS/AV, and
then decide you want to twirl a bunch of knobs because they look interesting
and you might need them, then you have opted for an approach that's going to
take some time.

Heck, our software isn't targeted for this market segment, but set up is
basically download and run the messaging/directory server installer, run a
single configuration script that only asks a handful of simple questions,
install your certificate(s) for SSL/TLS, and finally hook in whatever AS/AV
solution you're using. (The last step typically consists of around five
commands in our configuration utility.) At this point you'll have a
full-function server set up for a single domain offering SMTP, SUBMIT, POP, and
IMAP. You'll then need to add the accounts you want using the LDAP tool of your
choice, or alternately you can install the provisioning tools, but that's
almost certainly overkill for a small setup with only a handful of accounts.

And yes, there's one of those nasty configuration files underneath it all, but
you never edit directly. It's all done through a management interface. The
template the initial configuration generator for it is only 310 lines long, and
a lot of that is comments, boilerplate, and XML cruft.

Frankly, these days the problem isn't setting things up, it's ongoing
maintenance. What's your backup strategy going to be? What's your approach for
monitoring store disk space and the MTA queues going to be? When our customers
call in the odds are it's going to be a maintenance or performance issue, not a
"I need to reconfigure" issue.

> Is it because it's really that hard, or is it because the right tools don't
> exist, or the problem hasn't been properly reduced to practice?   I think it's
> more the latter than the former.   It sounds like Postfix has improved since I
> last did a Postfix configuration, but it's still got too many knobs, because it
> addresses too many disjoint use cases.

Sigh. The existence of knobs does not imply that you have to turn them or even
know what they do. Our product has literally thousands of configuration
settings, but the way it's designed you don't even know they're there unless
you're using them.  And if you're running a simple single host setup it's
unlikely that you'd ever need much more than the default configuration. But if
you do need them, there's extensive documentation on them all, as well as
technical support.

> And it looks like mailpile is starting
> to wrestle with the identity problem in a useful way, and there is other
> interesting work going on in the usability/identity/security areas, but I'm not
> hearing people here talk about how that's being worked on in the IETF.   I
> would hate to see a bunch of ad-hoc protocols with no spec and no interop.

Since I categorically reject your thesis that initial setup of a simple mail
server is the Hurculean task you claim it is, I have no idea what work the IETF
could undertake to solve this nonexistent problem.

				Ned

---

From: Hector Santos <[email-masked]>
Date: Sat, 31 Oct 2015 13:40:16 -0400

> could undertake to solve this nonexistent problem.

Not sure if this helps... and I have not read the entire thread, I 
agree, SMTP is relatively simple. I think a good job was done in this 
regard for the IETF "SMTP" default design and operation.

Every package will differ.  We are 100% commercial, shrink wrapped, 
boxed, CD, etc. So I have always strived for "Customer comes first," 
"Getting it right the first time," high quality, lower TCO, 
simplicity, shorter lead time in installation to full operation so my 
IETF inputs are generally with that engineering mindset.  SMTP is just 
the networking part of the integrated application hosting platform (a 
BBS for the old school folks), so everything needs to be 
installed/setup relatively easy for the non-expert, lower experienced" 
administrator, sysop, operator, even "help desk" tech support guy.

Overall, we have to program the IETF output for our customers and for 
the most part, I used the sensible "IETF defaults" for "out of the 
box" operations.  I don't want additional support cost/problems so I'm 
very conservative and only support technically logical new things or 
changes that can work without trouble and doesn't open additional can 
of worms. I also tend to avoid anything that isn't cooperatively 
competitive, i.e, I won't support a vendor idea that benefits them or 
few only and exerts more trouble for others. For any exploration, I 
will add the necessary flexibility, revert switches, etc, if it makes 
sense and always, backward compatibility is first.

Server-side p-code compiled scripting allows for programmable hooks 
into the hosting platform, including the SMTP process. Threaded slave 
calls are made at each SMTP state. SPF is processed at RCPT state to 
begin processing the collected client IP, EHLO, MAIL FROM and RCPT TO 
SMTP level session parameters.  A YES/NO (250/550) response at RCPT. 
Greylisting, DKIM, ADSP, ATPS and now DMARC is processed at DATA 
state. A 250/45x/55x) response at DATA.  SPF FAIL Rejects are done at 
SMTP before DATA.  All this extra processing is important because it 
points to the increased SMTP overhead that we experience that can 
affect scaling needs.  What is the industry new average increase 
session time?  For my site where I have everything enabled, its about 
1-5 seconds or more.

Finally, in regards to the DMARC parts of this thread. I see more DKIM 
related changes that has a high potential for more overhead, more 
processing, more waste and work, i.e, double DKIM signature 
processing.  I am not a fan of the 5322 rewriting idea. It has the 
potential to conflict 5322.From stable platform designs.  But I will 
explore the "controls" that can be added or that may already exist.

-- 
HLS