Subject: [domainrep] Reinventing opt-in, message streams, reputation as abidance

From: Alessandro Vesely <[email-masked]>
Date: Sun, 17 Oct 2010 16:39:50 +0200

Hi all,
I wonder if this issue may be part of domainrep, as an intermediate 
step toward collaborative domain reputation tracking.  As concisely as 
I can, the issue is as follows:

   To modify usual opt-in mechanisms so as collect and share
   evidence about subscription requests.  This revised opt-in
   mechanism is to be standardized and applies equally well
   to mailing lists, newsletters, and bulk email of any kind.

The rationale should be self evident, especially considering that 
opt-in is mandatory in some countries --how can it ever be enforced if 
accuseds can only produce their own word?  More reasons below, 
including why opt-in may be relevant for domain reputation.

*message streams*
List Domain Signing Practices have been aired on the DKIM mailing 
list, hypothesizing possible relationships between the domain in 
List-Id or List-Post header fields and the d= tag.  Perhaps, that idea 
has to be revised, as newsletters don't often sport List-* fields.

I think we agree that it is fairly easy to mechanically identify 
whether a message belongs to a given message stream.  For opt-in 
purposes, we need a statement that identifies a given list with a 
specific message stream.

*why would MLMs change their behavior*?
This opt-in revision provides a receipt that a sender can use to 
respond to any charge of illicit sending, including abuse reports. 
Perhaps, this is more of a relief for newsletters than mailing lists 
proper.

By cooperating with target MTAs, mailing lists can provide a better 
user experience, e.g. by avoiding vacation messages, easier delivery 
to specific folders and whitelisting.

Reasons why subscribers or MLMs may want to keep subscription data as 
secret as possible may originate from fear of discrimination, or 
commercial competition concerns.  Divulging data to reputation 
trackers is not strictly necessary, in case such issues cannot be 
overcome.

*reputation as abidance*
A receipt --the evidence that a user opted in-- is a very interesting 
piece of data.  Adding them up results in the number of subscribers. 
Weighting that with the number of dubious deliveries can lead to 
determining a fairly good reputation measure, ranging 0 to 1 as 
Nathaniel said it should.

What is a dubious delivery (DD)?  Obviously, a delivery of a message 
that belongs to a list stream that the recipient hasn't subscribed to. 
  DDs can be detected by MDAs even with no human intervention:  That's 
what makes the resulting measure fairly good.  At 100% adoption it 
would be exact.

Note that a sender doesn't have to be scored 1.0 to be whitelisted.  A 
reasonable amount of "auto-subscriptions" can be tolerated, and users 
may be required to confirm them within a few DDs.

Bad guys can game such systems by creating huge sets of fake 
subscribers, so that real DDs always weight low in comparison. 
However, each ADMD can track its own partial view of reputation, and 
possibly exchange it with others in its network of trust, as already 
discussed on this list.  A rater can tell apart the good from the bad 
by monitoring automated feedback loops.

*conclusions*
Thus far, I've only spelled some requirements:  ADMD's mail servers 
should be involved in certifying a user's subscription.  We don't need 
a Global Mailing Lists Registry, but having some trusted raters would 
be a Good Thing (TM).

Implementation could be as simple as signing subscription requests and 
adding some CCs, or provide for a full blown subscription management 
protocol.

Would this fly?

---

From: Jeff Macdonald <[email-masked]>
Date: Tue, 19 Oct 2010 09:22:32 -0400

On Sun, Oct 17, 2010 at 10:39 AM, Alessandro Vesely <[email-masked]> wrote:
> Hi all,
> I wonder if this issue may be part of domainrep, as an intermediate step
> toward collaborative domain reputation tracking.  As concisely as I can, the
> issue is as follows:
>
> The rationale should be self evident, especially considering that opt-in is
> mandatory in some countries --how can it ever be enforced if accuseds can
> only produce their own word?  More reasons below, including why opt-in may
> be relevant for domain reputation.
>
> *message streams*
> List Domain Signing Practices have been aired on the DKIM mailing list,
> hypothesizing possible relationships between the domain in List-Id or
> List-Post header fields and the d= tag.  Perhaps, that idea has to be
> revised, as newsletters don't often sport List-* fields.
>
> I think we agree that it is fairly easy to mechanically identify whether a
> message belongs to a given message stream.  For opt-in purposes, we need a
> statement that identifies a given list with a specific message stream.

<snip>

Doesn't a DKIM domain identifier do just that? And to recognize
opt-in, couldn't receiving systems allow those identifiers to just be
added to the recipients 'safe list'?

I don't think it is necessary to identify what the message stream type
is. I think it is good enough to say if it is wanted or not, with an
unknown stream being treated as neutral.

-- 
Jeff Macdonald
Ayer, MA

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 19 Oct 2010 16:47:33 +0200

On 19/Oct/10 15:22, Jeff Macdonald wrote:
> On Sun, Oct 17, 2010 at 10:39 AM, Alessandro Vesely<[email-masked]>  wrote:
>>  *message streams*
> Doesn't a DKIM domain identifier do just that?

No, for example, I get d=mipassoc.org for all of their lists.

> And to recognize opt-in, couldn't receiving systems allow those
> identifiers to just be added to the recipients 'safe list'?

Yes, but asking users to manually allow an identifier every time they 
subscribe to a list does not yield much.  What if gmail (or other 
mailbox provider) provided a web form for you to confirm subscriptions 
to external lists?  That form would also let users specify what 
(existing or new) folder they'd like the corresponding stream messages 
to land at, all with a single click.

> unknown stream being treated as neutral.

You either subscribed to a list or you didn't.  There is no such thing 
as a neutral subscription.

---

From: Jeff Macdonald <[email-masked]>
Date: Tue, 19 Oct 2010 15:40:40 -0400

> No, for example, I get d=mipassoc.org for all of their lists.

Ah, so, that is a configuration issue that Dave should fix. ;)

> subscribe to a list does not yield much.

So users do an 'add to address book' which in many implementations is
the same as adding to the 'safe list'. Adding an attribute for the
DKIM domain is just an extension of that. Opt-in is recorded at the
receiver side.

> What if gmail (or other mailbox
> provider) provided a web form for you to confirm subscriptions to external
> lists?

That seems to me to be just a roundabout way of doing 'add to address book'.

> That form would also let users specify what (existing or new) folder
> they'd like the corresponding stream messages to land at, all with a single
> click.

I think that is a bit out of scope, but yes, I'd love a web based
email provider to provide the same functionality of my procmail
recipes of the past.

> You either subscribed to a list or you didn't.  There is no such thing as a
> neutral subscription.

Email streams are not limited to mailing lists, but yes, in the case
of lists I agree with you.

As a side note, I find that using just d= for web/free mail providers
provides no value. Here having i= be populated with the Author address
if the hosting provider is willing to vouch for it may be of value.
Just thinking out loud here.

-- 
Jeff Macdonald
Ayer, MA

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 20 Oct 2010 12:46:56 +0200

On 19/Oct/10 21:40, Jeff Macdonald wrote:
> On Tue, Oct 19, 2010 at 10:47 AM, Alessandro Vesely<[email-masked]>  wrote:

As good as it may be, whitelisting by address book is limited by some 
facts:

* local idiosyncrasies are possible, as it is not standardized,
* users not always add senders to an address book, specially for
   newsletters, and
* even when they do, they don't do it synchronously with the
   confirmation, so that some messages may get through in between.

>> What if gmail (or other mailbox provider) provided a web form for
>> you to confirm subscriptions to external lists?
>

Except that the limitations above can be overcome.  Since users would 
run that dialog /instead of/ confirming their subscriptions at the 
sender's, they would have to do it every time they subscribe to a 
list.  Mailbox providers would then take care of forwarding each 
confirmation to the relevant sender.

>> That form would also let users specify what (existing or new)
>> folder they'd like the corresponding stream messages to land at,
>> all with a single click.
>

Yes, it's out of scope.  That extra functionality is just the kind of 
sugar that may make indirect subscriptions appealing.

> of lists I agree with you.

Except for any added sugar, for mailing lists it would be pretty 
useless to verify subscriptions the way I'm proposing.  Instead, most 
of the people I've talked to are quite skeptical about the handling of 
mail addresses they (have to) give to commercial newsletters.

Although justified by a number of cases, that skepticism is possibly 
unfair toward honest organizations or ESPs.  While initiatives like 
Project Honey Pot tend to highlight the bad guys, verified opt-ins 
would contribute positive reputation to signers of honestly managed 
newsletters.

I think this proposal should not aim at affecting deliverability in 
any way.  It is just to compute domain reputation.

> Just thinking out loud here.

OTOH, signing the "From" header field they substantially convey the 
same information.  Although i= could be good to identify lists, 
List-Id or List-Post look much clearer.  In retrospect, maybe i= could 
have been avoided altogether.

---

From: Jeff Macdonald <[email-masked]>
Date: Thu, 21 Oct 2010 09:48:36 -0400

On Wed, Oct 20, 2010 at 6:46 AM, Alessandro Vesely <[email-masked]> wrote:
>> I wrote:
> OTOH, signing the "From" header field they substantially convey the same
> information.  Although i= could be good to identify lists, List-Id or
> List-Post look much clearer.  In retrospect, maybe i= could have been
> avoided altogether.

By having a signer provide more data in i=, it could provide an
indication that the signer has done more due diligence. By having the
i= be the From, the domain owner could be asserting that in addition
of taking responsibility for the message, the domain owner has
confidence in the Author.

It is of course up to the receiving system if such information is
worth anything. The alternative is to have free mail providers not
sign messages for Authors of unknown quality.

i= is suppose to be opaque. So we don't really know if it is tied to
an Author. But it does provide a 'stream' within a 'river' (where d=
is the river) if you know what I mean.

-- 
Jeff Macdonald
Ayer, MA

---

From: "J.D. Falk" <[email-masked]>
Date: Thu, 21 Oct 2010 08:57:20 -0600

> Would this fly?

I've seen similar ideas before, but they've never gone anywhere.  That's not to say this can't, just...it's a big project, requiring a lot of changes to all types of MUAs and all types of MLMs.

I'd urge you to attempt it as an interoperability project with some open source (or proprietary) implementations, and work the bugs out that way before proposing it as a standard.

In any case, I don't think it fits into DOMAINREP just yet -- DOMAINREP will be complex and busy enough by itself.

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 22 Oct 2010 21:19:54 +0200

On 21/Oct/10 16:57, J.D. Falk wrote:
> I've seen similar ideas before, but they've never gone anywhere.
> That's not to say this can't, just...it's a big project, requiring
> a lot of changes to all types of MUAs and all types of MLMs.

No changes are needed on MUAs, as they don't recognize COI, AFAIK.

MLMs and other COI implementations would require changes, though.

> I'd urge you to attempt it as an interoperability project with some
> open source (or proprietary) implementations, and work the bugs out
> that way before proposing it as a standard.

Thank you for this injection of realness.  However, some foregoing 
paperwork is necessary in order to build a working prototype for an 
interoperable 3-way COI.  In particular, a bulk sender needs some 
means to contact the relevant domain's mail server.  For example, when 
it gets a subscription request from [email-masked] it has to learn 
whether example.com can either do the confirmation with their user on 
its behalf and then report the result, or if it shall do the 
confirmation as usual and then possibly send a notification to the domain.

Would it make sense to send notifications to [email-masked], 
assuming they should be interested --like some do with abuse@ for ARF?

Then, I would need at least one mailing list (or other bulk sender) as 
well as some mail domains with users that might subscribe to that 
list(s).  Perhaps, I should send an appeal seeking for participants to 
the experiment... Any hint?

> In any case, I don't think it fits into DOMAINREP just yet --
> DOMAINREP will be complex and busy enough by itself.

Do you mean using this list to discuss the prototype, or working at a 
draft when the prototype will work and domainrep will be a WG?

In either case, this triple opt-in has no other relevant outcome than 
computing good reputation values for domains that make use of email 
marketing. (Are there conspicuous domains that don't do it?)  So, I 
think I'd have a hard time making sense of it out of here.  Would it 
be possible to make a sub-list/sub-group?

---

From: Tony Hansen <[email-masked]>
Date: Sun, 24 Oct 2010 21:23:09 -0400

On 10/21/2010 9:48 AM, Jeff Macdonald wrote:
> is the river) if you know what I mean

I've felt for a while that we needed an extension to the DKIM key 
records that allowed a domain to express how it generates i= values, 
that is, to shine some light into that opaqueness.

This would be a purely optional field that would allow a domain to say, 
for example, that it uses author's email address identities in the i= 
field that it has verified through its authentication processes.

But the idea is not limited there, as a domain could state that it uses 
some other unique identity that's tied to the author's authenticated 
identity.

Thoughts?

     Tony Hansen
     [email-masked]

---

From: John Levine <[email-masked]>
Date: 25 Oct 2010 04:11:20 -0000

>field that it has verified through its authentication processes.

I'm trying to figure out what problem this solves.

"Yeah, we send a lot of spam, but it all has a real return address"!

R's,
John

---

From: Tony Hansen <[email-masked]>
Date: Mon, 25 Oct 2010 08:28:34 -0400

> "Yeah, we send a lot of spam, but it all has a real return address"!
Right now the value in the localpart of the i= value is totally opaque.
Right now you can't say anything about the reputation of the localpart 
identifier in i= because you don't know anything about the policy of the 
sender when creating it.
This would allow the sender to state their generating policy, and the 
verifier to take advantage of it.
I feel that it can strengthen the white list uses of DKIM.

     Tony Hansen

---

From: "John R. Levine" <[email-masked]>
Date: 25 Oct 2010 10:27:16 -0400

> I feel that it can strengthen the white list uses of DKIM.

Can you offer some examples?  It still seems to me that by the time you 
know that you believe a signer's rule for constructing the i= value, you 
probably want to whitelist all his mail anyway.

Conversely, if the i= really indicates that there are multiple mail 
streams, why wouldn't it be better for him to use different d= domains?

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Douglas Otis <[email-masked]>
Date: Mon, 25 Oct 2010 14:47:26 -0700

On 10/25/10 5:28 AM, Tony Hansen wrote:
> identifier in i= because you don't know anything about the policy of 
> the sender when creating it.
Currently, many of the larger domains have a significant number of 
compromised accounts issuing messages through their servers.  Transient 
opaque identifiers could be used to indicate that this identifier is 
useful to the sender for identifying potentially problematic accounts, 
without this field reflecting any specific email-address.  Adopting a u= 
parameter for such use might be more appropriate.  The i= is currently 
defined to match against the Author Domain email-address in ADSP, and to 
identify the on-behalf-of identity for the signature.  This value might 
be opaque, or it might not be.  It would seem better to have an 
identifier specifically defined to be opaque.

Just consider the problems related to FB with their "opaque identifiers" 
that consistently mapped to personally identifiable information.  Using 
this type of trace mechanism for email would be valuable, but it must be 
done with great care.

-Doug