Subject: RE: Appeal: Publication of draft-lyon-senderid-core-01 inconflictwith referenced draft-schlitt-spf-classic-02

From: "william(at)elan.net" <[email-masked]>
Date: Sat, 27 Aug 2005 12:00:30 -0700 (PDT)

On Fri, 26 Aug 2005, John Glube wrote:

> |The only relevant boundary is between what the sender
> |controls and what they do not. All that any sender,
> |forwarder or any other mail injector can ever be expected
> |to do is to define the boundaries of the systems they
> |control.
> |
> |Once that boundary is defined the definition is fair game
> |for any party to use to interpret it to meet their
> |operational needs.
>
> I have a real problem with this last position.
>
> The underlying thrust behind sender authentication is to
> establish a framework for use by certification and
> reputation services, both external and internal to
> facilitate the delivery of ham, while rejecting spam. [1]

With such a statement you're making an assumption that the technology
(i.e. SPF and SID) is only good for sender authentication and anti-spam.

This is a common mistake, especially in regards to SPF, and is largely
due to how the technology has been marketed by many. But it is all not
technically correct as SPF aims at stopping spoofing of RFC2821 MAILFROM 
address and to prevent backcuster of delivery failure messages when 
address is misused.

Also what is true about SPF may not be true about SID and as they operate
on different identities you're bound to see problems if incorrect record 
is used.

> A sender publishes a policy record based on a protocol in
> anticipation that recipient networks will use that record
> in accordance with that protocol.

Correct. And with two conflicting specifications it is difficult to 
understand what the intent of the sender is. This is a problem for
both SID and SPF protocols and unless resolved it discourages
participation of users in these experiments and thus IETF would
be enable to get true results about usability of the technology.

[skip]

> "The conflict" apparently is that the Schlitt draft
> recommends against use of spfv1 records for use with PRA
> authentication, while the Lyon draft says that receivers
> can use the spfv1 records for use with PRA authentication
> and therefore senders need to ensure their spfv1 records
> are suitable for this purpose.
>
> How did this conflict arise? It arose after the closure of
> MARID and prior to the filing of senderid-core-00.
>
> Let's remember the scene. The open source community was up
> in arms over the IPR disclosure by Microsoft.

The IPR disclosure's problem was not actually IPR limited text itself,
the main problem was how it happened and that many (I'll go as far as
to say majority of individual participants at MARID) lost trust in 
Microsoft especially after the disclosure of their overreaching patent 
applications. It is really failure of Microsoft to be able to play
fair and work together with open-source and similar communities even when 
both sides were willing to try for common good (and the technical people 
from Microsoft participating at MARID were not at all responsible for it, 
they were victims of the decisions made quite deep at their company).

Further work of the WG would have been very difficult with two sides not 
being able to easily talk to each other (which would make it difficult to 
come to consensus, although if given time to cool down it could have been 
possible, but you can see this difficulty even now with SPF and SID drafts).

> At the same time America Online had publicly withdrawn its support
> for Sender-ID due to the lack of backward compatibility between
> spf2.0 and spf1.

I don't think it was the reason. They knew about luck of backwards
compatibility even before and did not say anything. I'll go as far
as to say that somebody asked AOL to temporary withdraw support and
they were willing to play along.

> Mr. Wong, over the strong objections of many within the SPF
> community, went ahead and negotiated an arrangement that
> allowed for the backward compatibility of PRA
> authentication with spfv1 records. [2]

The problem is that Meng never understood about real causes for failure
of MARID as I tried to describe above and did not understand concerns of 
those who actually created SPF (he was editor of the original draft and 
"managed" the group but not really author of the technology which came 
together based on work of individual participants at spf-discuss), he 
assumed himself to be appropriate representative of open-source-like 
group that created SPF and that he could on their behalf continue to
talk and work further with MS to create common standard as MS wanted
from MARID, but in the way he did it, he just showed further where
MARID was failing and why it was appropriate for work to now be done
on two *separate* "experiments" as IETF asked.

The result is that there was a lot of outrage at the spf-discuss in 
October/November 2004 when new individual submission SID draft came
out with reuse of v=spf1 records and consequently Meng was removed
from being person responsible for representing views of SPF and 
replaced with group of people (SPF Council), plus primary editor
of the SPF documents was also replaced. This was not unexpected if
one could understand what happened at MARID.

> Further to this arrangement:
>
> * The Lentczner draft for spfv1, which only supported mail
> from authentication and the Lyon draft for spf2, which
> supported both mail from and PRA authentication were filed
> with the IESG.
>
> * Both drafts flowed from MARID.

Partially true (drafts did "flow" from MARID, but there were already
significant differences then the actual MARID drafts).

And as far as Mark Lentczner's spf draft it was already partly in the
way SPF community wanted as far as documenting real v=spf1, but he was 
just not willing to take it further (probably also did not understand
what happened at MARID and wanted to continue that work largely as is, 
plus he probably knew about agreements Meng was at the same time making 
behind the scenes with MS and AOL).

> * The Lyon draft supported backward compatibility and in
> response America Online announced its continued support for
> Sender ID.

It would be nice to know what happened behind the scenes. In fact it
what you see here is an example of why IETF standardization work should
be done in public instead of private.

> * Microsoft published an SPF wizard which supported the
> publication of spfv1 records, subject to the need of
> publishers to be satisfied their spfv1 records could be
> used for both mail from and PRA authentication.
>
> Shortly afterwards, due in large measure to the anger and
> upset over Mr. Wong's management of the relationship with
> Microsoft and the resulting failure to fully separate the
> Sender Policy Framework from Sender ID after the collapse
> of MARID, the SPF Community elected a Council to run the
> affairs of the SPF Community.
>
> In accord with its mandate, the Council proceeded to
> withdraw the Lentczner draft from the IESG and substitute in
> its place the Schlitt draft.

See above, I think I explained more how this all really came about.

> The IESG was now faced with a situation where the Lyon
> draft, which had relied upon the Lentczner draft, both of
> which flowed out of MARID, was now relying on the Schlitt
> draft for its underpinning.

Yes. And it is all result of that unfortunate attempt to "continue
MARID" with spf1 taken by Meng instead of proceeding as me and several 
others originally expected as far as having v=spf1 documented first and 
then quickly move to SPF2 as separate document series based on MARID and 
proceeding further to UnifiedSPF (i.e. one common format for policy 
records with separate identities/scopes and how to interpret those 
identity records). That second document series would have been real 
successor to MARID but it did not happen.

What you have now is that neither of the documents approved by IESG is 
really MARID documents and they really do need to be viewed as separate 
individual submissions and should be judged on their own merits.

SPF draft is best attempt so far to document SPF protocol as it was
developed prior to MARID (with only few things borrowed from MARID)

draft-lyon-senderid and other SID documents should be viewed as separate 
private submission of proprietary Microsoft CID implementation which 
instead of original CID XML DNS record has now adapted SPF record syntax.
And as separate submission made to IETF, it should also be responsibility 
of IESG that their proposal does not come into conflict with other LMAP
proposed experiments and does not break any other IETF standards.

> In turn the Schlitt draft re-introduced helo authentication
> based on historical usage and contains the recommendation
> which conflicts with the backward compatibility arrangement
> in the Lyon draft. [3]

Again, please understand that v=spf1 document does not represent MARID 
work perse - its separate submission documenting pre-MARID protocol 
slightly further developed and better documented (with that documenting
being in part result of working at MARID).

> Since neither side was prepared to budge, the IESG
> ultimately allowed both drafts to proceed forward as
> experiments, with very strong disclaimers.

Yes, mistake on their part.

> Now the IESG is faced with an appeal from its decision to
> allow publication of the Lyon draft based on the conflict
> with the Schlitt draft.

I believe at this point the appeal is only to IETF chair and not to IESG.

[An interesting situation BTW - because IESG as a whole made decision to
  proceed with drafts publication where as now only one person from IESG
  has opportunity to overrule them. For government-like structure this
  appears to show where IETF Chair has power of veto like President has
  over decisions of Congress, but President represents separate branch
  of government, where as in countries with parliamentary democracies
  the prime-minister typically can not overrule the parliament]

> What to do? The appellant could withdraw his appeal and
> allow "sleeping dogs to lie where they are."
>
> However, if the appellant persists, which seems likely,
> then given the historical background, the IESG should:
>
> * Withdraw approval for both the Lyon and Schlitt drafts;
> and

That would not be appropriate. The appeal is only against SID draft
(should have been against all 3 SID drafts actually - they can't
be published separately). SPF draft should continue to be viewed
as separate submission and decided on its own merit.

So should (as individual submission on its own merit) be viewed
whatever MS comes up with and certainly there should be no conflicts 
that arise from its when they want to reuse somebody else's record
or fields - it should be done only in a way that effects are limited
to those who agree to participate in their experiment.

> * Call upon the authors of the respective drafts to
> reconcile their differences at which time both drafts can
> proceed to be published as experimental drafts.

You can "call upon the authors" once again but it has been done
before and it has not resulted in reconciliation so far. The
conflict that became clear at the end of MARID is still there
and so is inappropriate use of somebody else's work that MS is
so famous for.

> In turn, if a negotiated settlement can't be reached within
> a suitable time frame, the IESG could decide to call upon
> the authors of the Schlitt and Lyon drafts to re-file their
> documents as informational RFCs only, merely documenting
> matters for historical purposes, which the IESG could then
> approve for publication.

Even for informational purposes publishing conflicting specifications
is not appropriate. There really is not much of a choice here - if 
IETF/IESG agrees there is a conflict, it must request one of the drafts
to be withdrawn and fixed (and if author does not do it, IESG would have 
no choice but to block it from going further). Since its the SID draft 
that depends on SPF draft, the SID would have to be the one to be 
withdrawn (i.e. SID can't be published without SPF draft where as SPF 
draft can be published without SID). In separate post I'll show why its 
correct to withdraw SID draft for other reasons too.

As far as MARID work IESG wanted, the correct thing is start working on 
SPF2/3 where we left off at MARID and create it as new series where all 
scopes are clearly separated. But what IESG really wanted was to 
experiment with these technologies (I doubt they cared as to particulars
of the record syntax) and in fact SPF1 does in  as far as providing
way to see how RFC2821 MAIL FROM verification can be achieved. For MS
the same could be achieved if they instead worked on further on original
Caller-ID (or it could be achieved if they published real MARID in the
way documents went to MARID at last-call in August with only spf2.0 used).

[skip]

> [2] At the time that Mr. Wong was negotiating the
> arrangements with America Online and Microsoft, I was in
> strong opposition.

Most others as well and he did not understand that it was just make
situation that brought end of MARID a lot worse which is exactly
what happened.

> In my view, if pressed, Microsoft would have been prepared
> to amend its IPR and re-write its license to meet the
> concerns raised during MARID, in exchange for access to the
> existing base of spfv1 records through the concept of
> backward compatibility, rather than allow Sender ID to die.

I suspect we both know some of the things that were not made
public and so I'll just note that happen to disagree. I do not
believe Microsoft would have been able to do anything more - it
really was not something that was up to the people who we were
working with, they've done all they could on this matter and in
fact said as much.

> The present situation, given the appeal to the IESG,
> potentially creates the same opportunity, if one believes
> there is merit in saving this particular experiment.

I do not believe it can happen now either and I think trying to
negotiate would hit the same road-block as before and it would not
be anything people seen as standing behind SID at MS (and thus
involved in such negotiations) could do anything about.

But if reuse of spf1 records is really realy the only way for MS
and it wants to continue, then the only possibility for negotiation
I see is to get it part the way for both sides. This would involve:
   1. MS agrees to change its draft and only use positive results of
      SID verification on v=spf1 records (but not fail, softfail or
      results if record is absent) and that for negative results real
      SPF2.0 record would be needed.
   2. SPF community agrees to soften v=spf1 draft and allow for use
      of positive results with other identities (I brought this up before
      and while some thought it was ok, others thought there are is too
      much potential for confusion and incorrect results even then - no
      way can we come to consensus unless its really part of negotiation
      and MS gives something else to make it worth it).
   3. Both sides agree to work further on the next version of protocol
      that would have clear separation of scopes and core protocol
      document that both sides could then reuse [this would be real
      successor to MARID]. Both sides agree that once such new documents
      are ready, they will begin to promote this new version.

---
William Leibzon
Elan Networks
[email-masked]

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "william(at)elan.net" <[email-masked]>
Date: Sat, 27 Aug 2005 22:04:52 -0700 (PDT)

On Sat, 27 Aug 2005, Douglas Otis wrote:

> On Sat, 2005-08-27 at 12:00 -0700, william(at)elan.net wrote:
>
> This overlooks a problem related to abuse-feedback techniques accruing
> to "Sender-ID verified" identities.  An erroneous positive verification
> based upon a PRA, unchecked by the sender perhaps due to licensing
> issues, could be a serious concern.  These SPF records are public and
> outbound servers are often shared.

I did not say its good way for the future, but it does eliminate the cases 
of failures [i.e. email not delivered] due to record being used by incorrect
protocol, which  is a lot worse then what you describe. And what I said 
is that this maybe an acceptable [temporary] compromise (not something that
either side is fully happy with) while waiting for permanent solution, which
is #3 on on my list -.both sides agree to work on the next version of SPF 
that has clear scoping and when finished promote that instead of spf1.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 28 Aug 2005 20:39:55 +0200

John Glube wrote in mxcomp:

> I should have written:

> |The underlying benefit of e-mail authentication for senders
> |interested in e-mail delivery is to establish a framework that
> |supports certification and reputation services, both external
> |and internal to facilitate the delivery of ham, while rejecting
> |spam [1]

That was not the main purpose of SPF.  "SHOULD check HELO" was
added in 2005, because it was always allowed and possible, and
it makes sense for the reasons you have stated.

But the main idea of SPF is to get rid of backscatter, numerous
bogus bounces / challenges / other auto-responses to forged
Return-Paths.  With side effects, some hard, others desirable.

Above all _save_ good bounces because SMTP without any bounces
won't work.  To save the good bounces and other auto responses
it is essential to get rid of the bad backscatter a.s.a.p.

How fast it's adopted by senders or receivers is less important
- that "smart" spammers stay away from FAIL-protected addresses
is the point.

SPF is designed to offer an immediate benefit for participants,
especially spoofed "senders".

> receivers can use their sender policy records for what ever
> purpose they want, irrespective of the stipulated protocols
> is not acceptable.

Of course, receivers could decide to block everybody with(out)
sender policy, count characters in a sender policy, or "check"
a Message-ID, etc.  Their server, their rules.

Nevertheless those interested to avoid backscatter will try to
follow the rules as defined in the SPF RfC.  Or they use a
proper subset, e.g. "without FAIL-chance I won't waste time",
or "let's only use PASS + white lists", or "MAIL FROM stuff is
tricky, forget it, but the HELO tests are nice".

Other checks (PRA, Message-ID, policy length in characters)
are NOT RECOMMENDED, because that's not what the publisher of
a v=spf1 policy intended.

And as William pointed out, it would be also FUBAR to try a
HELO-test with a spf2.0/pra policy.  A PRA-policy is not for
HELO tests.  Receivers are free to try it anyway, but they are
far beyond anything specified in the senderid drafts if they
do this.

And of course this won't work, any "FAIL" they'd get would be
a fantasy-FAIL, not a senderid-FAIL, let alone a SPF-HELO-FAIL.

SID like SPF is a protocol, it works as designed, receivers
are on their own if they break or twist it.  One thing that
is erroneously specified in SID (= _not_ working as designed)
is to check PRA against v=spf1.

That's the point of the appeal.  No amount of weasel words can
change this, it is a serious security bug and has to be fixed.

> it seems SPF folks supporting the appeal are not interested
> in pushing back on Microsoft as it relates to the IPR

Why should we ?  It's none of our business, that's something
SID-implementors would discuss with MS.  I'm confident that
MS can handle all IPR questions about SID without any "help"
from the SPF community.

> simply want to fully sever the relationship between spfv1 and
> spf2.0.

That's not the case, it's okay to share the same DNS RR, it's
okay to use v=spf1 as spf2.0/mfrom, it's okay to use the same
result codes, the same error handling, an almost identical
syntax, and the same processing limits.

So why should they reinvent the wheel, "build on the work of
others".  I've seen statements from Wayne where he offers to
help with spf2.0.  I'm sure that Meng would also help.  I've
offered op=pra as a way to get an "spf2.0/mfrom,pra" effect
in a "v=spf1 op=pra" record for those who explicitly want it:

http://purl.net/xyzzy/home/test/draft-spf-6-3-options-08.txt

So far they didn't react and want any help - no big surprise,
from a technical POV neither SPF nor SID are rocket science.

> In that case, I presume the SPF Community has no problem
> with these potential developments:

I skip this, ex falso quodlibet.

> there is every possibility that those supporting "technical
> purity" within the SPF Community are in for a rude awakening

The "rude awakening" when an SDO like the IETF intentionally
allows conflicting standards ("experiments") only to please MS
would be far worse.
                   Bye, Frank (please fup2 general list)

---

From: wayne <[email-masked]>
Date: Sun, 28 Aug 2005 23:51:00 -0500

> it makes sense for the reasons you have stated.

The "SHOULD check HELO" was added in 2004, not 2005.  This was changed
to "RECOMMENDED" in 2005.

> Return-Paths.  With side effects, some hard, others desirable.

The DKIM proto-WG is currently going through the process of trying to
specify "What is DKIM intended for?"  It's a good idea and one that
the SPF folks skipped in 2003.

So, I disagree that main idea of SPF is to get rid of backscatter and
such.  That is certainly *one* of the reasons that *some* (maybe even
most) people had for promoting SPF, but certainly not the only reason.

Other reasons include the desire by domain owners to not have their
domain name forged in the MAIL FROM and HELO domains, whether they
caused backscatter or not.  Also, by creating identities that could be
verified, you could make more reliable use of whitelists and
blacklists.  Another is that it could be used as a basis to help
protect the 2822.From: header, if you can correctly identify the cases
where the 2821.MAILFROM will differ from the 2822.From:.  I suspect
that there are going to be several other answers from other SPF
supporters.

If your only goal is to get rid of backscatter, things like SES do a
much better job.

-wayne

---

From: Douglas Otis <[email-masked]>
Date: Sat, 27 Aug 2005 17:35:18 -0700

Neither of the path registration schemes provide domain protection at
shared servers.  Even with MTA checks to mitigate domain spoofing at
shared servers, these checks may also be limited by the same licensing
issues.

This retains a need to understand the assurances made by the sender with
respect to the scope of the record, and is unrelated to how a failure is
handled.  Assume the miscreants will know what passes.  

-Doug

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 30 Aug 2005 00:11:51 +0200

wayne wrote:

> The "SHOULD check HELO" was added in 2004, not 2005.
> This was changed to "RECOMMENDED" in 2005.

SHOULD is the same as RECOMMENDED (2119 section 3).  BTW, the
first three sections together need only ten lines including
two blank separator lines, that could be a record.

<http://spf.mehnle.net/Council_Resolution/11> => 2004-12-04

Right, it was 2004-12, not 2005.  I probably confused it with
the time when I read the Council log or minutes.

> The DKIM proto-WG is currently going through the process of
> trying to specify "What is DKIM intended for?"  It's a good
> idea and one that the SPF folks skipped in 2003.

Yes for the former (DKIM) - no idea what you (SPF) did in 2003,
I'm the one RMX-fan who ended up joining SPF because I thought
that it _is_ RMX ;-)

> certainly *one* of the reasons that *some* (maybe even most)
> people had for promoting SPF, but certainly not the only
> reason.

> Other reasons include the desire by domain owners to not have
> their domain name forged in the MAIL FROM and HELO domains,
> whether they caused backscatter or not.

Where it doesn't cause backscatter I'm not very worried (please
no offers to join the "bounces-to" fraction), or in other words
if I'd want S/MIME I'd know where to find it.

> you could make more reliable use of whitelists and blacklists

Yes, that was covered by what John said, and I agreed with it,
"it makes sense for the reasons you have stated".  When you say
"WL or BL" those are IMO special incarnations of "reputation".
Maybe the poor men's edition.

> If your only goal is to get rid of backscatter, things like
> SES do a much better job.

AFAIK for SES you have to control all mail-outs (minimally you
could get away with the MUA) and all corresponding MXs, where
you'd block bogus bounces.  Not realistic for many users.  Bye