Subject: Re: Deployment Cases

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Wed, 26 Dec 2007 20:28:37 -0800

I don't see the point you are trying to make here.

If we need some expertise and don't feel we have it in the ietf we go find someone who does. We have the ability to tap into any of the top universities economics departments and this stuff is surely understood somewhere in the fortune 500 companies here.

We are in the business of communication, of collaboration. If we can't do either shame on us.

What is worse, to think we are expert economists or that because we are not economists the whole field must be irrelevant? I don't think we need to go too deep here, just thinking in terms of deployment as the challenge is a start.

As for defining marketting as bending of facts. There are two processes that can be thought of as marketting. Shifting product is really a form of sales. I am talking about the processes you go through to determine whether your product meets a genuine market need. Fudging such a study is self deception and only leads to tears.

I always thought that the point of engineering was that you define yourself by the ends you are seeking to achieve, not a particular skill set. If you need a skill you seek to acquire it. 

Sent from my GoodLink Wireless Handheld (www.good.com)

 -----Original Message-----
From: 	Brian E Carpenter [mailto:[email-masked]]
Sent:	Wednesday, December 26, 2007 12:32 PM Pacific Standard Time
To:	Hallam-Baker, Phillip
Cc:	[email-masked]
Subject:	Re: Deployment Cases

Phill,

On 2007-12-24 07:32, Hallam-Baker, Phillip wrote:
> Problem: We have a lot of specifications that work fine, but have not seen deployment
>  
> Examples: Most security specifications, IPv6, etc. etc.
>  
> Cause: The economic case for deployment is not made. This is a particular concern when the protocol deplends on the network effect. Until the network is established the protocol has minimal value.
>  
> Proposed Solution: Deployment Cases
>  
> As engineers we are now used to considering use cases, that is representative use scenarios that illustrate particular intendeded uses. The output of use cases are requirements.
>  
> Deployment cases are similar except that we consider the economic incentives for transition. 

IMHO, whenever the IETF has tried to do anything like this, we've
failed. A good (er, bad) example was draft-ietf-iab-case-for-ipv6,
which had so many issues that it was allowed to curl up and die.

Basically, we aren't business-oriented people or economists,
and we don't tolerate marketing-style bending of the facts.

I'm pessimistic about trying this in the IETF, even for
technologies that are aimed at the greater good, where the
economic arguments are not about short term returns for
individual companies.

     Brian
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Spencer Dawkins <[email-masked]>
Date: Thu, 27 Dec 2007 08:13:21 -0600

Re: Deployment CasesPhil,

I think I kinda do see what Brian's point is. I don't think it should be a conversation-ender, but Brian is pointing out an issue that we need to work through...

As an organization of individuals developing protocol specifications - that's who we are, and that's what we do - we don't even have a natural way to interact with operators, except to invite them to participate as individuals. That has not worked particularly well for a long time.

(as an aside: We need one, and from time to time we have made intentional efforts to interact with operators on a specific topic (most recently, Dave Meyer and friends doing the NOG tour to talk about, and listen to concerns about, SHIM6 and IPv6 multihoming), but the point is that interaction with operators doesn't happen as business-as-usual.)

I think Brian is saying the same thing about economic/financial analysis - I agree with your statement ("we need to go find someone who does" have expertise in this area), but the devil is in at least a couple of details:

o who do we talk to, and 

o what does that conversation look like?

You suggested two sources of input, university economics departments and F500 economics departments. It's worth noting that who we ask will shape what we hear back - visualize this type of discussion for peer-to-peer SIP. Do we ask Columbia.edu? ISPs? PTTs? Vonage? Skype? military? public safety? None of these are a priori WRONG...

But I don't see us getting one answer back, reading it, and accepting it without asking questions. What's the forum for "rough consensus" about deployment incentives?

Who actually asks for input? When? Do we hope the people we ask will attend IETF meetings so we can discuss with them? attend IESG telechats specifically about new work? or something else?

Who analyses what we hear back? This would almost certainly involve changes to the IETF leadership and/or structure, because the position descriptions for IAB and IESG don't say anything about expertise in this area. Again, this isn't beyond imagination, we just can't ignore it.

While I would not suggest adoption of IEEE processes without thought, it's worth being aware that IEEE 802 uses "five criteria" in evaluating new work (sample for IEEE 802.21 at http://www.ieee802.org/21/802_21_5Criteria.doc), and some of the criteria, "Broad market potential" and "economic feasibility", seem to touch on what we're talking aout here.

Is this, broadly speaking, what you are thinking about, for IETF?

Thanks,

Spencer

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 28 Dec 2007 08:27:34 +1300

I don't want to repeat myself unduly, but I believe that
the IETF is institutionally incapable of taking this type
of approach, for exactly the same reasons that's it's quite
good at doing protocol design. I think that the organisations
that do emphasise business cases and deployment have a record
of being (a) narrowly focussed on a single technology and
(b) not doing particularly good protocol design. (Pick your
own example...) It isn't simply a matter of skill sets - it's
something much deeper in the nature of the organisation.

That being said, I think it would be excellent for one of
our criteria for chartering new work to be deployability,
including economic deployability. That isn't explicit in
section 2.1. of RFC 2418, unfortunately.

Given the way things work in the IETF, I think a document
discussing what design features make a protocol economically
deployable would be very valuable. There are some hints
of this in RFC 3439, but not much.

     Brian

On 2007-12-28 03:13, Spencer Dawkins wrote:
> ------------------------------------------------------------------------
>

---

From: Dave Crocker <[email-masked]>
Date: Thu, 27 Dec 2007 12:51:10 -0800

Spencer Dawkins wrote:
  > I think I kinda do see what Brian's point is.
...
> As an organization of individuals developing protocol specifications - 
> that's who we are, and that's what we do - we don't even have a natural 
> way to interact with operators,
...
> I think Brian is saying the same thing about economic/financial analysis 

We do not need to conduct intricate analysis, nevermind economic analysis.

However we do need to have a basis for believing that the work we are doing 
will actually get used.  Anyone investing years of effort ought to have a 
basis for believing not only that a document will get produced but that it 
will result in implementations and that the implementations will get deployed 
and that the deployments will get used.

Otherwise, we are spending aggregate millions of dollars per standard as an 
academic exercise.

At base, this falls into the category of "market research".  The fact that we 
are engineers does not mean that we get pretend market issues are irrelevant.

Consider job hunting.  Most people care about the nature of the work but they 
also care about whether they will get paid.  They assess a variety of factors, 
to decide that a reliable paycheck will be forthcoming.  That process of 
assessment is a kind of business analysis.  Even engineers do it.

This used to be well understood in the IETF.  And as we turn out process 
document after process document, we ought to consider how little we pay 
attention to ones we have had around for a long time.

Consider IETF Working Group Guidelines and Procedures, 
<http://ietf.org/rfc/rfc2418.txt>:

> 2.1. Criteria for formation
> 
>    When determining whether it is appropriate to create a working group,
>    the Area Director(s) and the IESG will consider several issues:
...
>     - What are the risks and urgency of the work, to determine the level
>       of effort required?
...
>     - Does a base of interested consumers (end-users) appear to exist
>       for the planned work?  Consumer interest can be measured by
>       participation of end-users within the IETF process, as well as by
>       less direct means.

The rest of the list is pretty interesting, I think, but these two bullets 
pertain to the current thread:  A basis for assessing urgency and community 
pull, to use the result.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Christian Huitema <[email-masked]>
Date: Thu, 27 Dec 2007 16:35:58 -0800

> However we do need to have a basis for believing that the work we are
> doing will actually get used.

We went through that many times. The best way we have found so far is to verify that the proposed working group has a sufficient constituency. This has the advantage of not requiring economic or business judgments from the IESG. After all, one should assume that the participants who are expressing interest did their homework, business plans and the like.

-- Christian Huitema

---

From: Dave Crocker <[email-masked]>
Date: Thu, 27 Dec 2007 16:42:09 -0800

Christian Huitema wrote:
> We went through that many times. The best way we have found so far is to
> verify that the proposed working group has a sufficient constituency. This
> has the advantage of not requiring economic or business judgments from the
> IESG. 

Right.

We don't have a problem with regularly producing specifications that do not
get used.

So there's no need to look for improvement in this arena.

d/

ps. There is a difference between "economic or business judgements" and simply 
trying to assess whether there is a serious desire by an Internet community to 
use the work.  The difference is fundamental, so it's best not to confuse them.

-- 

   Dave Crocker

---

From: Franck Martin <[email-masked]>
Date: Fri, 28 Dec 2007 18:41:15 +1200

The problem is not to produce specifications, but to get them used.

The "What makes a protocol successful" presentation, shows that the
best protocols are the ones given to IETF for it to refine and
complete. They have already a user pull when they reach IETF.

There is a weak link between IETF and NOGs.

IETF is full of "engineer politics"

There is no feedback loop on the specifications, including oversight
on what the IETF is doing. Who knows what the IETF is? What does it
do? etc... The specialized press does not follow the IETF work to give
hints on what is likely to get adopted or not.

The IETF journal is a good step in this direction. The IETF
fellowships brings new blood. More steps need to be taken.

You may need an ISAG (Internet Strategic Advisory Group), made of non
engineers with one IETF liaison, who would provide vision, direction,
analysis as well as reach out to communities. It could be composed of
one rep from each NOG, one rep from the ICANN community, one rep from
the business community, one rep from the media/press community, one
rep from the user community.

Cheers

Dave Crocker wrote:
>
>
> Christian Huitema wrote:
>>> However we do need to have a basis for believing that the work we
>>> are doing will actually get used.
>>
> ps. There is a difference between "economic or business judgements"
> and simply trying to assess whether there is a serious desire by an
> Internet community to use the work.  The difference is fundamental,
> so it's best not to confuse them.
>

- --
- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Franck Martin
[email-masked]
"Toute connaissance est une réponse à une question"
G. Bachelard

---

From: "TS Glassey" <[email-masked]>
Date: Sun, 30 Dec 2007 09:20:40 -0800

Christian in respinse to your answer below,

0)    Why does the IETF care whether its IP is used?

It operates a standard's process not a commercial eneity who's existence is 
based on licensing, so if the IETF has this concern that impacts its 
impartial status as an open and fair entity I think.

1)    what process is used to make that decision and what constitutes 
'sufficient' in this term?

2)    what Business Justification's would be needed for the IESG to do 
anything???

Can you explain to me specifically what "the IESG's Business Justification" 
requirements are for and what they constrain..

Todd Glassey

----- Original Message ----- 
From: "Christian Huitema" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, December 27, 2007 4:35 PM
Subject: RE: Deployment Cases

We went through that many times. The best way we have found so far is to 
verify that the proposed working group has a sufficient constituency.

This has the advantage of not requiring economic or business judgments from 
the IESG. After all, one should assume that the participants who are 
expressing interest did their homework, business plans and the like.

-- Christian Huitema

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 31 Dec 2007 13:01:41 +0100

> complete. They have already a user pull when they reach IETF.

I don't think that's valid statistics: obviously many of the protocols  
in question were already successful before they were given to the  
IETF, which isn't necessarily the case for protocols developed "in- 
house".

---

From: Dave Crocker <[email-masked]>
Date: Mon, 31 Dec 2007 12:09:57 -0800

Iljitsch van Beijnum wrote:
> in question were already successful before they were given to the IETF, 
> which isn't necessarily the case for protocols developed "in-house".

That's the point:  protocols created in the IETF, over the last 10 years, do 
not have a very good record of deployment and use.  Protocols created outside 
the IETF and then brought into the IETF have a better track record.

This suggests that the IETF has become a place to refine existing (proven) 
work and to expand its community of adopters, rather than to formulate initial 
work and gain initial adoption.

On reflection, this should not be viewed as heretical or even problematic.

When a group grows to have the kind of massively diverse participation that 
the IETF now gets, it is rarely going to be possible to develop or maintain 
the kind of coherent focus needed to get solid, cohesive specifications 
written: Things move from getting "different perspectives" to getting 
"constant objections".

A successful protocol needs a tight design focus and a a clear market need. 
Our current base of participation makes it difficult to formulate either.

d/

-- 

   Dave Crocker

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 31 Dec 2007 15:22:44 -0500

I think this whole discussion would benefit from some concrete examples.

What wholly new protocols has the IETF developed in the past decade?  
Which ones would you consider successful or not?

Almost by necessity, newer protocols tend to cover niches, relatively  
speaking, as opposed to broad swaths of technical territory  
("asynchronous message-based communication", "object retrieval", "name  
mapping").

Henning

On Dec 31, 2007, at 3:09 PM, Dave Crocker wrote:

>> I don't think that's valid statistics: obviously many of the  
>> protocols in question were already successful before they were  
>> given to the IETF, which isn't necessarily the case for protocols  
>> developed "in-house".
>
>
> That's the point:  protocols created in the IETF, over the last 10  
> years, do not have a very good record of deployment and use.   
> Protocols created outside the IETF and then brought into the IETF  
> have a better track record.
>
> This suggests that the IETF has become a place to refine existing  
> (proven) work and to expand its community of adopters, rather than  
> to formulate initial work and gain initial adoption.
>
> When a group grows to have the kind of massively diverse  
> participation that the IETF now gets, it is rarely going to be  
> possible to develop or maintain the kind of coherent focus needed to  
> get solid, cohesive specifications written: Things move from getting  
> "different perspectives" to getting "constant objections".
>
> A successful protocol needs a tight design focus and a a clear  
> market need. Our current base of participation makes it difficult to  
> formulate either.
>
> d/
>
> -- 
>
>  Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Mon, 31 Dec 2007 12:47:04 -0800

> Which ones would you consider successful or not?

Well, that's such a reasonable question, I did a subjective review of Proposed 
Standard RFCs for the last number of years -- ignoring that most recent and 
going back to rougly RFC 2500 -- looking for acronyms that were for 
significant IETF-generated efforts.

Note that I said subjective.  I meant it two ways.  One is that I merely did a 
scan, rather than anything more diligent.  The second is that the list 
reflects my personal reactions as to 'significance' of the effort:

IPv6
DNSSec
Enum
XMPP
CPIM
BEEP
MSRP
SCTP
IPSec
SASL
MPLS
CalDav
RTP
NETCONF
IMAP4
PPP
(a variety of PKI efforts?)
LDAP
OPES
VPIM
FFPIM
GEOPRIV
Diameter
iSCSI
DDDS
SIEVE

A number of questions come to mind:

1.  What additions or removals should be made to the list?

2.  Has the protocol (or set of protocols that pertain to the acronym) gained 
widespread adoption and if so, what does "widespread" mean and what does 
"adoption" mean?  Having 100M+ daily users is definitive, but what about 
smaller numbers that nonetheless dominate an important niche, such as 
"LAN-email office workers"?

3. ...?

d/

>>> protocols in question were already successful before they were given 
>>> to the IETF, which isn't necessarily the case for protocols developed 
>>> "in-house".
>>
>>
>> Protocols created outside the IETF and then brought into the IETF have 
>> a better track record.
>>
>> (proven) work and to expand its community of adopters, rather than to 
>> formulate initial work and gain initial adoption.
>>
>> When a group grows to have the kind of massively diverse participation 
>> that the IETF now gets, it is rarely going to be possible to develop 
>> or maintain the kind of coherent focus needed to get solid, cohesive 
>> specifications written: Things move from getting "different 
>> perspectives" to getting "constant objections".
>>
>> A successful protocol needs a tight design focus and a a clear market 
>> need. Our current base of participation makes it difficult to 
>> formulate either.
>>
>> d/
>>
>> -- 
>>
>>  Dave Crocker

-- 

   Dave Crocker

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 31 Dec 2007 16:28:43 -0500

Thanks for the list; the cut-off point is probably somewhat  
subjective, but I see at least several protocols on the list that one  
can consider reasonably successful, as in having several well-known  
implementations, shipping as part of common desktop or server  
operating systems, references by other standards-bodies and/or large- 
scale operational deployments:

Enum
XMPP
SCTP
IPSec
MPLS
CalDav
RTP
IMAP4
PPP
LDAP
GEOPRIV (right now, mostly through the LLDP-MED effort, rather than  
directly)
Diameter (3G)
iSCSI
SIEVE

I'm probably missing some due to unfamiliarity. This doesn't seem like  
such a bad batting average.

MSRP isn't really finished yet, so that's probably a bit premature to  
judge.

On Dec 31, 2007, at 3:47 PM, Dave Crocker wrote:

> 2.  Has the protocol (or set of protocols that pertain to the  
> acronym) gained widespread adoption and if so, what does  
> "widespread" mean and what does "adoption" mean?  Having 100M+ daily  
> users is definitive, but what about smaller numbers that nonetheless  
> dominate an important niche, such as "LAN-email office workers"?
>
> 3. ...?
>
> d/
>

---

From: Franck Martin <[email-masked]>
Date: Tue, 01 Jan 2008 10:45:18 +1200

I thought XMPP came from outside (www.jabber.org)?

Henning Schulzrinne wrote:
> subjective, but I see at least several protocols on the list that
> one can consider reasonably successful, as in having several
> well-known implementations, shipping as part of common desktop or
> server operating systems, references by other standards-bodies
> and/or large-scale operational deployments:
>
> Enum
> XMPP
> SCTP
> IPSec
> MPLS
> CalDav
> RTP
> IMAP4
> PPP
> LDAP

- --

---

From: Ned Freed <[email-masked]>
Date: Mon, 31 Dec 2007 16:31:45 -0800 (PST)

> I thought XMPP came from outside (www.jabber.org)?

I believe that's correct.

Sieve is another interesting case. It was originally developed "at" the IETF
but not "by" the IETF, in that there were various informal meetings where it
was designed but the initial documents were all individual submissionss. It was
only much later that a working group was put in place.

Sieve is also interesting in that have been quite a few active participants,
including document authors, who rarely if ever attend meetings (and I believe
some never have). The authorship of the various specifcations is also quite
diverse along several axes (e.g., geography, past IETF participation, corporate
backing). All in all, it's been a fun group to be involved in.

I'm not drawing any conclusions here, just pointing out potentially interesting
diversity in how things have been done.

				Ned