Subject: Beggars _can_ be choosers?

From: "David W. Hankins" <[email-masked]>
Date: Wed, 1 Aug 2007 09:03:06 -0700

On Tue, Jul 31, 2007 at 06:59:58PM -0400, John C Klensin wrote:
> Almost independent of the IPv6 autoconfig issues, I find it
> deeply troubling that we seem to be unable to both
> 
> 	* get the ducks lined up to run IPv6 fully and smoothly,
> 	with and without local/auto config.
> 	
> 	* get a DHCP arrangement (IPv4 and, for those who want
> 	to use it, IPv6) that performs reliably, consistently,
> 	and largely invisibly (if I have to worry about what a
> 	DHCP server is doing, it isn't working well).
> 
> and have both of those working seamlessly no later than Sunday
> afternoon of the meeting.
> 
> If we can't do that, we should be very seriously reviewing our
> protocols and specifications: that sort of thing shouldn't be,
> in any sense, an experiment at this stage.

Wow!  Is that an "IETF First" for anyone else?

Ever since my first IETF, I was well aware that many folks held to the
unfortunate fallacy that, "because we do X at IETF meetings and it
works allright, it is therefore sufficient for the rest of the
Internet."  No matter how much people point out the error in this
thinking, it is perpetuated...as recently as the IETF 69 tech plenary,
where we were told that firewalls were becoming obsolete, evidenced by
their lack of use at IETF meetings.

There's only one word for it:  Astounding.

I have never, until now, heard the contrary fallacy attempted.  That
is, "because we did X at an IETF meeting and it did not work
allright, it is therefore insufficient for the Internet."

That's a new one on me.

Clever, but wrong: networks much larger than 1,200 laptops use DHCPv4
on a daily basis all over the Internet without similar symptoms.

-- 
Ash bugud-gul durbatuluk agh burzum-ishi krimpatul.
-- 
David W. Hankins	"If you don't do it right the first time,
Software Engineer		     you'll just have to do it again."
Internet Systems Consortium, Inc.		-- Jack T. Hankins
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Joel Jaeggli <[email-masked]>
Date: Wed, 01 Aug 2007 09:12:14 -0700

David W. Hankins wrote:

> as recently as the IETF 69 tech plenary,

Told by whom? I was one of the people at the the microphone. I work for
a firewall vendor, and I asked a question (probably rhetorical) of the IAB.

> on a daily basis all over the Internet without similar symptoms.

Start by asking the contractor, the volunteers and the IAD for a
postmortem on the operation of the network. anything else is just
speculation.

> 
> 
> ------------------------------------------------------------------------
>

---

From: Ralph Droms <[email-masked]>
Date: Wed, 1 Aug 2007 12:29:16 -0400

I seem to remember that the idea of a postmortem was discussed at  
some point.  I don't know that anything came of that discussion.   
Having some facts and data to examine probably beats anecdotal  
observations about network behavior.

I think David is wise to observe that experience like "DHCP doesn't  
work" doesn't always carry over to "we need to fix DHCP".  Failure of  
one protocol service may just be a symptom of some other failure.   
And failure of a DHCP service request is likely to be an immediately  
and easily observable symptom of some other network failure.

- Ralph

---

From: "David W. Hankins" <[email-masked]>
Date: Wed, 1 Aug 2007 10:34:47 -0700

On Wed, Aug 01, 2007 at 09:12:14AM -0700, Joel Jaeggli wrote:
> Told by whom?

Hain...something like that?  I can't remember.  You'll have to check
whatever minutes or recordings are up.

> postmortem on the operation of the network. anything else is just

Why?

It's a lot of work, and I think we already know the conclusion, given
what we can observe as users.

It berates our volunteers and sponsors who provide us with the network
services during our stay...treating them like children that have to be
kept after school to explain themselves.

It embarrasses vendors who provide us with gear or software, making
them less likely to provide either.

And all we're going to discover is, "It hurts when you go like this,
so don't do that then."

A useless and frivolous excercise.

If we wanted to stop begging, and really buy a network for every
IETF meeting, complete with SLA's and funding extensive testing
before opening it 'live', then that's one thing.  One totally
rediculous thing.

But in lieu of an absence of begging, we should stop being so
choosy.

They did a good job, and we more or less had net the entire meeting.

I don't think we have anything to complain about.

---

From: Joel Jaeggli <[email-masked]>
Date: Wed, 01 Aug 2007 11:11:18 -0700

> kept after school to explain themselves.

I'm one of those volunteers, operating that part of the service was not
my responsibility... My point is to both you and the people complaining
about the network, Drawing conclusions from an incomplete picture is
fraught with peril.

> them less likely to provide either.

There's a difference between trashing someone in public and describing
your issue it doesn't always involve somebody else's hardware Diplomacy
is key.

> A useless and frivolous excercise.

At IETF 55 I had issues with all the terminal room macs coming up with
the same ip address because they all had the same hostid, between that
and some unexpected behavior from our dynamic dns implmentation I gained
some experience on running the dhcp and dns services that has been
folded back into what future hosts did for other meetings.  I don't
think the exercise is frivolous, it's only useless if we don't learn
anything from it or we overdo it and it devolves in navel gazing. Self
criticism either public or otherwise is an essential component of self
improvement.

> before opening it 'live', then that's one thing.  One totally

We have a network contractor who has provided services at the ietf68 and
ietf69. We pay them money, they do stuff. We also have volunteers.

> They did a good job, and we more or less had net the entire meeting.

Indeed. How do we (I'm being rather inclusive here) do better next time?

> I don't think we have anything to complain about.

I ask myself fairly frequently (about 3 times a year) "What can I do to
make the next one better?" Not everyone can or should volunteer to help
but the pool of volunteers is not that big...

---

From: Douglas Otis <[email-masked]>
Date: Wed, 1 Aug 2007 11:16:58 -0700

On Jul 31, 2007, at 6:30 PM, John C Klensin wrote:

> And, while I'm picking on DHCP because I personally had more  
> problems with it, I see IPv6 authconfig as being exactly the same  
> issue: we are telling the world that these things work and they  
> should be using them; if we can't make them work for our own  
> meetings...

Whether one regards IPv6 as "ready for prime-time" depends upon  
location.  IPv6 appears to represent a metric measurement in the only  
industrially developed nation, despite a 1975 act of Congress, still  
is using fahrenheit, ounce, pound, inch, feet, and mile.  There will  
always be problems offering an excuse not to adopt change, even when  
the rest of world has.  Oddly, a 2x4 is neither, but might be  
required to promote change.

-Doug

---

From: John C Klensin <[email-masked]>
Date: Wed, 01 Aug 2007 15:01:40 -0400

--On Wednesday, 01 August, 2007 09:03 -0700 "David W. Hankins"
<[email-masked]> wrote:

> Ever since my first IETF, I was well aware that many folks
> held to the unfortunate fallacy that, "because we do X at IETF
> meetings and it works allright, it is therefore sufficient for
> the rest of the Internet."  No matter how much people point
> out the error in this thinking, it is perpetuated...as
> recently as the IETF 69 tech plenary, where we were told that
> firewalls were becoming obsolete, evidenced by their lack of
> use at IETF meetings.
> 
> I have never, until now, heard the contrary fallacy attempted.
> That is, "because we did X at an IETF meeting and it did not
> work allright, it is therefore insufficient for the Internet."
> 
> That's a new one on me.

David,

I think you misunderstand my comment, or at least its intent.
When we hold a meeting, we make choices --explicitly or
implicitly-- about the protocols we are going to support on the
network for the meeting.  We can reasonably decide that we do
(or do not) want to support IPv6, FTP, or the FooBar protocol.
Reasonable people may disagree with those choices.

However, whatever the choices are, if we cannot have a network
that runs seamlessly at an IETF meeting with the choices we
make, then there is a problem.  That problem may be about
operations, choices of hardware or software, the protocols
involved or how they are specified, or something else.  Whatever
the source of the problem, I believe we owe it to ourselves --
and to the community we are trying to convince to use our
standards-- to get it diagnosed, fixed, and reported on in
sufficient detail that neither we, nor anyone else, need to make
the mistakes again.

If even a small part of the problem turns out to be that we have
developed a protocol that is either sufficiently fragile, or
sufficiently complex and option-laden, or specified in a way
that good-faith vendor implementations don't interoperate with
very high reliability, then we have a problem that I believe we
are obligated to address.  If we have somehow ended up with
protocols that will not run, in combination, in a satisfactory
way over a less-than-perfect network, then we ought to revisit
those protocols to see if they can be made more robust.

We claim to develop protocols for the public Internet, with all
of the strange and sometimes-locally-unpredictable behavior the
public Internet can exhibit.   We often observe that it is much
easier to develop a protocol (or a product) for a
perfectly-predictable LAN environment in which packets never get
lost and always arrive in a timely fashion, in order and without
unexpected fragmentation; on which hostile attacks never occur;
and so on.   It has traditionally been one of our norms that, if
protocols come along that work acceptably only on such LANs, we
either don't touch them in the IETF or carefully and publicly
document their limitations.

To me, this is what having a standards body that focuses on
interoperability and things that actually work in the real, and
nasty, world is all about.  There are certainly bodies,
companies, and consortia out there who develop and aggressive
market protocols and products while using competing ones
internally because they know, at some level, that the products
or protocols they are pushing just don't work well enough.  We
have historically not done that and I hope we don't ever go
there.

This is also just another version of the "eat our own dogfood"
story: if we don't find the dogfood palatable --whether because
of its basic specification or its formulation or packaging in
practice-- then we need to do something about it.

> Clever, but wrong: networks much larger than 1,200 laptops use
> DHCPv4 on a daily basis all over the Internet without similar
> symptoms.

I know that.  I've also got some hypotheses as to why we have
problems and they don't, but my hypotheses aren't backed by
solid data and analysis and hence aren't worth much.  So do you
have an explanation for the repeated IETF problems?  And, if
not, are you willing to join me in suggesting it is about time
the IETF gets to the bottom of these problems, gets the finger
pointed in an appropriate direction, and gets the problem or
problems fixed?

Or do you still think we disagree or that my comments are
fallacious?

     john

---

From: "David W. Hankins" <[email-masked]>
Date: Wed, 1 Aug 2007 13:32:06 -0700

> about the network, Drawing conclusions from an incomplete picture is

I've drawn no final conclusions, and would similarly advise others
not to do so.

I certainly would say that, given what we can observe as users, the
possible explanations for the DNS and DHCP service outages reside in
a _very_ limited set; hardware, software, configuration, or some
combination.

None of those are "IETF business" in the sense of things which we can
observe or change through "Reviewing our protocols and
specifications" as John Klensin appears to have suggested.

> At IETF 55 I had issues with all the terminal room macs coming up with

A volunteer writing up an essay on the goings on, because he knows the
work to be fruitful, is one thing.

A bunch of bearded geeks surrounding a table, pounding their fists,
and demanding the proverbial "answers" is entirely another.

So if you feel like doing it, do it.

But it shouldn't be an imperative.

> ietf69. We pay them money, they do stuff. We also have volunteers.

If the contractor has an SLA to provide DHCP services, that's
something IETF's administrative staff should take up with them.

I suspect they don't.

> Indeed. How do we (I'm being rather inclusive here) do better next time?

I don't think we can.  By its definition, the IETF network is built
with different hardware, different software running on it, and
different operators every time.

This is an entropic nightmare...you're not just changing one variable...
you change all of them.

There's no sane way to expect 100% resilient network operations at
every event put together like that unless you _really_ start spending
money like water on it.

Just give up and stop being so choosy.

> but the pool of volunteers is not that big...

Involve the vendors earlier.  They're probably (not) using the network
that's broken anyway.

Speaking from my own experience, I don't find out that our software is
in use until after things are broken and someone manages to find me
through a network of friends (7 layers of separation seems to be more
like 3).

Take ten minutes to write us a note with your plans beforehand, page
us on ietf@ if you can't find us, and maybe we'll answer back with
the cellphone that will work while we're in town, and ignore us if
we send back a "you need to use fancy new features x, y z to test
them for me".

---

From: Bill Fenner <[email-masked]>
Date: Wed, 1 Aug 2007 16:55:52 -0400

>A good start would be explaining what exactly went wrong with the  
>DHCP server(s) this time. "We have a problem and we're working on it"  
>is not all that helpful.

I wasn't directly involved in debugging this, but this is what I gathered
from later discussions:  The bottom line seemed to be a DHCP server that
was configured to use DNS UPDATE combined with a DNS server that was
configured to refuse DNS UPDATE.  The DHCP server started out working OK,
but apparently had more and more threads working on sending updates to the
DNS server and started to fail to be able to usefully send DHCP responses.
After a restart, it would serve fine for a while and then bog down again.
Each tweak to the configuration would seem to fix the problem since the
associated restart would cause service to be zippy again for a while.

  Bill

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 1 Aug 2007 23:02:57 +0200

> Why?

So we can do better in the future.

> kept after school to explain themselves.

Nonsense. Without a good network we can't do our work properly during  
the meeting (if worst comes to worst we'll have to actually listen to  
the person presenting, <gasp>) and we're all paying enough money to  
attend IETF meetings to expect something professional, which includes  
information about what went wrong.

> them less likely to provide either.

Vendors who both build products that don't work right AND are easily  
embarrassed we don't need.

> They did a good job, and we more or less had net the entire meeting.

I was very happy with the wireless coverage and speed (the fact that  
this meeting I could use 802.11a for the first time probably helped)  
but the DHCP stuff was quite annoying.

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 02 Aug 2007 11:32:46 +0200

We've got an IAD, whose responsibility it is to check that contractors 
(YES, there were contractors, not just volunteers, involved in the 
network setup) do what they contracted to do.

I'm sure he knows enough to ask for help in evaluating the answer, if he 
doesn't understand it.

Asking people to explain what's going on does NOT necessarily mean that 
it's useful to have the question, or the answer, broadcast across the 
whole IETF list.

It's Ray's responsibility to figure out how to handle the "quality of 
network" issue. I'm not going to second-guess him in public.

                            Harald

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 02 Aug 2007 16:08:10 +0200

> <[email-masked]> wrote:

...
> have an explanation for the repeated IETF problems?

Generically, I think there must be two places to look for the
explanations (plural, certainly):

1. The fact that the network is expected to be shaken down within
hours instead of progressively over some large number of days.
It goes from small scale test to full load in about 24 hours.

2. The fact that the client systems are highly heterogeneous
and some of them will probably be running beta code.

That combination of circumstances is going to stretch any dogfood.

> And, if

Personally I have no doubt that it's strongly in Verilan's
enlightened self-interest to get to the bottom of the problems.
It would be interesting to know if the observed failure modes
are caused by bad implementation, by lack of robustness
in the various protocol designs, or both.

      Brian

---

From: Keith Moore <[email-masked]>
Date: Thu, 02 Aug 2007 11:55:12 -0400

> specifications" as John Klensin appears to have suggested.
doesn't follow.  in particular, I don't think we in IETF pay nearly
enough attention to architecture, nor to making our protocols easy to
configure. 

DHCP, in particular, strikes me as a nightmare - a hodgepodge of
unrelated attributes, many of which have no business being dictated to
hosts by the network.  gluing DHCP to DNS creates another set of
problems, also based on dubious assumptions about the relationship
between a host's identity and the attachment point of a host to the
network.  and this all strikes me as a consequence of developing network
configuration protocols "organically", i.e. without much forethought.

now it might be that this specific incident has little to do with the
lack of a configuration architecture in IP.  but I do think that IETF
would do well to analyze such incidents when they do occur and try to
learn from them.  such lessons might impact future protocol designs at
both an architectural and a detailed level. 

more broadly, I wish there were a better feedback path from operators to
IETF to take advantage of the breadth of operational experience.  it's
not as if the incidents occurring in IETF meeting networks are typical;
it's just that we experience them directly and as a group rather than
indirectly or as individuals.

Keith

---

From: "David W. Hankins" <[email-masked]>
Date: Thu, 2 Aug 2007 10:56:30 -0700

> I think you misunderstand my comment, or at least its intent.

I absolutely did, but I think reasonably so.  This version is no
longer crazy, even noble.

But I'm not sold on it.

> Or do you still think we disagree or that my comments are

We're half in agreement.

Where a problem manifests, and an operator intended the configuration
as being reasonable and correct, then the reasons why the setup is
neither, or what failed, should be documented.  I would expect we
should hear very loud noises in any event where this is the case.

Where a problem manifests, and it was an accidental mishap of
misconfiguration?  It's a waste of time to pursue and document these
with the kind of rigorous investigation you suggest.

I suspect this is just a position we will disagree on, but I will
explain myself a little;

First, people who make mistakes on accident are not going to read a
document telling them not to do that beforehand.  If they did read
the document beforehand, well we're talking about _mistakes_ here...
they're going to make mistakes anyway.  Nor will they look to an RFC
after it is done.  They'll seek FAQs, mailing lists, and their
products' support chains.  They'll seek an expert on the subject
because there are simply too many references to read in a timely
fashion to find the one that tells you what mistake you made.

So it is certain that documenting this will make no real difference
to anyone, and I think the "IETF Mindshare" gains are dubious at best.

Second, there are just so many different ways to misconfigure your
network, we would be working at this until the heat death of the
universe.  Of course we'd want this to be a general investigation into
accidental configurations, and not a DHCP Witch-Hunt.  For IETF 69
alone, we'd have to look into why the DNS servers were reliably
reachable but not reliably answering until Tuesday.  To continue on
beyond merely the mishaps of IETF 69, we'd have to provide such
advice as:

"Do not accidentally create ethernet duplex mismatches."

"Do not accidentally load the full Internet BGP table into a 2501.
 Use Filters."

"Do not accidentally urinate on your router while it is running.  In
 fact, avoid accidental urination altogether.  Messy."
 (True story, a housecat in Maui did this to one of our Portmasters)

"Do not make typos, especially not ones which happen to pass parsing
 tests, such as but not limited to reversing digits on subnet masks."

Or my personal favorite:

"Do not accidentally run outdated software with known bugs, possibly
 including well-used security vulnerabilities.  Upgrade!"

This list could go on!

Basically, I think the best thing to do is to just expect our
network's operators to raise the flag themselves if they think it
is useful.  That is, that something might come of it.

This ietf@ business of inserting ourselves into the situation because
of some possibility of a difficult problem that we hope we might be
useful in addressing is another waste of time.

---

From: "David W. Hankins" <[email-masked]>
Date: Thu, 2 Aug 2007 11:06:17 -0700

> configuration protocols "organically", i.e. without much forethought.

We clearly have a very different perception of reality.

> indirectly or as individuals.

Contrarily, we should specifically seek not to make decisions about
events that have affected us personally, because we have biases in our
position towards recommended changes.

That is, it further closes the gap on making IETF protocols designed
explicitly for operation at IETF meetings, and nowhere else.

---

From: "Eastlake III Donald-LDE008" <[email-masked]>
Date: Thu, 2 Aug 2007 14:35:57 -0400

The metric system has been legal in the US since 1895 when the US agreed
to "adopt" it in exchange for France agreeing to Greenwich, England, for
the Prime Meridian.

Donald

-----Original Message-----
From: Douglas Otis [mailto:[email-masked]] 
Sent: Wednesday, August 01, 2007 2:17 PM
To: John C Klensin
Cc: Iljitsch van Beijnum; [email-masked]; [email-masked]
Subject: Re: DHCP failures (was RE: Do you want to have more
meetingsoutside US ?)

-Doug

---

From: Dave Crocker <[email-masked]>
Date: Thu, 02 Aug 2007 12:12:48 -0700

> It goes from small scale test to full load in about 24 hours.

That argues for re-using venues known to work well and, of course, keeping the 
rest of the technical and operations details as stable as possible.  (Yeah, 
some change is required, over time, and any change carries some risk.)

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Dave Crocker <[email-masked]>
Date: Thu, 02 Aug 2007 12:17:20 -0700

Bob Hinden wrote:
> It was also interesting to open the Mac network control pannel, enable 
> my Airport (WLAN) interface, and see the IPv6 global address appear 
> almost instantaneously and in many case having to wait many seconds to 
> minutes for DHCP provided IPv4 address to appear.

Any chance this was merely due to a difference in scaling, with IPv4 DHCP 
usage being large-scale and IPv6 being small?

I suppose the more constructive way to ask this is:  Does anyone know why one 
worked better than the other?

d/
-- 

   Dave Crocker

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 2 Aug 2007 22:57:05 +0200

On 2-aug-2007, at 21:17, Dave Crocker wrote:

>> It was also interesting to open the Mac network control pannel,  
>> enable my Airport (WLAN) interface, and see the IPv6 global  
>> address appear almost instantaneously and in many case having to  
>> wait many seconds to minutes for DHCP provided IPv4 address to  
>> appear.

> Any chance this was merely due to a difference in scaling, with  
> IPv4 DHCP usage being large-scale and IPv6 being small?

> I suppose the more constructive way to ask this is:  Does anyone  
> know why one worked better than the other?

I don't think there was any IPv6 DHCP, and if there was, most hosts  
wouldn't have used it because they don't implement it. The advantage  
of stateless autoconf over DHCP is that with stateless autoconf, a  
singe router advertisement multicast to all IPv6 hosts can provide an  
unlimited number of hosts with address information (the hosts still  
need to do duplicate address detection, but since no reply means  
success it's hard to fail here) so it's eminently more scalable than  
DHCP.

---

From: Sam Hartman <[email-masked]>
Date: Thu, 02 Aug 2007 18:02:12 -0400

>>>>> "Iljitsch" == Iljitsch van Beijnum <[email-masked]> writes:

    Iljitsch> On 2-aug-2007, at 21:17, Dave Crocker wrote:
    >>> It was also interesting to open the Mac network control
    >>> pannel, enable my Airport (WLAN) interface, and see the IPv6
    >>> global address appear almost instantaneously and in many case
    >>> having to wait many seconds to minutes for DHCP provided IPv4
    >>> address to appear.

    >> know why one worked better than the other?

    Iljitsch> I don't think there was any IPv6 DHCP, and if there was,
    Iljitsch> most hosts wouldn't have used it because they don't
    Iljitsch> implement it. The advantage of stateless autoconf over
    Iljitsch> DHCP is that with stateless autoconf, a singe router
    Iljitsch> advertisement multicast to all IPv6 hosts can provide an
    Iljitsch> unlimited number of hosts with address information (the
    Iljitsch> hosts still need to do duplicate address detection, but
    Iljitsch> since no reply means success it's hard to fail here) so
    Iljitsch> it's eminently more scalable than DHCP.

Let's be clear here.  The scaling properties of stateless autoconf are
better than DHCP in cases where I want to give a uniform configuration
to all nodes on the link and where all the configuration I want to
hand out is supported by stateless autoconf.

Issues such as giving hosts hostnames, dynamic dns updates, etc can
change to the scalining properties of the entire IPV6 configuration
experience to be different than the base scaling properties of
stateless autoconf.

--Sam