Subject: [OPSAWG] AD Review: draft-ietf-opsawg-automated-network-configuration

From: Ronald Bonica <[email-masked]>
Date: Mon, 11 Jun 2012 22:39:12 -0400

Authors,

I'm sorry that you had to wait so long to get this document through WG Last Call. In order to compensate, I started my AD review a bit early. The following are a few questions:

1) What is the scope of your document? Does it apply to routers in an ISP core? ISP edge? Enterprise edge? 

2) Many networks automate configuration in ways diverge significantly from the model presented in your draft. Is there a reason why your model is superior to the models used in those networks?

3) What is the purpose of this document? Please comment on the following possible motivations:
- to provide guidance to those who have not already experienced success in the area of automated configuration
- to convert networks that have already experienced success to the model that you describe
- to motivate the work described in Section 9
- other

4) In Section 9, do we really need three different ways to locate the configuration server? Couldn't we avoid all of this work by pre-configuring everything that the device needs to identify and contact the configuration server? (Many networks do this today.)

5) Is G6 really a gap? Are you suggesting that we build security mechanisms into every network management protocol that lacks them? Or should we just avoid those protocols?

6) Who is the intended audience for this draft? What behavior change do you expect from the intended audience? 

--------------------------
Ron Bonica
vcard:       www.bonica.org/ron/ronbonica.vcf

---

From: Randy Bush <[email-masked]>
Date: Tue, 12 Jun 2012 16:20:48 +0900

> 2) Many networks automate configuration in ways diverge significantly
>    from the model presented in your draft. Is there a reason why your
>    model is superior to the models used in those networks?

has the model in the draft actually been implemented and used in a real
network?

randy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 12 Jun 2012 10:37:07 +0200

> 2) Many networks automate configuration in ways diverge significantly from the model presented in your draft. Is there a reason why your model is superior to the models used in those networks?

Can you provide details how they diverge significantly? This would
help a lot.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ronald Bonica <[email-masked]>
Date: Wed, 13 Jun 2012 21:33:00 -0400

Juergen,

Most of the networks with which I am familiar have a unique, homespun, configuration model. These models differ from yours, and each other, in many ways.

One significant difference is that in many networks, the network element does not need to discover the configuration server. In some networks, the configuration server address and associated authentication information is preconfigured. In other networks, the configuration server calls the network element.

                                             Ron

> -----Original Message-----
> From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-
> university.de]
> Sent: Tuesday, June 12, 2012 4:37 AM
> To: Ronald Bonica
> Cc: [email-masked];
> [email-masked]
> Subject: Re: [OPSAWG] AD Review: draft-ietf-opsawg-automated-network-
> configuration
> 
> Can you provide details how they diverge significantly? This would help
> a lot.
> 
> /js
> 
> --

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 14 Jun 2012 09:53:35 +0200

Hi Ron,

This depends I guess, also, upon the type of the network elements and
the deployment environment. For example in enterprise VoIP deployments,
one of the most common models is that IP phones get the configuration
server information by DHCP - this is the second option discussed in the
draft.  

Regards,

Dan

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Ronald Bonica
> Sent: Thursday, June 14, 2012 4:33 AM
> To: Juergen Schoenwaelder
> Most of the networks with which I am familiar have a unique, homespun,
> configuration model. These models differ from yours, and each other,
in
> many ways.
> 
> One significant difference is that in many networks, the network
> element does not need to discover the configuration server. In some
> networks, the configuration server address and associated
> authentication information is preconfigured. In other networks, the
> configuration server calls the network element.
> 
>                                              Ron
> 
> 
> > Cc:
[email-masked];
> help
> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg

---

From: Randy Bush <[email-masked]>
Date: Thu, 14 Jun 2012 18:00:49 +0900

> the deployment environment. For example in enterprise VoIP
> deployments, one of the most common models is that IP phones get the
> configuration server information by DHCP - this is the second option
> discussed in the draft.

and backbone networks don't.  and ron's point was mileage varies
greatly.

i ask again, is there operational experience with this draft?  and i
will be less subtle.  i know the internals of a few large scale (though
not wireless) service provider automation systems, and i do not
recognize them here.

randy

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 14 Jun 2012 12:14:25 +0200

> From: Randy Bush [mailto:[email-masked]]
> Sent: Thursday, June 14, 2012 12:01 PM
> To: Romascanu, Dan (Dan)
> Cc: opsawg

That was my point as well. Large IP networks does not mean that all
devices are connected directly to the backbone. Configuring the
addresses of configuration servers to be accessed by devices does not
mean that they need (can, at reasonable costs) be individually
configured in tens of thousands of new network devices. 

> not wireless) service provider automation systems, and i do not

I am not a co-author of this draft, but I recognize the model of using
DHCP to find the configuration server and establishing a channel for
secure configuration, as being used in some large scale deployments that
I know. These are not service provider deployments, but large enterprise
networks, as far as I understand the I-D also covers this use case. 

Regards,

Dan

---

From: Tom Taylor <[email-masked]>
Date: Thu, 14 Jun 2012 09:27:59 -0400

I wrote the original draft out of my own head. The intention was to 
identify gaps that it might be worth following up on, but it was 
basically an analytical exercise done with input from my co-authors. My 
practical knowledge of operations dates back a bit, to my days as a 
circuit switching OAM planner at BNR and Nortel.

The general structure of the document is still as I put it together. I 
readily admit that Juergen brings far more knowledge of the topic to the 
table than I do, and the document has benefited from his subsequent 
stewardship.

On 14/06/2012 5:00 AM, Randy Bush wrote:
> .

---

From: Ronald Bonica <[email-masked]>
Date: Thu, 14 Jun 2012 10:36:25 -0400

Exactly! This is why my first question was about the scope of the document? Does it apply to ISP Cores? ISP Edge routers? CPE?

                                        Ron

> From: Romascanu, Dan (Dan) [mailto:[email-masked]]
> Sent: Thursday, June 14, 2012 3:54 AM
> To: Ronald Bonica; Juergen Schoenwaelder
> Subject: RE: [OPSAWG] AD Review: draft-ietf-opsawg-automated-network-
> configuration
> 
> Hi Ron,
>

---

From: Ronald Bonica <[email-masked]>
Date: Mon, 18 Jun 2012 12:21:27 -0400

Tom,

Let me ask the question another way. Having read your draft, how would you expect the community to behave differently? Would you expect them to develop something that they aren't already developing? Would you expect them to do something that they aren't already doing.

The answer to this question defines the utility of the draft.

                              Ron

> Behalf Of Tom Taylor
> Sent: Thursday, June 14, 2012 9:28 AM
> To: Randy Bush
> Cc: opsawg
> readily admit that Juergen brings far more knowledge of the topic to
> the table than I do, and the document has benefited from his subsequent
> stewardship.
> 
> > will be less subtle.  i know the internals of a few large scale
> > (though not wireless) service provider automation systems, and i do
> > not recognize them here.
> >
> > randy

---

From: Tom Taylor <[email-masked]>
Date: Mon, 18 Jun 2012 19:07:32 -0400

I certainly never contemplated getting operators to do things 
differently -- just hoped to make sure they had the tools they needed if 
they chose to use them

Tom

---

From: Randy Bush <[email-masked]>
Date: Tue, 19 Jun 2012 08:12:03 +0900

hi tom

> differently -- just hoped to make sure they had the tools they needed
> if they chose to use them

the downside with this well-intentioned inventive approach is code bloat
and bugs.  otoh, bright ideas that ops really will use are worth it.
it's a balance and trade-off.

randy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 21 Jun 2012 13:35:59 +0200

> I'm sorry that you had to wait so long to get this document through WG Last Call. In order to compensate, I started my AD review a bit early. The following are a few questions:

Here is _my_ answer to your questions.

1) What is the scope of your document? Does it apply to routers
   in an ISP core?  ISP edge? Enterprise edge?

   I think the document is not specific to certain types of
   equipment.  That said, it likely has wider applicability to
   devices that are deployed at the network edges rather than the
   core and perhaps likely more to cellular network and
   enterprise network gear. At least some of the examples
   discussed on the list were coming from those environments.

2) Many networks automate configuration in ways diverge
   significantly from the model presented in your draft. Is there
   a reason why your model is superior to the models used in
   those networks?

   The scope of this questions remains unclear to me. Part of
   that is that people only say "significantly different",
   detailing things with homebrewed magic. Another part is that
   the document does not detail a specific model either, so I am
   not even sure what the question targets. If model means the
   five phases introduced in section 3 and further detailed in
   subsequent sections, then I believe these phases are indeed
   happening in the real world (perhaps sometimes a phase is
   skipped). This model was introduced in an attempt to untangle
   things into something that allows a more organized discussion.

3) What is the purpose of this document? Please comment on the following
   possible motivations:

   - to provide guidance to those who have not already
     experienced success in the area of automated configuration

   Yes. I think this document is relevant for (a) network
   operators (in the generic sense) who decide to roll out lots
   of new devices and who think about how to implement things
   and (b) network equipment vendors who like to add features to
   their products that make the roll out of lots of new devices
   simpler.

   - to convert networks that have already experienced success to
     the model that you describe

   I guess this is of little relevance - if you have solved this
   problem and you are happy, you likely don't care or even do
   not want to talk how you do it (as we have seen on the list).

   I would say identify the work. Whether these things should be
   acted on is a different question that depends on available
   resources and the chances to be successful in the market place.
   But I think it is useful to be aware what uncovered spots are.

   - other

   General information to people interested in such questions for
   whatever reason.

4) In Section 9, do we really need three different ways to locate
   the configuration server? Couldn't we avoid all of this work
   by pre-configuring everything that the device needs to
   identify and contact the configuration server? (Many networks
   do this today.)

   Apparently, some devices use DHCP today for this purpose. DNS
   SRV records have seen some usage for service discovery, in
   particular in combination with mDNS. SLP is mentioned more for
   completeness since it is an IETF standard designed for service
   discovery.  Again, the list _identifies_ gaps, it does not
   intend to say they all should be addressed.

5) Is G6 really a gap? Are you suggesting that we build security
   mechanisms into every network management protocol that lacks
   them? Or should we just avoid those protocols?

   There once was a whole BOF on this topic. I do not want to
   rehash this here but as far as I can tell this is a problem
   where some vendors have work arounds that however never made
   any progress in the IETF, see [I-D.kwatsen-reverse-ssh] for
   example. The lack of any call-home standard affects
   unfortunatly quite a few of the IETF's management protocols.
   Anyway, once again, the list _identifies_ gaps, whether the
   IETF will act on them is another discussion.

6) Who is the intended audience for this draft? What behavior
   change do you expect from the intended audience?

   I guess this already partially answered in question 3). I
   think this document is primarily relevant for (a) network
   operators (in the generic sense) who decide to roll out lots
   of new devices and who think about how to implement things
   and (b) network equipment vendors who like to add features to
   their products that make the roll out of lots of new devices
   simpler. The document tries to give guidance and informs about
   trade-offs between different approaches.

   I am not sure about behavior change. As said before, if a
   network operator has a working solution, he likely won't
   change. If a network operator has a solution and is unhappy
   with it, he might try alternatives. But I think that prime
   audience is network operators and equipment vendors that do
   not have a lot of experience with the roll out of large
   numbers of devices.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 21 Jun 2012 13:38:20 +0200

> One significant difference is that in many networks, the network element does not need to discover the configuration server. In some networks, the configuration server address and associated authen…

Please note that pre-configuration is actually discussed as one of the
mechanisms in the document.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 21 Jun 2012 13:51:41 +0200

> not wireless) service provider automation systems, and i do not

Yes. The I-D discusses several options and some of them surely have
been widely used, others I am less aware of their usage (but as you
know, it is generally hard to establish usage of something). Some
examples might help you:

a) TR69 is as far as I know widely used by CPE to obtain the address
   of the so called Auto-Configuration Server. This is mentioned on
   page 3.

b) There was text added concerning pre-configuration provided by
   pluggable memory cards or nearfield communication due to a
   contribution to the mailing list that devices of a major vendor do
   this (section 4).

c) DHCP seems widely used for IP address acquisition (section 5.2).

d) Christopher Liljenstolpe suggested in one of his reviews to add
   text about anycasts. Before adding text, I specifically asked on
   the mailing list whether this is done and there were contributions
   saying that this is the case (section 5.3).

e) It seems UUIDs have proven useful to identify devices (section
   5.4).

f) I think there is evidence that several of the protocols mentioned
   in section 6 have been used to move configurations around.

I could probably go on for a while. Perhaps I do not understand of
"operational experience with this draft" means. This is an
informational document, outlining options available, discussing
trade-offs, identifying gaps.

/js

---

From: Ronald Bonica <[email-masked]>
Date: Thu, 21 Jun 2012 09:41:18 -0400

Juergen,

                                      Ron

> Sent: Thursday, June 21, 2012 7:36 AM
> To: Ronald Bonica
> > I'm sorry that you had to wait so long to get this document through
> WG Last Call. In order to compensate, I started my AD review a bit
> early. The following are a few questions:
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 21 Jun 2012 16:49:40 +0200

> The answer to this question defines the utility of the draft.

What is "the community"? I have at least three options:

a) community = IETF: I do not know whether the IETF wants to act on
   any of the identified gaps. I might have a personal opinion but
   that is not important I guess. The goal was to identify the gaps -
   are you saying that only makes sense if there is a defined action
   following? I do not think this is necessarily coupled - it can be
   fine to know there are certain gaps and to have reasons to do
   nothing about them.

b) community = vendor: A vendor planning to support large rollouts in
   their products and who reads this document hopefully have a more
   complete picture of the options out there. Hard to tell to what
   extend that changes their decisions.

c) community = operator: An operator planning to do large rollouts who
   has read the I-D hopefully has a more complete picture of the
   options out there. Hard to tell to what extend that changes their
   decisions.

Probably the utility is simply 42 (in the sense that I do not really
know what the AD is asking or how he likes to measure the utility of
such informational documents).

/js

---

From: Ronald Bonica <[email-masked]>
Date: Thu, 21 Jun 2012 14:13:08 -0400

Juergen,

When I read your draft, I assumed that you were addressing the operator community? Was that not the case? As a result, when I referred to "the community", I had the operator community in mind.

So, I will ask my question yet another way. Assume that the operator of a large network reads your draft. Although this operator has years of experience with large-scale rollouts, he doesn't see his own procedures in your document. Should he consider changing his procedures? If so, why?

                                               Ron

> Sent: Thursday, June 21, 2012 10:50 AM
> To: Ronald Bonica
> > Let me ask the question another way. Having read your draft, how
> would you expect the community to behave differently? Would you expect
> them to develop something that they aren't already developing? Would
> you expect them to do something that they aren't already doing.
> >

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 21 Jun 2012 22:09:00 +0200

> So, I will ask my question yet another way. Assume that the operator of a large network reads your draft. Although this operator has years of experience with large-scale rollouts, he doesn't see hi…

As said before, an operator who has a well working procedure and he
does not read anything that causes him to change his judgement of his
working procedure likely won't change, why should he? We all know
"never change a working system without a good reason". I do not see
how any of this is related to the question whether he sees his
procedure covered in the document or not, which is one of the
assertions in your statement.

Frankly, I remain puzzled what the secret procedures are that are not
covered. But since they are secret, I likely will never know. But
then, someone running a secrect procedure can't really be surprised
that his procedure is not covered. He likely should be happy that it
is not covered - so it remains a secret. ;-)

/js

---

From: Melinda Shore <[email-masked]>
Date: Thu, 21 Jun 2012 13:12:12 -0800

> such informational documents).

It seems to me that you and Ron may be talking past one another
to a certain extent.  One of the things that's really popped out
is that this model is not at all uncommon in voice and telecomm
environments, and the document doesn't really stress that, while
it also doesn't bring out very clearly that this model is not
applicable to things like CPE environments, etc.  I think Ron's
question is this: if I'm running an environment to which this
document is not applicable, what would I think when I read the
document?  Is it clear that it is *not* intended to apply to me?

Melinda

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 22 Jun 2012 07:05:25 +0200

On Thu, Jun 21, 2012 at 01:12:12PM -0800, Melinda Shore wrote:
> applicable to things like CPE environments, etc.  

Yes, it is likely we talk past each other. What is "this model"
you are talking about? Can you be more concrete?

> I think Ron's question is this: if I'm running an environment to
> which this document is not applicable, what would I think when I
> read the document?  Is it clear that it is *not* intended to apply
> to me?

This assumes that the document is intended to apply to some scenarios
but not to other scenarios. So far I have see statements that there
are secred homegrown mechanism not covered in the document. I do not
know what they are so I can't tell whether that is related to
identifiable environments or certain types of businesses or something
else. Are people looking for a disclaimer such as:

  Network operators executing procedures in certain environments that
  are not covered in this document and that work well for them may
  choose to continue using their working procedures.

Isn't that obvious? Does this not at the end apply to all
informational RFCs (and many other RFCs) as well?

My feeling is that I still have not understood what the concern is.
I am sorry for that. If someone else on the list can help resolving
this, this would be very much appreciated.

/js

---

From: t.petch <[email-masked]>
Date: Fri, 22 Jun 2012 16:42:50 +0100

---- Original Message -----
From: "Juergen Schoenwaelder" <[email-masked]>
To: "Melinda Shore" <[email-masked]>
Cc: <[email-masked]>
Sent: Friday, June 22, 2012 6:05 AM
> you are talking about? Can you be more concrete?

Juergen

I do see a particular model here, of downloading a configuration file
over a secure channel from a configuration server, all very Netconf-y or
SNMP-y.

There is a different model, which I see more often, of copying an image
to a disk, the image containing all the software and configuration
together.  Sometimes this is done from a CD on site, but it can also be
done over a suitably fast network.  It has something in common with what
you describe, but then so does almost any application that sets up a
secure channel and transfers information:-)

Configuring is different and reading the I-D I am left feeling that this
is missing, even if I find it harder to say just why that is so.  I
remember the advent of PCs with organisations being told with 100 PCs
you will not care how you configure, but when you have tens of
thousands, then not automating could destroy your business.  What
technology was used was irrelevant, it just had to be automated to the
point where the cost to the business was bearable, and in those days,
most businesses had not realised how expensive configuring devices on an
individual basis was.  That is common to all networks.

Equally, the cost of configuration changes can damage a business and so
may need to be eliminated, or cut to the bone.

Tom Petch