Subject: [alto] ALTO Operations and Management

From: Richard Alimi <[email-masked]>
Date: Sun, 29 Jul 2012 23:43:12 -0700

Hi All,

As you probably know, one of the new requirements on the protocol is:
   An ALTO client protocol MUST provide adequate
   mechanisms for operations and management support, as outlined in RFC
   5706 [RFC5706].

The points raised by RFC5706 are very good and should indeed be
solved. What I'm missing currently is a clear picture of exactly what
this requirement dictates for draft-ietf-alto-protocol.  In
particular, the scope of RFC5706 is *huge* - to address every
suggested point fully would take another couple of years (e.g.,
specify a management API for ALTO Servers, a configuration language,
techniques for verifying end-to-end operation, etc).

I'll suggest a way forwards - please provide feedback if you have a
chance.  I'd really like to get something mostly complete by the end
of IETF week if possible.

Proposal:

If we separate Operations (RFC5706, Section 2) from Management
(RFC5706, Section 3), I can make the following general observation:
- Operations: the protocol draft seems like a fine place to document
these, with the exception of 2.6 ("Verifying Correct Operation") which
is covered in Section 6.3 of draft-ietf-alto-deployments-04
- Management: draft-ietf-alto-deployments seems to already cover some
of this, and could probably be expanded to cover the rest (well, the
rest that we intend to solve) to keep the related information
together.  That seems preferable than copying a bunch of content from
draft-ietf-alto-deployments into draft-ietf-alto-protocol when we had
intentionally kept them separate in the first place.  The exception to
this might be dictating which protocols to use for some functions that
we know would be needed (syslog, IPFIX), which seem fine to have in
draft-ietf-alto-protocol.

Thoughts on that?  The intent isn't to just punt this over to another
draft, but rather to avoid duplicating text between different
documents and finish the protocol draft in a reasonable amount of
time.

If that *does* sound reasonable, I can write up some more concrete
text for the Operations section with a pointer indicating that
Management considerations are being documented in
draft-ietf-alto-deployments (seems like an Informative reference, so
that should be fine?)

If that *doesn't* sound reasonable, I'd really like to hear from
people what should be documented in the ALTO Protocol draft itself to
support Operations and Management.  Below my signature, I've included
some quick notes below about what I think *could* be discussed for
most of the points raised by RFC5706.  I don't think the authors of
the new requirement envisioned all of that to be included in
draft-ietf-alto-protocol, but I'd need some help in defining where to
draw the line.

Thanks,
Rich

-- Operations --

(1) Installation and Initial Setup
- By default an ALTO Server may be configured with a single PID
containing all IPv4/IPv6 addresses and a Cost Map with no entries.
Something more interesting should be configured explicitly (either
dynamically or statically).
- Specifications for underlying protocols (e.g., TCP, HTTP) should be
consulted for their available settings/defaults.

(2) Migration Path
- No migration path for ALTO Servers since there is no previous
protocol providing same functionality.
- Client-side: Applications making use of other data sources for
network information should consider ALTO an additional source of
information, but ALTO need not be the sole source of network
information.

(3) Requirements on Other Protocols and Functional Components
- Protocol-wise, ALTO assumes standard HTTP and JSON implementations exist.
- Functional-wise: An ALTO Server assumes that it can gather
sufficient information to populate Network and Cost maps.  "Sufficient
information" is dependent on the information being exposed, but likely
includes information such as IGP and EGP RIBs (more details in Section
2.2 of draft-ietf-alto-protocol-12).  Specific mechanisms have been
suggested in the past as extensions (e.g.,
draft-medved-alto-svr-apis-00).

(4) Impact on Network Operation
- Operators should consider impacts on (or integration with) Traffic
Engineering, since ALTO may shift traffic patterns.
- Privacy considerations for ISPs (see 10.1 of draft-ietf-alto-protocol-12)
- Impact of incorrect/faked guidance (see 10.3 of
draft-ietf-alto-deployments-04)

(5) Verifying Correct Operation
- For ALTO-specific monitoring and metrics, see 6.3 of
draft-ietf-alto-deployments-04

-- Management --

(1) Interoperability
- A common management API would be desirable given that ALTO Servers
may typically be configured with dynamic data from various sources,
and ALTO Servers are intended to scale horizontally for
fault-tolerance and reliability
  - management protocol should operate semantically at the level of
ALTO maps or other exposed information
- Entities that compute dynamic maps/guidance and entities that
measure ALTO's effects should use IPFIX information elements
- ALTO Servers should use syslog to log events.

(2) Management Information
- Informational Model
  - Data Sources - the set of data sources made available to an ALTO Server
  - Information Resource Directory - the set of ALTO Information
exposed by an ALTO Server
  - Section 4 and 5 of draft-ietf-alto-protocol-12 describe Network
Map, PIDs and Cost Maps, which are central entities of the
informational model.
  - ALTO guidance: ALTO Server may use internal logic/algorithms to
generate responses to ALTO Clients (map and endpoint-based queries);
may be customized to a particular deployment environment, so not
standardized (as long as responses follow protocol spec). May use
Network Map and Cost Map to answer queries, or may use custom logic.

(3) Fault Management
- To be discussed in 6.3 of draft-ietf-alto-deployments

(4) Configuration Management
- ALTO Server requires at least the following (high-level) logical inputs:
  - Data Sources (raw network , or explicitly-provided ALTO-level
information such as a Network Map and Cost Map)
  - Security policies
- Configuration interface should use ALTO data structures (as
specified in the ALTO Protocol itself) wherever possible to configure
the ALTO information exposed by an ALTO Server.  Specific syntax may
differ if porting to a language that is not JSON, but structures
should remain consistent.
- Specify standard configuration language and interface for interoperability
- Multiple ALTO Servers my be deployed for horizontal scalability. A
centralized configuration database may be used to ensure they are
providing the desired ALTO information with appropriate security
controls.  The ALTO information (e.g., Network Maps and Cost Maps)
being served by each ALTO Server, as well as security policies (HTTP
authentication, SSL/TLS client/server authentication, SSL/TLS
encryption parameters) intended to serve the same information should
be monitored for consistency.
- Verifying Correct Operation: Discussed in 6.3 of
draft-ietf-alto-deployments (and future revisions)

(5) Accounting Management
- Discussed in 6.3 of draft-ietf-alto-deployments (and future revisions)

(6) Performance Management
- List interesting counters
  - e.g., Number of requests for each service, responses, bytes
in/out, CPU utilization, ALTO map/guidance updates, number of PIDs,
map sizes (bytes, number of entries), etc
- Interesting data accessible by IPFIX
  - e.g., bytes per src/dst IP
- network devices
  - link utilization

(7) Security Management
- Readers should refer to documentation on HTTP and SSL/TLS.
- ALTO Servers will typically receive a large number of requests.
Intrusion Detection Systems that can separate malicious attacks from
noise are suggested when generating alerts.

Thanks,
Rich

---

From: "Y. Richard Yang" <[email-masked]>
Date: Tue, 31 Jul 2012 08:45:00 +0800

Hi Rich,

A very good effort to support Operations and Management. A rough separation
along the line between Operations and Management to use two documents seems
to be reasonable to me. If anyone has pointers on how another similar
protocol handles O/M, it will be helpful.

A particular item that appears to be challenging, as you also pointed out,
is Configuration Management (item 4 in your email). I can imagine that a
network operator may use a proprietary configuration specification system
or we need to develop a standard format (as you mentioned in your outline)?
If only static configuration is allowed, the task may be manageable, but
dynamic input (and transformation may be involved), and hence I see
challenge here. A complex Configuration Management Specification may also
lead to a complex management API (e.g., consistency of extracting data from
multiples ounces such as MIBs).

Another comment is on Fault Detection and Verfcation of Operation. Since
your these is on robust configuration/verification, we know it can get
complex/sophisticated quickly. Simply responding to some probes may not be
enough... How do we agree on the detection and verification scheme?

The 3rd comment is on other Services beyond Net and Cost Maps. Do we
include them?

Thanks!

Richard

On Sunday, July 29, 2012, Richard Alimi wrote:

> _______________________________________________
> alto mailing list
> [email-masked] <javascript:;>
> https://www.ietf.org/mailman/listinfo/alto
>

---

From: Richard Alimi <[email-masked]>
Date: Tue, 31 Jul 2012 08:06:49 -0700

On Mon, Jul 30, 2012 at 5:45 PM, Y. Richard Yang <[email-masked]> wrote:
> network operator may use a proprietary configuration specification system or
> we need to develop a standard format (as you mentioned in your outline)?

RFC5706 seems to strongly hint (and I agree) that interoperability is
important even for configuration management.  If I were an operator
using equipment from multiple vendors, I might write the necessary
scripting and abstractions to ease management, but I'd probably prefer
this to be uniform and not have to implement my own abstraction layer.

> If
> only static configuration is allowed, the task may be manageable, but

Completely agreed.  I think most deployments would benefit from having
dynamic updates, particularly in the cost maps.  I could imagine a
couple of ways to approach such an API:
(1) Provide an API to explicitly configure maps (and perhaps some sort
of equivalent for the endpoint-based services), and leave mechanisms
computing the ALTO information (and integration with data sources) out
of scope.
(2) Do (1) and also specify how this ties into various data sources.

> enough... How do we agree on the detection and verification scheme?

I'm not sure :) Fault detection and verification can be complicated
enough when you know most things about the system.  In the ALTO case,
there are at least a few things that are going to make this much
harder:
- We tell ALTO clients that this is just another data source and that
it is purely advisory.  It isn't necessarily clear whether changes are
due to ALTO or some other data source.
- In certain deployment environments, application developers probably
will not configure ALTO Clients to supply performance feedback or
application-level details even if we provided an interface
- This is only a portion of Internet traffic, and it isn't clear
whether a particular packet was influenced by ALTO or not since ALTO
doesn't (currently) have any dataplane components.
- ALTO guidance from one ISP may influence traffic patterns in another ISP.

>
> The 3rd comment is on other Services beyond Net and Cost Maps. Do we include
> them?

Yes - they should be included too.  Apologies for omitting them from
my initial note - I was mainly trying to sketch the overall components
involved.

Thanks!
Rich

>> [email-masked]