Subject: Re: [alto] New Version Notification for draft-lachosrothenberg-alto-brokermdo-01.txt

From: Danny Alex Lachos Perez <[email-masked]>
Date: Mon, 2 Jul 2018 21:24:53 -0300

Dear WG members

This new version (-01) considers a section on benefits and open questions
(section 7) where we analyze the advantages and open issues in the
broker-assisted architecture. Moreover, we removed the Property Map
Extension section because the current Property Map draft [0] already
supports property values encoded as JSONArray.

Other changes include (i) update the Problem Statement and Challenges
section and (ii) many minor style and grammar edits.

Comments and feedback will be highly appreciated.

Thank you very much

[0]
https://tools.ietf.org/html/draft-ietf-alto-unified-props-new-04#section-4.6

Ss

Danny Lachos

On Mon, Jul 2, 2018 at 4:18 PM <[email-masked]> wrote:

>
> A new version of I-D, draft-lachosrothenberg-alto-brokermdo-01.txt
> has been successfully submitted by Danny Alex Lachos Perez and posted to
> the
> IETF repository.
>
> Name:           draft-lachosrothenberg-alto-brokermdo
> Revision:       01
> Title:          ALTO-based Broker-assisted Multi-domain Orchestration
> Document date:  2018-07-02
> Group:          Individual Submission
> Pages:          22
> URL:
> https://www.ietf.org/internet-drafts/draft-lachosrothenberg-alto-brokermdo-01.txt
> Status:
> https://datatracker.ietf.org/doc/draft-lachosrothenberg-alto-brokermdo/
> Htmlized:
> https://tools.ietf.org/html/draft-lachosrothenberg-alto-brokermdo-01
> Htmlized:
> https://datatracker.ietf.org/doc/html/draft-lachosrothenberg-alto-brokermdo
> Diff:
> https://www.ietf.org/rfcdiff?url2=draft-lachosrothenberg-alto-brokermdo-01
>
> Abstract:
>    Evolving networking scenarios (e.g., 5G) demand new multiple
>    administrative domain (aka multi-domain) orchestration models.  This
>    document proposes the use of Application-Layer Traffic Optimization
>    (ALTO) services to offer topology and resources addressing network
>    service discovery and provisioning by multi-domain orchestrators.
>    The ALTO services with the proposed protocol extension offer
>    aggregated views on various types of resources contributing to a more
>    simple and scalable solution for resource and service discovery in
>    multi-domain, multi-technology environments.
>
>
>
>
>
> Please note that it may take a couple of minutes from the time of
> submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> The IETF Secretariat
>
>

---

From: "Y. Richard Yang" <[email-masked]>
Date: Sun, 8 Jul 2018 21:47:03 -0400

Dear Danny, Christian,

I started to read this draft. It addresses an important, timely issue.
Below is the first part (on Sections 1-6) of my review, and I will send the
second part on Sections 7-9 by tomorrow.

Great work!
Richard

====
Existing proposals for the network service orchestration are
   intrinsically conceived for single administrative domain scenarios.

     [yry] Add citations right after existing proposals?

   For example, in the standard service orchestration model described in
   ETSI NFV MANO framework [ETSI-NFV-MANO], one orchestrator is supposed to
work within one administrative domain.  The analysis of
   orchestration and management of network Services over multiple

     [yry] case, network vs Services

   administrative domains have begun to be addressed by ETSI
   in [ETSI-NFV-MANO-MDO].

     [yry] The last sentence gives a related work. To make reading easier,
     it may help to differentiate between proposals (first sentence) and
analysis (this sentence).

   All such proposals described above envision the potential
   introduction of new business model approaches, including federation
   models [PPP-5:2013] among administrative domains.  In this context,
   this document considers each network operator involved in the
   community advertises its abstracted capabilities (e.g., software/
   hardware resources, physical/virtual network functions, etc.) to a
   broker (i.e., 3rd party).  This latter, in its turn, provides or

     [yry] latter -> broker

   assists coordinate E2E network services spanning multi-domain
   networks.

5.  Problem Statement and Challenges

   The provision of a complete E2E network service requires chaining
   services provided by multiple network operator with multiple

     [yry] operator -> operators

   technologies.  In this multi-domain environment, the orchestration
   process will require an advertise mechanism through which single

     [yry] advertise -> advertisement, single -> individual

   domains can describe their capabilities, resources, and VNFs in an
   interoperable manner.  Moreover, a discovery mechanism is also
   necessary so that source domains can obtain candidate domains (with

     [yry] Although one can infer what source/candidate domains mean,
     it can help if you define them.

   the corresponding connectivity information) which can provide a part
   of the service and/or slide in an E2ENS requirement.

     [yry] slide -> slice?

   In order to the advertising and discovery process works in a proper
   way, a number of challenges can be identified:

     [yry] "In order to" -> "In order that"

   Lack of Abstractions:  Multiple vendors with heterogeneous
        technologies need an information model to adequately represent
        in confidentiality-preserving fashion the resource and topology
        information.

   Scalability:  Involves the distribution of topology and resource
        information in a peer-to-peer fashion (MdO-to-MdO).  Multi-

     [yry] Note that the previous item mentions generic information model
     but here mentions only topology and resource information. It can help
if
     the scoping is clarified up front: only topology/resource.

        operator multi-domain environments where the information
        distribution is advertised in a peer-to-peer model scales
        linearly.  It means more MdO interconnections one has, the more

      [yry] Why linearly? There can be ambiguity. Assume the total
      transmission load of one domain grows linearly with n, which is
      the number of domains. Total system scales w/ n^2. But there can be
other peer-to-peer systems such as CHORD which publish
(advertises) with a constant cost. It helps to specify some more details.

        it "costs" to distribute.

   Flexibility:  Considers that a distributed approach does not allow
        domains without physical infrastructure (e.g., without BGP or
        BGP-LS) to advertise resource capabilities and networking
        resources.  Such procedures consist in deploying and configuring
        physical peering points for these domains.

     [yry] The description above is not clear to me yet. One can run BGP
w/o a physical infrastructure. It helps if you can clarify this challenge.

   Complexity:  Refers to the discovery mechanism to pre-select
        candidate domains, accounting for resources and capabilities,
        necessary for an E2E network service deployment.  An intrinsic
        complexity exists in the process of assembling, logically
        organizing, and enabling abstraction views of different
        resources and capabilities in multi-domain scenarios.

    [yry] A summary comment on the 4 items. Are they challenges, issues
    or requirements? The wording appears to have multiple aspects: lacking
    of abstractions seems to imply issues; flexibility seems to imply
    requirements... Is the set complete, for example how about security,
autonomy, privacy ...---some you touch right below?  One more comment is
that the terms used (e.g., flexibility) can be too generic. If you go a bit
more specific, for example, Discovery Flexibility, it may help.

====

On Mon, Jul 2, 2018 at 8:25 PM Danny Alex Lachos Perez <[email-masked]>
wrote:

>> _______________________________________________
> alto mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/alto
>

-- 
-- 
 =====================================
| Y. Richard Yang <[email-masked]>   |
| Professor of Computer Science       |
| http://www.cs.yale.edu/~yry/        |
 =====================================

---

From: "Y. Richard Yang" <[email-masked]>
Date: Mon, 9 Jul 2018 17:13:52 -0400

Danny, Christian,

Very interesting work. I finished a review of Sec. 6, and below is
the second part of my review.

Richard

====
6.  Proposed Approach

   The primary design goal for ALTO-based Broker-assisted Multi-domain
   Orchestration is to discover resource and topology information from
   different administrative domains involved in the federation, while
   also safeguarding the privacy and autonomy of every domain.

     [yry] Connect to all requirements above?

   In the architectural proposal shown in Figure 1, a broker component
   is conceived to be working as coordinator of a set of MdOs, whose key

     [yry] coordinator -> a coordinator; whose it is not clear righ away
     what "whose" refers to, MdOs or broker?

   components are: the Inter-domain Resource (IdR), the Inter-domain
   Topology (IdT) and the ALTO Server.

                              BROKER COMPONENT
      [yry] COMPONENTS

              +--------------------------------------------+
              |                                            |
              |             +-----------------+            |
              |             |                 |            |
          XXXXXXXXXXXXXXXXXXX ALTO SERVER(s)  |            |
          X   |             |                 +            |
          X   |             +----------------+\            |
          X   |            /                   \           |
          X   |           /                     \          |
          X   |  +------+/+-------+     +---------------+  |
          X   |  | Inter-domain   |     | Inter-domain  |  |
          X   |  | Topology (IdT) |     | Resource (IdR)|  |
          X   |  +-^------^-------+     +---^-------^---+  |
          X   |    |      |                 |       |      |
          X   +--------------------------------------------+
          X        |      |                 |       |
          X        |      |                 |       |
       +--X--------+---+--------------------+       |
       |               |  |                         |
       |               |  +------------+------------+---+
       |     MdO1      |               |                |
       |               +<------------->+                +---+
       +---------------+               |      MdO2      |   |
                                       |                |   |
                                       +-+--------------+   |
                                         |                  |
   Legend:                               |      MdO(n)      |
   XXX ALTO Protocol                     +------------------+

       Figure 1: Broker-assisted Multi-operator Network Architecture

    [yry] Does it help to label all, or key edges, not only the single
    XXX, on which protocol (protocols) will be used? You mentioned
    BGP/BGP-LS, ...

6.1.  Inter-domain Resource (IdR) Component

   It creates a hierarchical database that contains inter-domain
   resource information such as resource availability (i.e., CPU,
   memory, and storage), Virtual Network Functions (VNFs) and Physical
   Network Functions (PNFs) supported and Service Access Points (SAPs)
   to access those resources.  UNIFY [UNIFY.NFFG], TOSCA [TOSCA], ETSI-

     [yry] It can be ambiguous what Service Access Points refer to,
     only VNFs/PNFs or all? The idea of a hierarchical database appears to
    be not fully specified.

   NFV [ETSI-NFV-MANO], among other data models can be used to create
   the interface between IdR and MdOs.

      [yry] IdR not defined yet---appeared only in the section title.

6.2.  Inter-domain Topology (IdT) Component

   A hierarchical TED (Traffic Engineering Database) that contains
   inter-domain network topology information including additional key
   parameters (e.g., throughput and latency of links).  This information
   can be retrieved from each MdO through BGP-LS or REST interfaces.

6.3.  ALTO Server Functionalities

   The ALTO server component is the core of the broker layer.  Multiple
   logically centralized ALTO servers use the information collected from
   IdR and IdT modules to create and provide abstract maps with a

     [yry] module vs component

   simplified view, yet enough information about MdOs involved in the
   federation.  This information includes domain-level topology, storage
   resources, computation resources, networking resources and PNF/VNF
   capabilities.

     [yry] Consistency: missing CPU, as above?

   As an ALTO client, each MdO sends ALTO service queries to the ALTO
   server.  This server provides aggregated inter-domain information
   exposed as set ALTO base services defined in [RFC7285], e.g., Network
   Map, Cost Map and ALTO extension services, e.g., Property
   Map [DRAFT-PM], Multi-Cost Map [RFC8189], Path Vector [DRAFT-PV].

   For example, when a source MdO receives a customer service request,
   it checks whether or not it can deliver the full service.  If it is
   unable to do so, the MdO consumes from the ALTO Server the Property
   Map service to have a clear global view of the resource information
   offered by other MdOs.  This information allows discovering which
   candidate MdOs may be contacted to deliver the remaining requirements
   of a requested end-to-end service deployment.  The connectivity
   information among discovered MdOs can be retrieved by a Cost Map
   service, responding, for instance, a path vector with the AS-level
   topology distance between the source MdO and candidate MdOs.

6.4.  Filtered Cost Map Extension

     [yry] A bit disruptive transition. Why only Filtered Cost Map?
     It helps to state the basic ALTO services that broker uses and
     state that this is the one needs extension---I actually think
     this should be a new service, instead of filtered cost map.

   The ALTO server MUST provide connectivity information for every SG
   link in the SG path for an E2E requirement.  This information is the
   AS-level topology distance in the form of path vector, and it
   includes all possible ways for each (source node, destination node)
   pair in the SG link.

     [yry] Is it necessary to limit to only AS-hop distance? I assume
     that other distances such as bw, latency can be useful as well.

   In this section, we introduce a non-normative overview of the
   Filtered Cost Map defined in Section 6.1 of [DRAFT-PV] [1].

   The specifications for the "Media Types", "HTTP method",
   "Capabilities" and "Uses" (described in Section 6.1 of [DRAFT-PV]
   [2]) are unchanged.

6.4.1.  Accept Input Parameters

   The ReqFilteredCostMap object in Section 6.1.2 of [DRAFT-PV] [3] is
   extended as follow:

     object {
       [NFFG sg;]
     } ReqFilteredCostMap;

      [yry] If sg is optional, the req can have an empty body. What is
      the response to an empty request? Non-optional makes more sense,
      to me.

      [yry] One key question that we should think: is it possible that
      NFFG is an example of a more *generic* query model, for example,
      waypoint model, so that the generic query is
        src -> dst: waypoints?

     object {
       JSONString nfs<1..*>;
       JSONString saps<1..*>;
       NextHops sg_links<1..*>;
       REQs reqs<1..*>;
     } NFFG;

     object {
       JSONNumber id;
       JSONString src-node;
       JSONString dst-node;
     } NextHops;

     object {
       JSONString id;
       JSONString src-node;
       JSONString dst-node;
       JSONNumber sg-path<1..*>;
     } REQs;

   sg:  If present, the ALTO Server MUST allow the request input to
      include an SG with a formatted body as an NFFG object.  An NFFG
      object contains NFs, SAPs, SG links representing logical
      connections between NFs, SAPs or both and E2E requirements as a
      list of ids of SG links.

   It is worth noting that further versions of this draft will define a
   more elaborated NFFG object to support extended parameters such as
   monitoring parameters, resource requirements, etc.

    [yry] Also, need consistency requirement, for example, src-node string
    appears in union of nfs, saps ...

6.4.2.  Response

   If the ALTO client includes the path vector cost mode in the "cost-
   type" or "multi-cost-types" field of the input parameter, the
   response for each SG link in each E2E requirement MUST be encoded as
   a JSONArray of JSONArrays of JSONStrings.  Anyone of the sub-arrays
   indicates a potential candidate path calculated as the per-domain
   topological distance corresponding to the amount of traversing
   domains.

      [yry] The encoding is quite interesting, but without an example,
      and a bit explanation, it can be hard to understand the nested
      data structure. I feel that the model can be a generic model,
      such as waypoint, beyond SG; see above.

   Moreover, as defined in Section 6.3.6 of [DRAFT-PV] [4], If an ALTO
   client sends a request of the media type "application/alto-
   costmapfilter+json" and accepts "multipart/related", the ALTO server
   MUST provide path vector information along with the associated
   Property Map information (e.g., entry points of the corresponding
   foreign domains), in the same body of the response.

   Section 6.5.2 gives an example of the Filtered Cost Map query and the
   corresponding responses.

6.5.  Examples of Message Exchange

   This section list a couple of examples of the Property Map and
   Filtered Cost Map queries and the corresponding responses.  These
   responses are based on the information in Table 1 and Table 2 of a
   use case implementation described in Appendix A.

6.5.1.  Property Map Service

   In this example, the ALTO client wants to retrieve the entire
   Property Map for PID entities with the "entry-point", "cpu", "mem",
   "storage", "port" and "nf" properties.

     GET /propmap/full/inet-ucmspn HTTP/1.1
     Host: alto.example.com
     Accept: application/alto-propmap+json,application/alto-error+json

     HTTP/1.1 200 OK
     Content-Length: ###
     Content-Type: application/alto-propmap+json
     {
       "property-map": {
           "pid:AS1": {
               "entry-point": [ "http://172.25.0.10:8888/escape" ],
               "cpu": [ "50.0" ],
               "mem": [ "60.0" ],
               "storage": [ "70.0" ],
               "port": [ "SAP1" ],
               "nf": [ "NF1", "NF3" ]
           },
           "pid:AS2": {
               "entry-point": [ "http://172.26.0.10:8888/escape" ],
               "cpu": [ "10.0" ],
               "mem": [ "20.0" ],
               "storage": [ "30.0" ],
               "nf": [ "NF2" ]
           },
           "pid:AS3": {
               "entry-point": [ "http://172.27.0.10:8888/escape" ],
               "cpu": [ "80.0" ],
               "mem": [ "90.0" ],
               "storage": [ "100.0" ],
               "port": [ "SAP2" ],
               "nf": [ "NF1", "NF3" ]
           }
       }
     }

       [yry] A key issue that one may encounter in multi-domain
       aggregation is ambiguity of aggregated resources. Is the semantics
       that each resource is the *local* resource? If so, is there
       a naming/scoping requirement on naming entities?

       [yry] If you do hierarchical brokers and each broker can hide the
details, the resource aggregation can become ambiguous: (1) broker 1 can
       say it has 10 units of CPUs, and broker 2 can say the same, but they
       may aggregate the 10 from the same individual domain.

6.5.2.  Filtered Cost Map Service

   The following example uses the Filtered Cost Map service to request
   the path vector for a given E2E requirement.  The SG request
   information in Table 2 is used to describe the service, and it is
   composed of three NFs (NF1, NF2, and NF3) and two SAPs (SAP1 and
   SAP2).  Links connecting the NFs and SAPs ("sg_links" tag) are also
   included, followed by an E2E requirement ("reqs" tag) with
   information about the order in which NFs are traversed from SAP1 to
   SAP2.

   Note that the request accepts "multipart/related" media type.  This
   means the ALTO server will include associated property information in
   the same response.

       POST /costmap/pv HTTP/1.1
       Host: alto.example.com
       Accept: multipart/related, application/alto-costmap+json,
             application/alto-propmap+json, application/alto-error+json
       Content-Length: [TBD]
       Content-Type: application/alto-costmapfilter+json

       {
         "cost-type": {
           "cost-mode": "array",
           "cost-metric": "ane-path"
         },
         "sg": {
           "nfs": [ "NF1", "NF2", "NF3" ],
           "saps": [ "SAP1", "SAP2" ],
           "sg_links":[
             {
               "id": 2,
               "src-node": "SAP1",
               "dst-node": "NF1",

             },
             {
               "id": 2,
               "src-node": "NF1",
               "dst-node": "NF2",
             },
             {
               "id": 3,
               "src-node": "NF2",
               "dst-node": "NF3",
             },
             {
               "id": 4,
               "src-node": "NF3",
               "dst-node": "SAP2",
             }
           ],
           "reqs": [
             {
               "id": 1,
               "src-node": "SAP1",
               "dst-node": "SAP2",
               "sg-path": [ 1, 2, 3, 4 ]
             }
           ]
         }
       }

   The ALTO server returns connectivity information for the E2E
   requirement provided by the ALTO Client request of the above example.
   Also, the response includes Property Map information for each element
   in the path vector.  In this case, it is retrieved a Property Map
   with the "entry-point" property, i.e., the URL of the MdO entry point
   for the corresponding network.

     HTTP/1.1 200 OK
     Content-Length: [TBD]
     Content-Type: multipart/related; boundary=example

     --example
     Content-Type: application/alto-endpointcost+json

     {
       "meta": {
         "cost-type": {
            "cost-mode": "array",
            "cost-metric": "ane-path"
          },
       },

       "cost-map": {
         "SAP1": {
           "SAP2": {
               "SAP1": {
                   "NF1": [
                     [ "AS1" ], [ "AS1", "AS2", "AS3" ]
                   ]
               },
               "NF1": {
                   "NF2": [
                     [ "AS1", "AS2" ], [ "AS3", "AS2" ]
                   ]
               },
               "NF2": {
                   "NF3": [
                     [ "AS2", "AS1" ], [ "AS2", "AS3" ]
                   ]
               },
               "NF3": {
                   "SAP2": [
                     [ "AS1", "AS2", "AS3" ], [ "AS3" ]
                   ]
               }
           }
         }
       }
     }

     --example
     Content-Type: application/alto-propmap+json

     {
       "property-map": {
         "pid:AS1": { "entry-point": "http://172.25.0.10:8888/escape" },
         "pid:AS2": { "entry-point": "http://172.26.0.10:8888/escape" },
         "pid:AS3": { "entry-point": "http://172.27.0.10:8888/escape" }
       }
     }

     --example--
   [yry] Very interesting design. It helps if you add some text to go over
   the meaning, such as [ "AS1" ], [ "AS1", "AS2", "AS3" ] are two
alternative
   paths.
====

On Sun, Jul 8, 2018 at 9:47 PM Y. Richard Yang <[email-masked]> wrote:

> On Mon, Jul 2, 2018 at 8:25 PM Danny Alex Lachos Perez <
> [email-masked]> wrote:
>
>> Dear WG members
>>

---

From: Danny Alex Lachos Perez <[email-masked]>
Date: Mon, 9 Jul 2018 19:16:52 -0300

Hello Richard

Thanks a lot for your reviews. We will go over your comments in detail.

Ss

Danny Lachos

On Mon, Jul 9, 2018 at 6:14 PM Y. Richard Yang <[email-masked]> wrote:

>>      but here mentions only topology and resource information. It can
>> help if
>>> This new version (-01) considers a section on benefits and open
>>> questions (section 7) where we analyze the advantages and open issues in
>>> the broker-assisted architecture. Moreover, we removed the Property Map
>>>> has been successfully submitted by Danny Alex Lachos Perez and posted
>>>> to the
>>>> IETF repository.
>>>>