Subject: RE: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

From: Linda Dunbar <[email-masked]>
Date: Thu, 22 May 2014 18:35:00 +0000

I have sent comments several times within NVO3 WG, but my comments hadn't been properly addressed.

So, I am sending them again; hopefully those comments can be addressed during the IETF last call.

NVO3 is about virtualization for data center (as stated in its charter statement). Inter-subnets communication (or inter Virtual Networks communication) is a big part (if not major part) of data centers traffic. Hosts in one subnet frequently communicate with hosts in different subnets or peers external to DC.

Yet the current framework draft focuses so much on encapsulating/decapsulating TS traffic, making NVO3 a lot like L2VPN or L3VPN MPLS network. Since IETF already has dedicated WGs for L2VPN and L3VPN, the NV03 WG should focus more on how inter-subnet communication is achieved in the overlay environment.

Here are the suggested changes:

-       The current Figure2 (Generic Reference model of NVo3) is like a clone to L2VPN. The NVO3 context reference model needs to add a gateway entity, as shown below, for relaying traffic from one VN to another VN.
                              _,....._
                           ,-'        `-.
                          /   External  `.
                         |     Network   |
                         `.             /
                           `.__     _,-'
                               `''''
                                  |
                             +---------+
                             | Gateway |
                             +----+----+
                             +----+----+
                             |   NVE   |
                             +-----+---+
       +--------+                  |                          +--------+
       | Tenant +--+               |                     +----| Tenant |
       | System |  |               |                    (')   | System |
       +--------+  |          ................         (   )  +--------+
                   |  +-+--+  .              .  +--+-+  (_)
                   |  | NVE|--.              .--| NVE|   |
                   +--|    |  .              .  |    |---+
                      +-+--+  .              .  +--+-+
                      /       .              .

-       There are good inter virtual network descriptions in http://datatracker.ietf.org/doc/draft‐yong‐nvo3‐frwk‐dpreq‐addition/. The content from this draft should be included in the general framework, especially:

      2.2. L2-3 NVE Providing IP Routing/Bridging-like Service (Framework Addition)
      L2-3 NVE is similar to IRB function on a router [CIRB] device today. It supports the TSes attached to the NVE (locally or remotely) to communicate with each other when they are in a same route domain, i.e. a tenant virtual network. The NVE provides per tenant virtual switching and routing instance with address isolation and L3 tunnel encapsulation across the core. The L2-3 NVE supports the bridging among TSes that are on the same subnet and the routing among TSes that are on the different subnets.

-       Section 2.3.1. L2 NVE Providing Ethernet LAN-Like services:
Need to add a paragraph to address that great amount of traffic in DC is across VN (or across subnets).  Need to describe how across subnet is performed.  E.g. relayed at L2/L3 gateway.

-       Section 2.3.2. L3 NVE Providing IP/VRF-like forwarding
RFC4364 is about IP over MPLS network, i.e. the underlay is MPLS network. But NVO3’s underlay is IP.

-       Page 13 Second bullet: the text says that each VNI is automatically generated by the egress NVE. Isn’t each VNI supposed to match a local virtual network (e.g. represented by VLAN)? When the same NVE acts as Ingress NVE for the attached VMs, aren’t the VNI for ingress direction statically provisioned? Why need automatic egress VNI generation?

-       Section 3.2 Multi-homing: LAG is usually used to bundle multiple ports on one device. In the multi-homing environment, there are multiple NVEs. Besides, LAG and STP in the multi-homing environment can’t really prevent loop. You will need something like TRILL’s Appointed Forwarders to prevent loops in multi-homing environment.

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

The IESG has received a request from the Network Virtualization Overlays WG (nvo3) to consider the following document:
- 'Framework for DC Network Virtualization'
  <draft-ietf-nvo3-framework-06.txt> as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final comments on this action. Please send substantive comments to the [email-masked]<mailto:[email-masked]> mailing lists by 2014-06-04. Exceptionally, comments may be sent to [email-masked]<mailto:[email-masked]> instead. In either case, please retain the beginning of the Subject line to allow automated sorting.

Abstract

       This document provides a framework for Network Virtualization
       Overlays (NVO3) and it defines a reference model along with logical
       components required to design a solution.

The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-nvo3-framework/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-nvo3-framework/ballot/

No IPR declarations have been submitted directly on this I-D.

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 22 May 2014 20:19:40 +0000

Key,

Comments are inserted below:

From: Ken Gray (kegray) [mailto:[email-masked]]
Sent: Thursday, May 22, 2014 2:58 PM
To: Linda Dunbar; [email-masked]; IETF-Announce
Cc: [email-masked]
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Personally, I think encap/decap manipulation is the essence of the "gateway" or inter-virtual network communication.  To me, technically, all the NVEs are gateways of a sort.  You're just specifying that the permutations are variant (the network isn't homogenous in it's encap)?

[Linda] The encap/decap is only sending packets within one VN. For example,
when a host "a" in subnet "A" sends a packet to host "b" in subnet "B", the MAC address from "a" is actually the gateway MAC address for subnet "A". So the Ingress NVE is the NVE to which that "a" is attached, and the egress NVE is the one that "A" gateway is attached (which are possibly collocated). The Gateway terminates the MAC header from host "a", and relays the packet to "B" VN (i.e. subnet), add a new MAC header (with DA=
"b" MAC, SA=Gateway MAC, and the associated VLAN), and sends out the newly constructed Ethernet packet. The NVE to which the Gateway is attached (or collocated) has to resolve the egress NVE to which "b" is attached, encap the packet and sends to the "b" NVE.

Some NVE can support Gateway function, i.e. IRB function as described in the Section 2.2. of
http://datatracker.ietf.org/doc/draft-yong-nvo3-frwk-dpreq-addition/.

More to the point, your drawing seems to imply that some sort of additional functionality other than a header fix-up in the NVE is required (Gateway function above the NVE).  While what is depicted  CAN be an implementation of a gateway, logically ... some control function (not necessarily local on that device ...orchestration, controller, operator) could dictate/impose such a translation/transcription onto NVE at different points in the infrastructure and achieve the same result/effect.

[Linda] The "gateway" can be embedded in some NVEs. But many NVEs can't support gateway functions. In order to support gateway function, the NVE has to have the inter-subnet policies, or access the Firewalls.
For example, subnet A can send packets to subnet B, but A can't send to C. If NVEs are on servers, they may not have the needed policy to relay traffic between two different subnets. The data packets may need to be sent to the designated gateways.  The framework draft should address those issues.

So, if it's a generic drawing ...we don't need specific examples (IMO) unless we want to rathole on implementations to avoid inferring one is preferred.  If you feel a need for a generic conceptional representation, then I'd advise tagging it as such and not associating any implementation details (which is what I think your yong doc pointer does).

[Linda] IMHO, the generic drawing should have components to handle inter-VN communication, instead of simple clone to L2VPN.

Linda

From: Linda Dunbar <[email-masked]<mailto:[email-masked]>>
Date: Thursday, May 22, 2014 2:35 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, IETF-Announce <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

-       There are good inter virtual network descriptions in http://datatracker.ietf.org/doc/draft-yong-nvo3-frwk-dpreq-addition/. The content from this draft should be included in the general framework, especially:

-       Section 2.3.2. L3 NVE Providing IP/VRF-like forwarding
RFC4364 is about IP over MPLS network, i.e. the underlay is MPLS network. But NVO3's underlay is IP.

-       Page 13 Second bullet: the text says that each VNI is automatically generated by the egress NVE. Isn't each VNI supposed to match a local virtual network (e.g. represented by VLAN)? When the same NVE acts as Ingress NVE for the attached VMs, aren't the VNI for ingress direction statically provisioned? Why need automatic egress VNI generation?

-       Section 3.2 Multi-homing: LAG is usually used to bundle multiple ports on one device. In the multi-homing environment, there are multiple NVEs. Besides, LAG and STP in the multi-homing environment can't really prevent loop. You will need something like TRILL's Appointed Forwarders to prevent loops in multi-homing environment.

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: "Ken Gray (kegray)" <[email-masked]>
Date: Fri, 23 May 2014 12:30:45 +0000

Linda,

Ultimately, your "gateway" function is manifested as a forwarding entry.  You're describing the mechanics of how that entry is derived.  While there are aspects of map distribution that might affect those mechanics - that are specific elements of nvo3, I don't think describing them at this point in the doc was the author's purpose (or necessary).

It appears to me that your comment is much more simple - that we have omitted (the obvious) "you probably need to route between subnets and/or domains".  Which I suggest as the more generic text if you think their example is insufficient.

Why illustrate A method of doing so? Why jump down the rabbit hole of "all possible ways this could manifest?  None of that can be codified by this document.

Sent from my iPhone

On May 22, 2014, at 4:19 PM, "Linda Dunbar" <[email-masked]<mailto:[email-masked]>> wrote:

Key,

To: Linda Dunbar; [email-masked]<mailto:[email-masked]>; IETF-Announce

[Linda] The encap/decap is only sending packets within one VN. For example,
when a host “a” in subnet “A” sends a packet to host “b” in subnet “B”, the MAC address from “a” is actually the gateway MAC address for subnet “A”. So the Ingress NVE is the NVE to which that “a” is attached, and the egress NVE is the one that “A” gateway is attached (which are possibly collocated). The Gateway terminates the MAC header from host “a”, and relays the packet to “B” VN (i.e. subnet), add a new MAC header (with DA=
“b” MAC, SA=Gateway MAC, and the associated VLAN), and sends out the newly constructed Ethernet packet. The NVE to which the Gateway is attached (or collocated) has to resolve the egress NVE to which “b” is attached, encap the packet and sends to the “b” NVE.

More to the point, your drawing seems to imply that some sort of additional functionality other than a header fix-up in the NVE is required (Gateway function above the NVE).  While what is depicted  CAN be an implementation of a gateway, logically … some control function (not necessarily local on that device …orchestration, controller, operator) could dictate/impose such a translation/transcription onto NVE at different points in the infrastructure and achieve the same result/effect.

[Linda] The “gateway” can be embedded in some NVEs. But many NVEs can’t support gateway functions. In order to support gateway function, the NVE has to have the inter-subnet policies, or access the Firewalls.
For example, subnet A can send packets to subnet B, but A can’t send to C. If NVEs are on servers, they may not have the needed policy to relay traffic between two different subnets. The data packets may need to be sent to the designated gateways.  The framework draft should address those issues.

So, if it's a generic drawing …we don't need specific examples (IMO) unless we want to rathole on implementations to avoid inferring one is preferred.  If you feel a need for a generic conceptional representation, then I'd advise tagging it as such and not associating any implementation details (which is what I think your yong doc pointer does).

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: "Ken Gray (kegray)" <[email-masked]>
Date: Thu, 22 May 2014 19:58:25 +0000

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 23 May 2014 16:24:58 +0000

Ken,

If you prefer to call it "inter-subnets and/or domains routing" instead of gateway, it would be fine with me.

IMHO, the NVO3 framework draft should cover this key function of data center networking.

The illustrated example is only to show the normal across subnets/domains traffic pattern, to show why it is necessary for NVO3 to address inter-subnet/domain issues.
The implication of inter-subnet/domain routing:

-          If a NVE performs the inter-subnet/domain routing, the NVE needs to have the needed policy. Some NVEs may have the needed policies for all VNs in DC, some NVEs may only have portion of the policies, some NVEs may not have any.

-          For NVEs adjacent to the "DC Gateway" in the Figure 1 of the framework draft have to maintain the mappings for all hosts/VMs in DC. For large DCs with Tens of Thousands (or hundreds of thousands) VMs, maintaining those mapping can be very intensive.

Hope the authors can also address those technical issues I brought up:

Linda

From: Ken Gray (kegray) [mailto:[email-masked]]
Sent: Friday, May 23, 2014 7:31 AM
To: Linda Dunbar
Cc: [email-masked]; IETF-Announce; [email-masked]
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Linda,

Sent from my iPhone

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 25 May 2014 08:45:59 +1200

A few comments below. I can't help feeling that NVO3 is creating
a monster, however.

> 4.1. Pros & Cons
...
>           - Traffic carried over an overlay may not traverse firewalls and
>             NAT devices.

I don't know whether  "may not" means "might not" or "must not",
and that completely determines what the sentence means. For example,
does it mean this?
       - Traffic carried over an overlay might fail to traverse firewalls and
         NAT devices.

I suggest reviewing every instance of "may not" to avoid this
ambiguity.

>           - Hash-based load balancing may not be optimal as the hash
>             algorithm may not work well due to the limited number of
>             combinations of tunnel source and destination addresses. Other
>             NVO3 mechanisms may use additional entropy information than
>             source and destination addresses.

Load balancing appears out of nowhere here. Are we supposed to assume
that load balancing is a requirement? Load balancing between what -
between different tenants, different physical DCs, different servers?

Also, there seems to be an assumption that load balancing is only
based on addresses. Actually it's usually based on ports as well,
and more or less by definition they are invisible to the underlay.
So it's worse than "may not work well".

I would have expected QoS support to also appear as a challenge,
for similar reasons. Isn't giving tenants a fair share of the underlay
capacity an issue? (There's a mention of traffic engineering later,
but surely you don't want this issue to be handled by operators
twiddling knobs?)

> 4.2.4. Path MTU
...
>        TCP will
>        adjust its maximum segment size accordingly.

And how will that work for non-TCP traffic?

>        It is also possible to rely on the NVE to perform segmentation and
>        reassembly operations without relying on the Tenant Systems to know
>        about the end-to-end MTU. The assumption is that some hardware
>        assist is available on the NVE node to perform such SAR operations.
>        However, fragmentation by the NVE can lead to performance and
>        congestion issues due to TCP dynamics and might require new
>        congestion avoidance mechanisms from the underlay network [FLOYD].

In a word: yuck. Surely you should be recommending against anything like
that, or any attempt to re-segment TCP on the fly.

>        Finally, the underlay network may be designed in such a way that the
>        MTU can accommodate the extra tunneling and possibly additional NVO3
>        header encapsulation overhead.

Surely you should be recommending this, which is by far the safest
solution. (And of course it should allow for the IPv6 minimum MTU.)

> 7. References
...
>        [NVOPS] Narten, T. et al, "Problem Statement : Overlays for Network
>                  Virtualization", draft-narten-nvo3-overlay-problem-
>                  statement (work in progress)

Nit: that draft was replaced a long time ago by
http://tools.ietf.org/html/draft-ietf-nvo3-overlay-problem-statement
(which is already in the RFC Editor queue).

    Brian

---

From: "Ken Gray (kegray)" <[email-masked]>
Date: Sat, 24 May 2014 16:57:25 +0000

Hi, Linda.  I just don't read a requirement to describe gateway functionality anywhere in … or into …the posted charter.  The charter focuses primarily on the the creation of scalable DCVPNs but seems to leave explicitly connecting DCVPNs together out.  If you felt that was an oversight, it should have been addressed when we/nvo3 were chartered.

If the authors want to substitute interchangeable terms like gateway, router, inter-vpn for this generic boundary…I don't have any preference.

Otherwise, I'm for building no more than we need … and leaving purposely generic interfaces where work is out of scope … generic.  We don't need specific examples unless we're supposed to build something, and should resist the unnecessary bloat.

From: Linda Dunbar <[email-masked]<mailto:[email-masked]>>
Date: Friday, May 23, 2014 12:24 PM
To: Ken Gray <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, IETF-Announce <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Ken,

If you prefer to call it “inter-subnets and/or domains routing” instead of gateway, it would be fine with me.

-          For NVEs adjacent to the “DC Gateway” in the Figure 1 of the framework draft have to maintain the mappings for all hosts/VMs in DC. For large DCs with Tens of Thousands (or hundreds of thousands) VMs, maintaining those mapping can be very intensive.

Linda

From: Ken Gray (kegray) [mailto:[email-masked]]
Sent: Friday, May 23, 2014 7:31 AM
To: Linda Dunbar
Cc: [email-masked]<mailto:[email-masked]>; IETF-Announce; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Linda,

Sent from my iPhone

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 27 May 2014 15:02:25 +0000

Hi Brian,

More to what you said about MTU, in general when tunnels return a
Packet Too Big (PTB) message the message may be lost due to filtering
middleboxes in the return path to the source of the originating packet.
This can be especially problematic when the PTB reports a size less
than 1500 bytes, since most original sources expect to see a minimum
path MTU of that size.

There may be certain circumstances under which the tunnel orientation
would provide assurance against PTB message loss but (again in general)
if this cannot be assured then the tunnel is susceptible to black holes
unless outer IP fragmentation is used.

By "outer IP fragmentation is used", I mean use it only until it is
proven that it is no longer necessary - then, stop using it.

Thanks - Fred
[email-masked] 

> From: ietf [mailto:[email-masked]] On Behalf Of Brian E Carpenter
> Sent: Saturday, May 24, 2014 1:46 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization)
> to Informational RFC
>

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 27 May 2014 19:00:48 +0000

Ken,

Thanks for bringing out the requirement.
The very first sentence of the NVO3 charter is about "Support for multi-tenancy ... of data centers". If most VMs in Data Centers communicate with peers in different subnets, then the inter-subnet communication should be addressed by NVO3 framework draft.

Maybe my interpretation of the charter wording is wrong. I will leave the interpretation job to ADs then.

Linda

From: Ken Gray (kegray) [mailto:[email-masked]]
Sent: Saturday, May 24, 2014 11:57 AM
To: Linda Dunbar
Cc: [email-masked]; IETF-Announce; [email-masked]
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Hi, Linda.  I just don't read a requirement to describe gateway functionality anywhere in ... or into ...the posted charter.  The charter focuses primarily on the the creation of scalable DCVPNs but seems to leave explicitly connecting DCVPNs together out.  If you felt that was an oversight, it should have been addressed when we/nvo3 were chartered.

If the authors want to substitute interchangeable terms like gateway, router, inter-vpn for this generic boundary...I don't have any preference.

Otherwise, I'm for building no more than we need ... and leaving purposely generic interfaces where work is out of scope ... generic.  We don't need specific examples unless we're supposed to build something, and should resist the unnecessary bloat.

Ken,

Linda

Linda,

Sent from my iPhone

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 28 May 2014 17:24:05 +0000

Just to add one more point to what Brian has raised:

The Overlay Pros&Cons discussion (the first bullet of 4.1) should mention that the benefit of overlay is positively correlated with the ratio of  "intermediate nodes" to edge nodes (NVE).

In many of today's typical data centers, Access Switches are connected to the DC Gateways via one layer of Aggregation Switches or directly, i.e. the "Intra-DC network" in the Figure 1 of the draft only consists of the aggregation switches if NVEs are implemented on the Access Switches. Under this environment, the only beneficiary of Overlay are the aggregation switches. For Data centers with Access Switches connected DC Gateways directly, the benefit disappears. 

Linda

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Brian E Carpenter
Sent: Saturday, May 24, 2014 3:46 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

A few comments below. I can't help feeling that NVO3 is creating a monster, however.

...
>           - Traffic carried over an overlay may not traverse firewalls and

I don't know whether  "may not" means "might not" or "must not", and that completely determines what the sentence means. For example, does it mean this?
       - Traffic carried over an overlay might fail to traverse firewalls and
         NAT devices.

I suggest reviewing every instance of "may not" to avoid this ambiguity.

>             source and destination addresses.

Load balancing appears out of nowhere here. Are we supposed to assume that load balancing is a requirement? Load balancing between what - between different tenants, different physical DCs, different servers?

Also, there seems to be an assumption that load balancing is only based on addresses. Actually it's usually based on ports as well, and more or less by definition they are invisible to the underlay.
So it's worse than "may not work well".

I would have expected QoS support to also appear as a challenge, for similar reasons. Isn't giving tenants a fair share of the underlay capacity an issue? (There's a mention of traffic engineering later, but surely you don't want this issue to be handled by operators twiddling knobs?)

...

>        congestion avoidance mechanisms from the underlay network [FLOYD].

In a word: yuck. Surely you should be recommending against anything like that, or any attempt to re-segment TCP on the fly.

>        header encapsulation overhead.

Surely you should be recommending this, which is by far the safest solution. (And of course it should allow for the IPv6 minimum MTU.)

...
>                  statement (work in progress)

Nit: that draft was replaced a long time ago by http://tools.ietf.org/html/draft-ietf-nvo3-overlay-problem-statement
(which is already in the RFC Editor queue).

    Brian

[email-masked]

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 28 May 2014 22:50:16 +0000

Linda

From: Larry Kreeger (kreeger) [mailto:[email-masked]]
Sent: Wednesday, May 28, 2014 4:36 PM
To: Ken Gray (kegray); Linda Dunbar
Cc: [email-masked]; [email-masked]; IETF-Announce
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

I think Ken makes a good point about the charter; However, we have put text into the architecture draft addressing routing between L2 VNs.  This functionality will have an affect on the control plane requirements as well.  Should we remove it given the charter...or should the charter change to encompass expanded functionality?

 - Larry

From: "Ken Gray (kegray)" <[email-masked]<mailto:[email-masked]>>
Date: Saturday, May 24, 2014 9:57 AM
To: Linda Dunbar <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, IETF-Announce <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Ken,

Linda

Linda,

Sent from my iPhone

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 28 May 2014 21:36:10 +0000

I think Ken makes a good point about the charter; However, we have put text into the architecture draft addressing routing between L2 VNs.  This functionality will have an affect on the control plane requirements as well.  Should we remove it given the charter…or should the charter change to encompass expanded functionality?

 - Larry

Ken,

Linda

Linda,

Sent from my iPhone

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: "Black, David" <[email-masked]>
Date: Fri, 30 May 2014 08:38:00 -0400

Following up as this one:

> -       Page 13 Second bullet: the text says that each VNI is automatically generated by the egress NVE.
> Isn't each VNI supposed to match a local virtual network (e.g. represented by VLAN)? When the same NVE
> acts as Ingress NVE for the attached VMs, aren't the VNI for ingress direction statically provisioned?
> Why need automatic egress VNI generation?
>
> The value of the VN context identifier is a value locally significant to an NVE and does not have to match a local virtual network.
> It is simply used as a demultiplexer. This value can be automatically computed or provisioned.

The complete sentence from the draft is:

       - One VN Context identifier per VNI: Each VNI value is automatically
          generated by the egress NVE, or a control plane associated with
          that NVE, and usually distributed by a control plane protocol to
          all the related NVEs.

The original comment appears to have overlooked the role of the control
plane.  The VNI match to a "local virtual network" may be trivial - it
may just be mapped to a few virtual ports, so I would prefer not to
introduce an LVN concept here.  I suppose one could statically assign VNIs,
but that seems like a rather unlikely approach, and I'll note that nvo3
activity generally anticipates use of a control plane, as described in
section 3.1.5 (and subsections) of this draft.

Thanks,
--David

From: nvo3 [mailto:[email-masked]] On Behalf Of LASSERRE, MARC (MARC)
Sent: Friday, May 30, 2014 5:27 AM
To: Linda Dunbar; Larry Kreeger (kreeger); Ken Gray (kegray)
Cc: [email-masked]; [email-masked]; IETF-Announce
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Linda,

Please see my comments below.

Thanks,
Marc

________________________________
From: nvo3 [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Thursday, May 29, 2014 12:50 AM
To: Larry Kreeger (kreeger); Ken Gray (kegray)
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; IETF-Announce
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC
Hope the authors can also address those technical issues I brought up:

Section 2.3.2 describes a L3 forwarding/routing service similar to IPVPN from a service definition perspective as worded in the text of this section.

The value of the VN context identifier is a value locally significant to an NVE and does not have to match a local virtual network. It is simply used as a demultiplexer. This value can be automaticaly computed or provisioned.

LAG and STP are mentioned as examples and can be used to prevent loops in multi-homed cases. It does not mean that other solutions such as TRILL are not usable.

The text could be made more generic if needed by saying that loop avaoidance techniques must be used in multi-homing environments.

Linda

Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; IETF-Announce

 - Larry

Ken,

Linda

Linda,

Sent from my iPhone

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 30 May 2014 14:46:18 +0000

Marc,

See my comments inserted below:

From: LASSERRE, MARC (MARC) [mailto:[email-masked]]

[Linda] This section is about multi-homing, i.e. having more than one NVEs for each access domain. LAG won't work. This example shows why:

Assume an access domain is multi-homed to NVE1 and NVE2,  where NVE1 has X downstream ports, NVE2 has Y downstream ports. Say NVE1 decapsulates the data packets coming from the underlay, if NVE1 doesn't have the DA of the inner packets, NVE1 floods the traffic to its attached access domain, which will reach NVE2. NVE2 can encapsulate the packets back to the underlay, which in turn come back to NVE1.

You need to change the text to say that LAG or STP won't prevent loop in multi-home environment. However, using NVA for NVEs to drops unknown DA packets can prevent the loops. If NVEs' VAPS forward packets per IEEE802.1Q, you need to use something like TRILL's Appointed Forwarder scheme to avoid loop in multi-home environment.

Linda

Linda

 - Larry

Ken,

Linda

Linda,

Sent from my iPhone

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 30 May 2014 15:01:48 +0000

Marc,

Questions inserted below:

[Linda]If the NVE's VAPS (VNI) are connected to VMs via Ethernet where VLAN is used, it is much easier to use locally significant VLAN. Why require egress to automatically generate local VNI value?  It would be better to change the text to:

"Each VNI value on egress NVE is either mapped to locally significant Virtual Network identifier (e.g. VLAN for Ethernet), or generated by NVA or Control Plane".

Linda

nvo3 mailing list

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 30 May 2014 09:27:01 +0000

Linda,

Thanks,
Marc

Linda

 - Larry

Ken,

Linda

Linda,

Sent from my iPhone

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 30 May 2014 08:24:11 +0000

Thanks Ken.

I fully agree with you in this and your previous posts.
BTW, I don't see a need to change the existing wording at this stage...

Marc

________________________________
From: nvo3 [mailto:[email-masked]] On Behalf Of Ken Gray (kegray)
Sent: Saturday, May 24, 2014 6:57 PM
To: Linda Dunbar
Cc: [email-masked]; [email-masked]; IETF-Announce
Subject: Re: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Ken,

<!--[if !supportLists]-->-          <!--[endif]-->If a NVE performs the inter-subnet/domain routing, the NVE needs to have the needed policy. Some NVEs may have the needed policies for all VNs in DC, some NVEs may only have portion of the policies, some NVEs may not have any.

<!--[if !supportLists]-->-          <!--[endif]-->For NVEs adjacent to the "DC Gateway" in the Figure 1 of the framework draft have to maintain the mappings for all hosts/VMs in DC. For large DCs with Tens of Thousands (or hundreds of thousands) VMs, maintaining those mapping can be very intensive.

Sent from my iPhone

---

From: "LASSERRE, MARC (MARC)" <[email-masked]>
Date: Fri, 30 May 2014 08:59:17 +0000

Hi Brian,

Thanks for your comments. See my replies inline.

Marc 

> Sent: Saturday, May 24, 2014 10:46 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [nvo3] Last Call: 
> <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network 
> Virtualization) to Informational RFC
> 

Hopefully not... The intent being to reuse as many existing solutions and to describe the actual gaps to offer a scalable DC solution.

> I don't know whether  "may not" means "might not" or "must 
> not", and that completely determines what the sentence means. 
> For example, does it mean this?

Yes, it does. It will be changed accordingly.

> I suggest reviewing every instance of "may not" to avoid this 

Ok.

> Load balancing appears out of nowhere here. Are we supposed 
> to assume that load balancing is a requirement? Load 
> balancing between what - between different tenants, different 
> physical DCs, different servers?

It is mentionned as one possible challenge. I agree this sentence is a bit out of place though.
Hence, I have no problem removing it. This is actually discussed in much more details in the dataplane requirements draft.

The text implied load balancing of TS traffic by NVEs over underlay paths.

> 
> Also, there seems to be an assumption that load balancing is 
> only based on addresses. Actually it's usually based on ports 
> as well, and more or less by definition they are invisible to 
> the underlay.

The text does mention that "additional entropy information" can be used. NVEs have such visibility before sending traffic over the underlay.

> 
> I would have expected QoS support to also appear as a 
> challenge, for similar reasons. Isn't giving tenants a fair 
> share of the underlay capacity an issue? (There's a mention 
> of traffic engineering later, but surely you don't want this 
> issue to be handled by operators twiddling knobs?)

Indeed. This is discussed in section 4.2.6.

> And how will that work for non-TCP traffic?

It is the while point of this section... It is either Path MTU discovery or IP fragmentation.

> In a word: yuck. Surely you should be recommending against 
> anything like that, or any attempt to re-segment TCP on the fly.

This sentence needs some rewording as the last sentence (starting with "however") was meant to say that this is not a desirable solution.

> Surely you should be recommending this, which is by far the 
> safest solution. (And of course it should allow for the IPv6 
> minimum MTU.)

Agreed.

> (which is already in the RFC Editor queue).

Ok

---

From: Lucy yong <[email-masked]>
Date: Fri, 30 May 2014 15:51:45 +0000

Marc,
[snip]
LAG and STP are mentioned as examples and can be used to prevent loops in multi-homed cases. It does not mean that other solutions such as TRILL are not usable.
[Lucy]  in this context, it is not proper to require TS using LAG or STP to prevent loops in multi-homed case.  So examples are not proper.

The text could be made more generic if needed by saying that loop avaoidance techniques must be used in multi-homing environments.
[Lucy] Suggest changing the text as you state here. If giving an example, use EVPN or TRILL.

Lucy

Linda

 - Larry

Ken,

Linda

Linda,

Sent from my iPhone

Linda

Linda
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of The IESG
Sent: Wednesday, May 21, 2014 9:33 AM
To: IETF-Announce
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Last Call: <draft-ietf-nvo3-framework-06.txt> (Framework for DC Network Virtualization) to Informational RFC

Abstract