Subject: Re: [nvo3] WG Last Call for draft-ietf-nvo3-overlay-problem-statement-02

From: Benson Schliesser <[email-masked]>
Date: Fri, 1 Mar 2013 18:29:48 -0500

Hi, Linda.

I think the NVO3 Problem Statement is intended to describe the problems
that motivate our work, rather than repercussions of a particular
approach. Speaking for myself, I observe that these issues are very much
solution-dependent, and I think they are more appropriate inputs for the
requirements drafts rather than the PS. As chair, I'd be interested in
hearing what the authors and other contributors think about this.

Cheers,
-Benson

On Mar 1, 2013, at 18:10, "Linda Dunbar" <[email-masked]> wrote:

  Thomas, et al,

This draft has done a very good job in describing the motivation for
overlay and its potential control/data plane.

However, the draft hasn’t described the issues introduced by overlay and
large DC issues not solved by overlay. For example:

-        *Multicast issues introduced by overlay: *

o   IP Encapsulation by NVE (or virtual switches on server)  means that any
layer 2 multicast  from or to VMs requires Layer 3 Multicast/broadcast
support in the core.

o   Majority of  virtual switches  in server won’t have attached VMs
participating in any Multicast groups. Therefore, it may not be cost
effective for them to support Layer 3 multicast protocol, e.g. PIM, etc.

o   For virtual switches that do have attached VMs participating in
multicast, there are much more dynamic states due to VM moves.

o   The head end replication will require hypervisor virtual switches to
keep up multicast member states,  which  can change frequently. Therefore,
those NVEs, if required to support multicast,  have to do more than MPLS
(MVPN) multicast supported by PE routers.

o

-        *Bottleneck at gateway nodes:*

o   The draft has emphasized greatly on the importance of tenant
separation. That means two VMs under the same NVE, if they belong to two
different virtual network instances, the communication between them have to
go all the way to their default gateway.

o   Some end stations even have specific default gateway configured. That
require all the inter VNI communication to go through their designated
gateway nodes.

o   Many data centers have their FW/LB co- located with GW nodes. That
means all the inter VNI communications have to hairpined to those GW nodes.

-        *issues that are not solved by overlay:*

o   The draft is for environment where VMs can move anywhere, which require
the Gateway nodes to be aware of individual VM addresses and their
corresponding egress NVEs.

o   Many large Data Centers have small number of gateway nodes interfacing
with external network, e.g. 2/4/8/more. Very often, those external facing
gateway nodes are the entrance and exit points for all virtual network
instances.

o   Very often each of them can receive traffic for all virtual network
instances hosted in the data center.

o   That means they all need to be aware of individual VMs and their
corresponding egress NVEs in the data center. i.e. they may all need host
routing. It is true that many routers can handle millions of routes, but
the question is if it is necessary for data center gateway nodes to be
those high capacity routers.

-

*Therefore, I think that the NV03 problem statement draft should add a new
section to describes issues introduced by overlay and issues not solved by
overlay. *

Linda Dunbar

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 1 Mar 2013 23:36:25 +0000

Benson,

I only point out some Data Center issues not solved by overlay and issues introduced by IP overlay. I think that NV03 as an IETF working group for data centers with large number of mobile virtual machines, the problem statement should have a section describing those points.

I really don't see how those issues are solution based. Can you elaborate why do you see those are solution based?

Best regards,
Linda

From: Benson Schliesser [mailto:[email-masked]]
Sent: Friday, March 01, 2013 5:30 PM
To: Linda Dunbar; [email-masked]
Subject: Re: [nvo3] WG Last Call for draft-ietf-nvo3-overlay-problem-statement-02

Hi, Linda.

I think the NVO3 Problem Statement is intended to describe the problems that motivate our work, rather than repercussions of a particular approach. Speaking for myself, I observe that these issues are very much solution-dependent, and I think they are more appropriate inputs for the requirements drafts rather than the PS. As chair, I'd be interested in hearing what the authors and other contributors think about this.

Cheers,
-Benson

On Mar 1, 2013, at 18:10, "Linda Dunbar" <[email-masked]<mailto:[email-masked]>> wrote:
Thomas, et al,

This draft has done a very good job in describing the motivation for overlay and its potential control/data plane.

However, the draft hasn't described the issues introduced by overlay and large DC issues not solved by overlay. For example:

-        Multicast issues introduced by overlay:

o   IP Encapsulation by NVE (or virtual switches on server)  means that any layer 2 multicast  from or to VMs requires Layer 3 Multicast/broadcast support in the core.

o   Majority of  virtual switches  in server won't have attached VMs participating in any Multicast groups. Therefore, it may not be cost effective for them to support Layer 3 multicast protocol, e.g. PIM, etc.

o   For virtual switches that do have attached VMs participating in multicast, there are much more dynamic states due to VM moves.

o   The head end replication will require hypervisor virtual switches to keep up multicast member states,  which  can change frequently. Therefore, those NVEs, if required to support multicast,  have to do more than MPLS (MVPN) multicast supported by PE routers.

o

-        Bottleneck at gateway nodes:

o   The draft has emphasized greatly on the importance of tenant separation. That means two VMs under the same NVE, if they belong to two different virtual network instances, the communication between them have to go all the way to their default gateway.

o   Some end stations even have specific default gateway configured. That require all the inter VNI communication to go through their designated gateway nodes.

o   Many data centers have their FW/LB co- located with GW nodes. That means all the inter VNI communications have to hairpined to those GW nodes.

-        issues that are not solved by overlay:

o   The draft is for environment where VMs can move anywhere, which require the Gateway nodes to be aware of individual VM addresses and their corresponding egress NVEs.

o   Many large Data Centers have small number of gateway nodes interfacing with external network, e.g. 2/4/8/more. Very often, those external facing gateway nodes are the entrance and exit points for all virtual network instances.

o   Very often each of them can receive traffic for all virtual network instances hosted in the data center.

o   That means they all need to be aware of individual VMs and their corresponding egress NVEs in the data center. i.e. they may all need host routing. It is true that many routers can handle millions of routes, but the question is if it is necessary for data center gateway nodes to be those high capacity routers.

-
Therefore, I think that the NV03 problem statement draft should add a new section to describes issues introduced by overlay and issues not solved by overlay.

Linda Dunbar

---

From: Benson Schliesser <[email-masked]>
Date: Fri, 1 Mar 2013 20:02:46 -0500

Hi, Linda.

To clarify: I described them as solution based because they might not be
present in all different solutions.

For example, a number of your issues stem from the gateway's location. But
some approaches might have distributed gateways. An L3 service in the NVE
might not experience any of these gateway issues, for example, depending on
the network design.

Nevertheless, these are definitely things to consider. Thus my suggestion -
I think it would be helpful if these issues were rephrased as requirements.

Cheers,
-Benson

On Mar 1, 2013, at 18:36, Linda Dunbar <[email-masked]> wrote:

  Benson,

I only point out some Data Center issues not solved by overlay and issues
introduced by IP overlay. I think that NV03 as an IETF working group for
data centers with large number of mobile virtual machines, the problem
statement should have a section describing those points.

I really don’t see how those issues are solution based. Can you elaborate
why do you see those are solution based?

Best regards,

Linda

*From:* Benson Schliesser [mailto:[email-masked]<[email-masked]>]

*Sent:* Friday, March 01, 2013 5:30 PM
*To:* Linda Dunbar; [email-masked]
*Subject:* Re: [nvo3] WG Last Call for
draft-ietf-nvo3-overlay-problem-statement-02

Hi, Linda.

Cheers,

-Benson

 Thomas, et al,

o

-

Linda Dunbar

---

From: Linda Dunbar <[email-masked]>
Date: Sat, 2 Mar 2013 03:10:56 +0000

Benson,

The generic architecture described in Figure 1 of the draft-ietf-nvo3-framework document shows that all end devices are under a group of DC-GWs:

A generic architecture for Data Centers is depicted in Figure 1:

                                    ,---------.
                                  ,'           `.
                                 (  IP/MPLS WAN )
                                  `.           ,'
                                    `-+------+'
                                 +--+--+   +-+---+
                                 |DC GW|+-+|DC GW|
                                 +-+---+   +-----+
                                    |       /
                                    .--. .--.
                                  (    '    '.--.
                                .-.' Intra-DC     '
                               (     network      )
                                (             .'-'
                                 '--'._.'.    )\ \
                                 / /     '--'  \ \
                                / /      | |    \ \
                          +---+--+   +-`.+--+  +--+----+
                          | ToR  |   | ToR  |  |  ToR  |
                          +-+--`.+   +-+-`.-+  +-+--+--+
                           /     \    /    \   /       \
                        __/_      \  /      \ /_       _\__
                 '--------'   '--------'   '--------'   '--------'
                 :  End   :   :  End   :   :  End   :   :  End   :
                 : Device :   : Device :   : Device :   : Device :
                 '--------'   '--------'   '--------'   '--------'

                 Figure 1 : A Generic Architecture for Data Centers

Can each of the GWs reach all End Devices? If yes, that means every GW needs to be aware of corresponding egress NVE for each end device, which is the GW bottleneck issue described in my email.

If each GW only reaches a subset of VNIs,   GW routers and their uplink (WAN) ports are not fully utilized. WAN ports are expensive.

The NV03 problem statement has detailed description of options of control plane.  IMHO, it is more important for the problem statement to address the pros and cons of central GW vs. distributed GW.

In addition, the IP encapsulation push all the L2 multicast to L3. IGMP/MLD snooping is very efficient in pruning  multicast distribution in Layer 2 and they are supported by very cheap switches. But L3 multicast is a different story. Those are all valid issues in having overlay networks in data center. I don't see why/how problem statement can avoid them.

Thanks, Linda

From: Benson Schliesser [mailto:[email-masked]]
Sent: Friday, March 01, 2013 7:03 PM
To: Linda Dunbar
Cc: [email-masked]
Subject: Re: [nvo3] WG Last Call for draft-ietf-nvo3-overlay-problem-statement-02

Hi, Linda.

To clarify: I described them as solution based because they might not be present in all different solutions.

For example, a number of your issues stem from the gateway's location. But some approaches might have distributed gateways. An L3 service in the NVE might not experience any of these gateway issues, for example, depending on the network design.

Nevertheless, these are definitely things to consider. Thus my suggestion - I think it would be helpful if these issues were rephrased as requirements.

Cheers,
-Benson

On Mar 1, 2013, at 18:36, Linda Dunbar <[email-masked]<mailto:[email-masked]>> wrote:
Benson,

Best regards,
Linda

From: Benson Schliesser [mailto:[email-masked]]
Sent: Friday, March 01, 2013 5:30 PM
To: Linda Dunbar; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] WG Last Call for draft-ietf-nvo3-overlay-problem-statement-02

Hi, Linda.

Cheers,
-Benson

o

-

Linda Dunbar

---

From: Benson Schliesser <[email-masked]>
Date: Sat, 02 Mar 2013 10:49:16 -0800

Linda, I understand the issues you're describing. But my point remains: 
they're effects of how any given solution works. There may be trade-offs 
between a solution's architectural choices (e.g. aggregation, 
delegation, and load balancing of various control and/or data plane 
functions) that change these properties.

Therefore, I don't think they're in scope of the Problem Statement. The 
PS is intended to capture our motivation etc and not architecture or 
requirements.

Cheers,
-Benson

On 3/1/13 7:10 PM, Linda Dunbar wrote:
>
> Benson,
>
> The generic architecture described in Figure 1 of the 
> draft-ietf-nvo3-framework document shows that all end devices are 
> under a group of DC-GWs:
>
> Can each of the GWs reach all End Devices? If yes, that means every GW 
> needs to be aware of corresponding egress NVE for each end device, 
> which is the GW bottleneck issue described in my email.
>
> If each GW only reaches a subset of VNIs,   GW routers and their 
> uplink (WAN) ports are not fully utilized. WAN ports are expensive.
>
> The NV03 problem statement has detailed description of options of 
> control plane.  IMHO, it is more important for the problem statement 
> to address the pros and cons of central GW vs. distributed GW.
>
> In addition, the IP encapsulation push all the L2 multicast to L3. 
> IGMP/MLD snooping is very efficient in pruning  multicast distribution 
> in Layer 2 and they are supported by very cheap switches. But L3 
> multicast is a different story. Those are all valid issues in having 
> overlay networks in data center. I don't see why/how problem statement 
> can avoid them.
>
> Thanks, Linda
>
> *From:*Benson Schliesser [mailto:[email-masked]]
> *Sent:* Friday, March 01, 2013 7:03 PM
> *To:* Linda Dunbar
> *Cc:* [email-masked]
> For example, a number of your issues stem from the gateway's location. 
> But some approaches might have distributed gateways. An L3 service in 
> the NVE might not experience any of these gateway issues, for example, 
> depending on the network design.
>
> Nevertheless, these are definitely things to consider. Thus my 
> suggestion - I think it would be helpful if these issues were 
> rephrased as requirements.
>
> Cheers,
>
> -Benson
>
>
>     I only point out some Data Center issues not solved by overlay and
>     issues introduced by IP overlay. I think that NV03 as an IETF
>     working group for data centers with large number of mobile virtual
>     machines, the problem statement should have a section describing
>     those points.
>
>     I really don't see how those issues are solution based. Can you
>     elaborate why do you see those are solution based?
>
>     Best regards,
>
>     Linda
>
>     *From:*Benson Schliesser [mailto:[email-masked]]
>     *To:* Linda Dunbar; [email-masked] <mailto:[email-masked]>
>     I think the NVO3 Problem Statement is intended to describe the
>     problems that motivate our work, rather than repercussions of a
>     particular approach. Speaking for myself, I observe that these
>     issues are very much solution-dependent, and I think they are more
>     appropriate inputs for the requirements drafts rather than the
>     PS. As chair, I'd be interested in hearing what the authors and
>     other contributors think about this.
>
>     Cheers,
>
>     -Benson
>
>
>         This draft has done a very good job in describing the
>         motivation for overlay and its potential control/data plane.
>
>         However, the draft hasn't described the issues introduced by
>         overlay and large DC issues not solved by overlay. For example:
>
>         -*Multicast issues introduced by overlay: *
>
>         oIP Encapsulation by NVE (or virtual switches on server)
>          means that any layer 2 multicast  from or to VMs requires
>         Layer 3 Multicast/broadcast support in the core.
>
>         oMajority of  virtual switches  in server won't have attached
>         VMs participating in any Multicast groups. Therefore, it may
>         not be cost effective for them to support Layer 3 multicast
>         protocol, e.g. PIM, etc.
>
>         oFor virtual switches that do have attached VMs participating
>         in multicast, there are much more dynamic states due to VM moves.
>
>         oThe head end replication will require hypervisor virtual
>         switches to keep up multicast member states,  which  can
>         change frequently. Therefore, those NVEs, if required to
>         support multicast,  have to do more than MPLS (MVPN) multicast
>         supported by PE routers.
>
>         o
>
>         -*Bottleneck at gateway nodes:*
>
>         oThe draft has emphasized greatly on the importance of tenant
>         separation. That means two VMs under the same NVE, if they
>         belong to two different virtual network instances, the
>         communication between them have to go all the way to their
>         default gateway.
>
>         oSome end stations even have specific default gateway
>         configured. That require all the inter VNI communication to go
>         through their designated gateway nodes.
>
>         oMany data centers have their FW/LB co- located with GW nodes.
>         That means all the inter VNI communications have to hairpined
>         to those GW nodes.
>
>         -*issues that are not solved by overlay:*
>
>         oThe draft is for environment where VMs can move anywhere,
>         which require the Gateway nodes to be aware of individual VM
>         addresses and their corresponding egress NVEs.
>
>         oMany large Data Centers have small number of gateway nodes
>         interfacing with external network, e.g. 2/4/8/more. Very
>         often, those external facing gateway nodes are the entrance
>         and exit points for all virtual network instances.
>
>         oVery often each of them can receive traffic for all virtual
>         network instances hosted in the data center.
>
>         oThat means they all need to be aware of individual VMs and
>         their corresponding egress NVEs in the data center. i.e. they
>         may all need host routing. It is true that many routers can
>         handle millions of routes, but the question is if it is
>         necessary for data center gateway nodes to be those high
>         capacity routers.
>
>         -
>
>         *Therefore, I think that the NV03 problem statement draft
>         should add a new section to describes issues introduced by
>         overlay and issues not solved by overlay. *
>
>         Linda Dunbar
>

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 4 Mar 2013 15:06:20 +0000

Benson,

Does it mean that the framework document or the data plane requirement should include the discussion on those issues?

Linda

From: Benson Schliesser [mailto:[email-masked]]
Sent: Saturday, March 02, 2013 12:49 PM
To: Linda Dunbar
Cc: [email-masked]
Subject: Re: [nvo3] WG Last Call for draft-ietf-nvo3-overlay-problem-statement-02

Linda, I understand the issues you're describing. But my point remains: they're effects of how any given solution works. There may be trade-offs between a solution's architectural choices (e.g. aggregation, delegation, and load balancing of various control and/or data plane functions) that change these properties.

Therefore, I don't think they're in scope of the Problem Statement. The PS is intended to capture our motivation etc and not architecture or requirements.

Cheers,
-Benson

                                    ,---------.
                                  ,'           `.
                                 (  IP/MPLS WAN )
                                  `.           ,'
                                    `-+------+'
                                 +--+--+   +-+---+
                                 |DC GW|+-+|DC GW|
                                 +-+---+   +-----+
                                    |       /
                                    .--. .--.
                                  (    '    '.--.
                                .-.' Intra-DC     '
                               (     network      )
                                (             .'-'
                                 '--'._.'.    )\ \
                                 / /     '--'  \ \
                                / /      | |    \ \

Thanks, Linda

From: Benson Schliesser [mailto:[email-masked]]
Sent: Friday, March 01, 2013 7:03 PM
To: Linda Dunbar
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] WG Last Call for draft-ietf-nvo3-overlay-problem-statement-02

Hi, Linda.

Cheers,
-Benson

Best regards,
Linda

Hi, Linda.

Cheers,
-Benson

o

-

Linda Dunbar

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 4 Mar 2013 15:13:18 +0000

Benson,

How about multicast issues associated with IP encapsulation? They are definitely not solution or any design issue.

-        Multicast issues associated with IP overlay:

Linda

Cheers,
-Benson

                                    ,---------.
                                  ,'           `.
                                 (  IP/MPLS WAN )
                                  `.           ,'
                                    `-+------+'
                                 +--+--+   +-+---+
                                 |DC GW|+-+|DC GW|
                                 +-+---+   +-----+
                                    |       /
                                    .--. .--.
                                  (    '    '.--.
                                .-.' Intra-DC     '
                               (     network      )
                                (             .'-'
                                 '--'._.'.    )\ \
                                 / /     '--'  \ \
                                / /      | |    \ \

Thanks, Linda

Hi, Linda.

Cheers,
-Benson

Best regards,
Linda

Hi, Linda.

Cheers,
-Benson

o

-

Linda Dunbar

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 4 Mar 2013 21:01:01 +0000

Benson,

NV03's charter states:
NVO3 will document the problem statement, the applicability,
and an architectural framework for DCVPNs within a data center
environment

One of the key differentiators that set  NV03's overlay apart from other IETF defined overlay (such as L2VPN, TRILL, LISP, etc) is that NV03 is for Data Center environment.

For Data Center, in addition to intra-virtual network communication, another big part is Inter Virtual Network Communication. The end stations (virtual or physical) may even have their default gateway specified.

For NVEs that can do L3 routing, if they are different from the specified "default gateway" of end stations, or if they don't have the capability (such as the needed firewall function for the VN), they can't simply route traffic among multiple Virtual Networks.

However, the current Problem statement only has one paragraph describing communication between Virtual Networks.

I think that it is necessary to add at least one more paragraph to describe the inter virtual network communication:

Some end stations might have specific default gateway configured. Under those environment,  the communication between two end stations in different Virtual Networks under one NVE might need to be hairpined to their corresponding Gateway nodes, which is not bandwidth efficient. This is especially true when the NVE to which the end stations are attached is different from the "default gateway" specified by the end stations.

Linda

Cheers,
-Benson

                                    ,---------.
                                  ,'           `.
                                 (  IP/MPLS WAN )
                                  `.           ,'
                                    `-+------+'
                                 +--+--+   +-+---+
                                 |DC GW|+-+|DC GW|
                                 +-+---+   +-----+
                                    |       /
                                    .--. .--.
                                  (    '    '.--.
                                .-.' Intra-DC     '
                               (     network      )
                                (             .'-'
                                 '--'._.'.    )\ \
                                 / /     '--'  \ \
                                / /      | |    \ \

Thanks, Linda

Hi, Linda.

Cheers,
-Benson

Best regards,
Linda

Hi, Linda.

Cheers,
-Benson

o

-

Linda Dunbar

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 4 Mar 2013 14:04:40 -0800

> One of the key differentiators that set  NV03’s overlay apart from other IETF defined overlay (such as L2VPN, TRILL, LISP, etc) is that NV03 is for Data Center environment.

Just an FYI that LISP has been running in data centers for a couple of years.

Dino

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Tue, 5 Mar 2013 01:59:35 +0000

Hi Linda,

I tend to agree with Benson here.  I think the potential issues that Linda raises can only be said to be issues if you consider a specific solution or data center architecture.  Depending on the choices made, there may be tradeoffs that will expose one or more of the issues raised.  Perhaps these belong in the Gap Analysis since the GA analyzes solutions.

 - Larry

From: Benson Schliesser <[email-masked]<mailto:[email-masked]>>
Date: Friday, March 1, 2013 5:02 PM
To: Linda Dunbar <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] WG Last Call for draft-ietf-nvo3-overlay-problem-statement-02

Hi, Linda.

Cheers,
-Benson

Benson,

I really don’t see how those issues are solution based. Can you elaborate why do you see those are solution based?

Best regards,
Linda

Hi, Linda.

Cheers,
-Benson

However, the draft hasn’t described the issues introduced by overlay and large DC issues not solved by overlay. For example:

o   Majority of  virtual switches  in server won’t have attached VMs participating in any Multicast groups. Therefore, it may not be cost effective for them to support Layer 3 multicast protocol, e.g. PIM, etc.

o

-

Linda Dunbar

---

From: Thomas D Nadeau <[email-masked]>
Date: Tue, 5 Mar 2013 07:37:35 -0500

I agree with both Larry and Benson; these things belong in the gap analysis. Lets keep the problem statement focused so we can wrap this thing up.

	--Tom

> From: Benson Schliesser <[email-masked]>
> To: Linda Dunbar <[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3