Subject: [nvo3] L3 Address migration in NVO3 mobility draft (now with the new name : draft-merged-nvo3-ts-address-Migration)

From: Linda Dunbar <[email-masked]>
Date: Thu, 9 Oct 2014 22:14:23 +0000

NVO3 participants:

Tom Herbert brought up that the L3 address migration should be discussed in the NVO3 mobility draft (now called NVO3-ts-address-migration).
As David Black pointed out in his email that L3 address migration is similar to L2 migration causing one subnet being spread across many.

The outbound traffic of fragmented L3 addresses doesn't have the same issue as L2 address migration, but the inbound traffic has the same issues as L2 address migration.

I added one small section to describe the L3 Address Migration. Please provide comments or suggestions.

------------------------------

L3 Address Migration
When the attachment to NVE is L3 based, TS migration can cause one subnetwork to be scatted among many NVEs, or fragmented addresses.
The outbound traffic of fragmented L3 addresses doesn't have the same issue as L2 address migration, but the inbound traffic has the same issues as L2 address migration (Section 6). In theory, host hosting at DCBR can achieve the optimal path forwarding in very fragmented network. But host routing can be challenging in a very large and highly virtualized data center, there could be hundreds of thousands of hosts/VMs, sometimes in millions, due to business demand and highly advanced server virtualization technologies.
Optimal routing of TS's inbound traffic. This means that as a given TS moves from one server to another, the (inbound) traffic originated outside of the TS's directly attached NVE, and destined to that TS be routed optimally to the NVE to which the server presently hosting that TS, without first traversing some other NVEs. This is also known as avoiding "triangular routing".
In theory, host hosting by every NVE (including the DCBR) can achieve the optimal path forwarding in very fragmented network. But host routing can be challenging in a very large and highly virtualized data center, there could be hundreds of thousands of hosts/VMs, sometimes in millions, due to business demand and highly advanced server virtualization technologies.
ECMP can be used by the DCBR or any NVEs that don't support host routing or can't access NVA to distribute traffic equally to any of the NVEs that support the subnet (VN). If an NVE doesn't have the destination of a data packet directly attached, it can query NVA for the target NVE to which the destination is attached, and encapsulate the packet with the target NVE as outer destination before sending it out.
Another approach is to designate one or two NVEs as designated forwarder for a specific subnet when the subnet is spread across many NVEs. For example, if high percentage of TSs of one subnet is attached to NVE "X", the remaining small percentage of the subnet is spread around many NVEs. Designating NVE "X" as the designated forwarder for the subnet can greatly reduce the "triangular routing" for the traffic destined to TSs in this subnet.

Linda Dunbar

---

From: Tom Herbert <[email-masked]>
Date: Fri, 10 Oct 2014 08:48:09 -0700

Hi Linda,

Thank you for adding this section to the draft. Some comments in line...

>
> L3 Address Migration
>
> When the attachment to NVE is L3 based, TS migration can cause one
> subnetwork to be scatted among many NVEs, or fragmented addresses.
>
> The outbound traffic of fragmented L3 addresses doesn’t have the same issue
> as L2 address migration, but the inbound traffic has the same issues as L2
> address migration (Section 6). In theory, host hosting at DCBR can achieve
> the optimal path forwarding in very fragmented network. But host routing can
> be challenging in a very large and highly virtualized data center, there
> could be hundreds of thousands of hosts/VMs, sometimes in millions, due to
> business demand and highly advanced server virtualization technologies.
>
> Optimal routing of TS's inbound traffic. This means that as a given TS moves
> from one server to another, the (inbound) traffic originated outside of the
> TS's directly attached NVE, and destined to that TS be routed optimally to
> the NVE to which the server presently hosting that TS, without first
> traversing some other NVEs. This is also known as avoiding "triangular
> routing".
>
Security should be considered here also. Security is strongest when
applied end-to-end. For example, if we are encrypting at the ingress
NVE, decryption should happen at the final NVE-- if the NVEs are on
host then this implies packets are encrypted during all of transit.
Also, when encrypting this may be over the full inner packet so that
virtual addresses are not visible to intermediate devices. With
ESP/GUE the header stack might be IP/UDP/GUE/ESP/IP; vnid is visible,
but virtual addresses aren't. Even if we are just using a security
cookie to validate a VNID, the outer IP addresses can be considered
for anti-spoofing. Triangular routing convolutes both of these cases I
believe.

> In theory, host hosting by every NVE (including the DCBR) can achieve the
> optimal path forwarding in very fragmented network. But host routing can be
> challenging in a very large and highly virtualized data center, there could
> be hundreds of thousands of hosts/VMs, sometimes in millions, due to
> business demand and highly advanced server virtualization technologies.
>
This is true, but there are some mitigating factors favoring host
routes. 1) a particular NVE should only be communicating with a subset
of virtual addresses at a given time. 2) Migrations are fairly rare
events and latency to adapt is already assumed to be in 10's of msecs.
An extra round trip or so to relearn a host route after migration is
not unreasonable.

So the host routes could be in a cache containing the working set
where there is a mechanism to resolve them.

> ECMP can be used by the DCBR or any NVEs that don’t support host routing or
> can’t access NVA to distribute traffic equally to any of the NVEs that
> support the subnet (VN). If an NVE doesn’t have the destination of a data
> packet directly attached, it can query NVA for the target NVE to which the
> destination is attached, and encapsulate the packet with the target NVE as
> outer destination before sending it out.
>
> Another approach is to designate one or two NVEs as designated forwarder for
> a specific subnet when the subnet is spread across many NVEs. For example,
> if high percentage of TSs of one subnet is attached to NVE “X”, the
> remaining small percentage of the subnet is spread around many NVEs.
> Designating NVE “X” as the designated forwarder for the subnet can greatly
> reduce the “triangular routing” for the traffic destined to TSs in this
> subnet.
>
I'm not sure this is practical. While there are reasons to schedule
VMs with physical locality in the DC, there are also reasons we
wouldn't do this (like we don't want a single device failure to be
able to take out all VM's of a customer). Also, I don't think we want
to constrain the job scheduler any more than it already is-- it's
likely over enough time that entropy will prevail so that VMs for
particular subnets are randomly distributed across the DC.

If we designate NVE "X" as a forwarder like you suggest, then when it
gets a packet for which there is a better route it could send the
equivalent of an ICMP redirect to back to the originator to eliminate
the triangular routing. Furthermore, if instead of acting as a
forwarder, NVE "X" is a resolver then a resolution protocol could be
implemented (ARP model) so that packets might never forwarded using
triangular routing which could address the end to end security issue I
posed above.

Thanks,
Tom

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 15 Oct 2014 21:02:54 +0000

Tom, 

I am a bit confused of your comments. See questions inserted below: 

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Tom Herbert

Security should be considered here also. Security is strongest when applied end-to-end. For example, if we are encrypting at the ingress NVE, decryption should happen at the final NVE-- if the NVEs are on host then this implies packets are encrypted during all of transit.
Also, when encrypting this may be over the full inner packet so that virtual addresses are not visible to intermediate devices. With ESP/GUE the header stack might be IP/UDP/GUE/ESP/IP; vnid is visible, but virtual addresses aren't. Even if we are just using a security cookie to validate a VNID, the outer IP addresses can be considered for anti-spoofing. Triangular routing convolutes both of these cases I believe.

[Linda] I would think the L3 address migration is orthogonal to "ingress NVE encryption & egress NVE decryption". Can you elaborate how those two issues related? 

> In theory, host hosting by every NVE (including the DCBR) can achieve 
> the optimal path forwarding in very fragmented network. But host 
> routing can be challenging in a very large and highly virtualized data 
> center, there could be hundreds of thousands of hosts/VMs, sometimes 
> in millions, due to business demand and highly advanced server virtualization technologies.
>
This is true, but there are some mitigating factors favoring host routes. 1) a particular NVE should only be communicating with a subset of virtual addresses at a given time. 2) Migrations are fairly rare events and latency to adapt is already assumed to be in 10's of msecs.
An extra round trip or so to relearn a host route after migration is not unreasonable.

So the host routes could be in a cache containing the working set where there is a mechanism to resolve them.

[Linda] I'd assume that NVA can always send update when a host is moved to a different NVE. So I am not concerned with the time taken to learn the new route. The text is intended to describe other options when the cache of a switch/router in a Data Center  can't have individual mappings for all VMs in the VNs supported (especially the gateway). 

> ECMP can be used by the DCBR or any NVEs that don’t support host 
> routing or can’t access NVA to distribute traffic equally to any of 
> the NVEs that support the subnet (VN). If an NVE doesn’t have the 
> destination of a data packet directly attached, it can query NVA for 
> the target NVE to which the destination is attached, and encapsulate 
> the packet with the target NVE as outer destination before sending it out.
>
> Another approach is to designate one or two NVEs as designated 
> forwarder for a specific subnet when the subnet is spread across many 
> NVEs. For example, if high percentage of TSs of one subnet is attached 
> to NVE “X”, the remaining small percentage of the subnet is spread around many NVEs.
> Designating NVE “X” as the designated forwarder for the subnet can 
> greatly reduce the “triangular routing” for the traffic destined to 
> TSs in this subnet.
>
I'm not sure this is practical. While there are reasons to schedule VMs with physical locality in the DC, there are also reasons we wouldn't do this (like we don't want a single device failure to be able to take out all VM's of a customer). Also, I don't think we want to constrain the job scheduler any more than it already is-- it's likely over enough time that entropy will prevail so that VMs for particular subnets are randomly distributed across the DC.

If we designate NVE "X" as a forwarder like you suggest, then when it gets a packet for which there is a better route it could send the equivalent of an ICMP redirect to back to the originator to eliminate the triangular routing. Furthermore, if instead of acting as a forwarder, NVE "X" is a resolver then a resolution protocol could be implemented (ARP model) so that packets might never forwarded using triangular routing which could address the end to end security issue I posed above.

[Linda] agree with you. I will update the text. The second option will incur triangular routing for hosts that are not attached to the designated NVE "X". This option is only intended for a scenario where host routing is not feasible on some NVEs. 

Thanks,
Tom

> Linda Dunbar

_______________________________________________
nvo3 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Tom Herbert <[email-masked]>
Date: Wed, 15 Oct 2014 17:23:46 -0700

> [Linda] I would think the L3 address migration is orthogonal to "ingress NVE encryption & egress NVE decryption". Can you elaborate how those two issues related?

I'll use a simple example to demonstrate this...

Suppose we have a customer that has requested encryption for data in
flight. So for communications between their two NVEs we need to have
an established SA between them (one SA should be sufficient between
the NVEs, we probably don't need an SA for each VN or virtual
address).

So if V1 and V2 are communicating endpoints, and E1 and E2 are the
corresponding NVEs, then we would have something like:

V1->E1->E2->V2

where communications between E1, E2 are encrypted-- V1 to E1 and E2 to
V2 are plaintext (where if they are co-resident with the respective
NVEs on a host that all communications are encrypted).

So now if we have triangular routing, then the path looks like:

V1->E1->X->E2->V2.

So the question is how to encrypt packets with this path. We can't use
an SA between E1 and E2 since E1 doesn't know that that it's peer is
E2 (if it did then we wouldn't be doing triangular routing). So in
order to encrypt end to end, it seems like we need to encrypt using
one SA between E1 and X, and another between X and E2-- but this is
not really end to end and I would assume it's a lot more load on X
than it bargained for.

> [Linda] agree with you. I will update the text. The second option will incur triangular routing for hosts that are not attached to the designated NVE "X". This option is only intended for a scenari…
It would be good if you can quantify what you mean by "host routing is
not feasible"? Are you envisioning NVEs that never use host routes but
always rely on triangular routing?

Tom

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 16 Oct 2014 16:04:30 +0000

Tom, 

Thank you very much for the example. The scenario you described is another reason that the triangular routing should be avoided.  
A few more questions:

- What kind of data centers will require NVEs to perform the encryption/decryption? 
- In those data centers, when packets are destined towards peers outside the data center, the NVE at the DC gateway will decrypt the packets, does it mean that gateway has to re-encrypt the packets before the packets leave the DC? 

- When the encryption/decryption among NVEs are required, does it mean a host (VM) can't be moved to a NVE that doesn't support the encryption/decryption? 

Thank you. Linda 

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Wednesday, October 15, 2014 7:24 PM
To: Linda Dunbar
Cc: [email-masked]
Subject: Re: [nvo3] Security issue on L3 address migration ( was RE: L3 Address migration in NVO3 mobility draft (now with the new name : draft-merged-nvo3-ts-address-Migration)

Suppose we have a customer that has requested encryption for data in flight. So for communications between their two NVEs we need to have an established SA between them (one SA should be sufficient between the NVEs, we probably don't need an SA for each VN or virtual address).

So if V1 and V2 are communicating endpoints, and E1 and E2 are the corresponding NVEs, then we would have something like:

V1->E1->E2->V2

where communications between E1, E2 are encrypted-- V1 to E1 and E2 to
V2 are plaintext (where if they are co-resident with the respective NVEs on a host that all communications are encrypted).

V1->E1->X->E2->V2.

So the question is how to encrypt packets with this path. We can't use an SA between E1 and E2 since E1 doesn't know that that it's peer is
E2 (if it did then we wouldn't be doing triangular routing). So in order to encrypt end to end, it seems like we need to encrypt using one SA between E1 and X, and another between X and E2-- but this is not really end to end and I would assume it's a lot more load on X than it bargained for.

>> routing can be challenging in a very large and highly virtualized 
>> data center, there could be hundreds of thousands of hosts/VMs, 
>> sometimes in millions, due to business demand and highly advanced server virtualization technologies.
>>
>> NVEs. For example, if high percentage of TSs of one subnet is 
>> attached to NVE “X”, the remaining small percentage of the subnet is spread around many NVEs.
It would be good if you can quantify what you mean by "host routing is not feasible"? Are you envisioning NVEs that never use host routes but always rely on triangular routing?

Tom

---

From: Tom Herbert <[email-masked]>
Date: Thu, 16 Oct 2014 11:59:32 -0700

> - What kind of data centers will require NVEs to perform the encryption/decryption?

IMO, the simplest answer is all kinds. This is really independent of
NVEs or network virtualization, it's a question of securing data
centers. We already see all data encrypted at rest in many DCs (ie. on
disk); it is not too much of an extrapolation to think that things
will want to move towards ubiquitous encryption within DCs. Even now
it's a pretty good bet that most DCs will need to encrypt all inter DC
traffic which is not an insignificant portion of traffic (can be 10%
or more of traffic). The motivation for more encryption is pretty
clear, there are real risks, the potential cost of a breach is only
increasing, there is a lot of untrusted third party code and stack (in
VMs). For cloud customers, we want to provide strong SLAs around
security. The reasons we don't have ubiquitous encryption are cost and
complexity which I hope will be solved in time. I have a slide deck
I've shared with a few NIC vendors on the opportunity implementing
line rate crypto for DC if you're interested.

> - In those data centers, when packets are destined towards peers outside the data center, the NVE at the DC gateway will decrypt the packets, does it mean that gateway has to re-encrypt the packets…
Maybe, but it's probably more likely to see encrypted tunnels between
DCs which can be independent of NV.

> - When the encryption/decryption among NVEs are required, does it mean a host (VM) can't be moved to a NVE that doesn't support the encryption/decryption?
No, encryption is part of the underlay network for security between
hosts. Technically, you might only need to do it if the path is not
secure (like inter DC), but in practice to enforce source routing and
the possibility of misdirected routes is still non-zero so determining
that a priori is hard.

The other consideration is the service we are offering a tenant. A
paranoid customer may want strong security SLA. If this means
encrypting all their packets NVE->NVE, then we we couldn't move their
VM behind an NVE that doesn't provide encryption. I would view this as
a scheduling constraint more than anything.

Tom