Subject: Re: [nvo3] "Enhancing Virtual Network Encapsulation with IPv6"

From: Tom Herbert <[email-masked]>
Date: Fri, 5 Sep 2014 14:55:23 -0700

On Sat, Aug 16, 2014 at 3:18 AM, Mark ZZZ Smith
<[email-masked]> wrote:
> Hi,
>
> I've put together some thoughts on how some of IPv6's capabilities may be used to enhance network virtualisation encapsulation. I'd be interested to know what people think.
>

Thanks for the draft! Here are some comments:

Carrying the Virtual Network Context ID in the Flow Label Field:

I'm not sure how practical this is. There's already deployment to use
the flow label as representative of inner flow (like for ECMP hashing,
etc.). If we split this field too much, we would start to lose entropy
for the hash (I would like to think we have at least 14 bits worth of
entropy here). For holding the whole Virtual Network ID in the flow
label, 20 bits is really limiting for scaling (I still think we need
32 bits), and partitioning the network into islands with overlapping
number spaces is an extremely unpleasant thought.

Carrying Tenant Packet Address and Other Information in IIDs:

Encoding virtual network IDs and tenant address into IPv6 addresses is
potentially very interesting. We have considered identifier/locator
(ILNP like) addressing with 64 bits for locator, 32 bits for virtual
network identifier, and 32 bit tenant address. With this we don't need
any encapsulation header, so the fact that we're doing NV is
transparent on the wire. This means all the offloads, and networking
optimizations for IPv6, TCP/IPv6, etc. will work without change.
Mobility is easy, just change identifier to locator mapping. Downside
is that we may need to do a lot NAT, and we still might need
additional header to ensure integrity/authenticity of virtual
networking identifier (it is nice that the addresses, including the
VNI, are covered in the pseudo header for the normal transport
checksum).

Tom

> Thanks very much,
> Mark.
>
>
>
>
> ----- Forwarded Message -----
>> From: "[email-masked]" <[email-masked]>
>> To: Mark Smith <[email-masked]>; Mark Smith <[email-masked]>
>> Cc:
>> Sent: Saturday, 16 August 2014 8:10 PM
>> Subject: New Version Notification for draft-smith-enhance-vne-with-ipv6-04.txt
>>
>>
>> A new version of I-D, draft-smith-enhance-vne-with-ipv6-04.txt
>> has been successfully submitted by Mark Smith and posted to the
>> IETF repository.
>>
>> Name:        draft-smith-enhance-vne-with-ipv6
>> Revision:    04
>> Title:        Enhancing Virtual Network Encapsulation with IPv6
>> Document date:    2014-08-16
>> Group:        Individual Submission
>> Pages:        17
>> URL:
>> http://www.ietf.org/internet-drafts/draft-smith-enhance-vne-with-ipv6-04.txt
>> Status:
>> https://datatracker.ietf.org/doc/draft-smith-enhance-vne-with-ipv6/
>> Htmlized:      http://tools.ietf.org/html/draft-smith-enhance-vne-with-ipv6-04
>> Diff:
>> http://www.ietf.org/rfcdiff?url2=draft-smith-enhance-vne-with-ipv6-04
>>
>> Abstract:
>>    A variety of network virtualization over layer 3 methods are
>>    currently being developed and deployed.  These methods treat IPv4 and
>>    IPv6 as equivalent underlay network technologies.  This memo suggests
>>    how IPv6's additional capabilities may be used to enhance Virtual
>>    Network encapsulation over an IPv6 Underlay Network.
>>
>>
>>
>>
>>
>> Please note that it may take a couple of minutes from the time of submission
>> until the htmlized version and diff are available at tools.ietf.org.
>>
>> The IETF Secretariat
>>
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Mark ZZZ Smith <[email-masked]>
Date: Sun, 7 Sep 2014 23:31:03 -0700

Hi Tom,

----- Original Message -----
> From: Tom Herbert <[email-masked]>
> To: Mark ZZZ Smith <[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> Sent: Saturday, 6 September 2014, 7:55
> Subject: Re: [nvo3] "Enhancing Virtual Network Encapsulation with IPv6"
> 
>>  I've put together some thoughts on how some of IPv6's capabilities 
> may be used to enhance network virtualisation encapsulation. I'd be 
> interested to know what people think.
>> 
> 

Thanks for your time reading it and your comments.

> etc.).

Is this use in the flow label fields of the outer IPv6 packets when they're the encapsulation protocol between NVEs in the underlay network?

I may be misunderstanding your comment, however just to be clear, the IPv6 tenant system to tenant system traffic within a virtual network can use their (inner) IPv6 flow label however they like. My draft is only about using the flow label field in the outer IPv6 header used to encapsulate/tunnel across the underlay network between the NVEs.

> If we split this field too much, we would start to lose entropy
> entropy here)
> For holding the whole Virtual Network ID in the flow

While I think it would be useful if the flow label was a bit bigger, I thought it would be unlikely that there would be any instances of IPv6 underlay networks that would need to support more than in the order of 1 million virtual networks.

I appreciate who you work for, does that mean Google might have use cases where more than a million virtual networks need to be supported over a single underlay network?  

> optimizations for IPv6, TCP/IPv6, etc. will work without change.

Agree. I looked at some of the measures in other encapsulations such as VXLAN (e.g., UDP header added to provide load balancing entropy), or NVGRE encoding of flow information in the GRE key sequence field (which would need new hardware/firmware to look at it for load balancing), and realised that IPv6 either already provided those mechanisms (e.g., flow label with the right contents/values) or they could be achieved by creatively using the IID field in the outer IPv6 addresses if NVEs were identified using /64s rather than /128s in the IPv6 underlay network.

I'm not quite sure about not needing any VN specific encapsulation header with what I've suggested. If the outer IPv6 flow label was used to carry the VN Context ID, then there would need to be some other header carrying a checksum to protect it, as the IPv6 header doesn't have a checksum to protect it. There would also be a need for a 'tenant packet type' field somewhere in a VN specific encapsulation header if more than just a single tenant packet type is to be supported.

I presented last week at Ausnog 2014 on what I've suggested in the ID, here are the slides if they'd be of interest: 

"Network Virtualisation: The Killer App for IPv6?"
http://www.users.on.net/~markachy/nvtkaipv6.pdf

Best regards,
Mark.

> checksum)

>>>  Htmlized:      
> http://tools.ietf.org/html/draft-smith-enhance-vne-with-ipv6-04
>>>  Diff:

---

From: Tom Herbert <[email-masked]>
Date: Mon, 8 Sep 2014 08:05:06 -0700

> I may be misunderstanding your comment, however just to be clear, the IPv6 tenant system to tenant system traffic within a virtual network can use their (inner) IPv6 flow label however they like. M…
Yes, I am referring to the outer flow label. In encapsulation we can
set this to the hash in the of the inner four tuple to route in the
network based the inner flow. This is similar to how we can set the
source port of UDP outer header to correspond to the inner flow (see
various UDP encap proposals).

> While I think it would be useful if the flow label was a bit bigger, I thought it would be unlikely that there would be any instances of IPv6 underlay networks that would need to support more than…
In a large networks it is likely that the virtual network ID space
will have structure, hierarchical assignment, special bits, etc. so
fragmentation is very possible.

> I appreciate who you work for, does that mean Google might have use cases where more than a million virtual networks need to be supported over a single underlay network?
Yes. Even if we didn't today, I wouldn't want to deploy something that
could limit our growth in the future (say for at least for ten years).
We've already learned this lesson in the IPv4 address space!

> I'm not quite sure about not needing any VN specific encapsulation header with what I've suggested. If the outer IPv6 flow label was used to carry the VN Context ID, then there would need to be some other header carrying a checksum to protect it, as the IPv6 header doesn't have a checksum to protect it.

Right, this is why I'd prefer to have the VNI in the address so that
it would be included in the pseudo header csum for UDP and TCP. The
use case in DC where no encap header could be relevant is when we are
converting internal communications (e.g. non-third party traffic) to
use VNs. Encapsulating in IPv6 should be no less secure, and using a
native packet format might be critical in maintaining performance
versus non-VN.

> There would also be a need for a 'tenant packet type' field somewhere in a VN specific encapsulation header if more than just a single tenant packet type is to be supported.
>
Wouldn't next-header be sufficient?

Tom