Subject: Re: [nvo3] Encapsulation considerations

From: Lucy yong <[email-masked]>
Date: Wed, 1 Apr 2015 15:26:40 +0000

Hi Eric,

A "service" might be too generic and cause ambiguous. It means an encapsulation for non-tunneling purpose. Thus, one type of encapsulation is for tunneling purpose, which is transport related; another is for non-tunneling purpose, which is non transport related.

The purpose of the draft is to provide guidelines for encapsulation design. IMO, it is good to point out two and make clear what issues that each type needs to consider in the draft. In fact, such thought is even important at architecture design level.

Regards,
Lucy

-----Original Message-----
From: Erik Nordmark [mailto:[email-masked]] 
Sent: Tuesday, March 31, 2015 11:58 PM
To: Lucy yong; Erik Nordmark; [email-masked]
Subject: Re: [nvo3] Encapsulation considerations

On 3/30/15 1:16 PM, Lucy yong wrote:
> Hi Eric,
>
> Here is my thought.
>
> Congestion considerations, Header Protection, Entropy, MTU and Fragmentation, and QoS are specific for an encapsulation for tunneling.
>
> Next-protocol, OAM, extensibility, Security, layering, and middle-box are applied to both types of encapsulations.
>
> Service model may only apply to the encapsulation for a service.

Lucy,

I'm not sure I have a crisp definition of a "service" vs. "tunneling".

But looking at your examples I think there are less differences than you claim.
For instance, if a "service" adds a header of some sort it needs to consider the MTU/frag implications of that header. And it needs to consider whether it needs to protect the header against bit errors (that are undetected by the link layer).

Furthermore, if the "service" can be applied to non-congestion controlled traffic, e.g., arbitrary Ethernet payload, then congestion needs to be considered.

So if we keep in mind that these are considerations and not requirements, then I don't think we need to try to find some way to separate them out. Upon considering something for e.g., SFC, it might turn out that no action is needed.

Thanks,
    Erik

>
> Regards,
> Lucy
>
>
>
>
>
>
> -----Original Message-----
> From: Erik Nordmark [mailto:[email-masked]]
> Sent: Saturday, March 28, 2015 12:32 AM
> To: Lucy yong; [email-masked]
> Subject: Re: [nvo3] Encapsulation considerations
>
> On 3/25/15 2:23 PM, Lucy yong wrote:
>> Here is a suggestion to the draft.
>>
>> There are two distinct encapsulation purposes.
>> 1) an encapsulation for tunneling purpose, i.e. transport related encapsulation, e.g. nvo3.
>> 2) an encapsulation for a service, i.e. transport independent encapsulation, e.g. sfc.
>>
>> Considerations for two types of encapsulations have difference. It is good for the draft to point out that and give separate considerations.
> Lucy,
>
> which considerations in the draft are different for the two types you suggest?
>
> Thanks,
>      Erik
>
>> Thanks,
>> Lucy
>>
>>
>> -----Original Message-----
>> From: nvo3 [mailto:[email-masked]] On Behalf Of Erik Nordmark
>> Sent: Wednesday, March 25, 2015 4:01 PM
>> To: [email-masked]
>> Subject: [nvo3] Encapsulation considerations
>>
>>
>> I presented part of this at the most recent NVO3 interim meeting.The full 12 areas of considerations where presented at RTGWG earlier this week.
>>     The draft is
>>       http://datatracker.ietf.org/doc/draft-rtg-dt-encap/
>>     and the slides are at
>>      http://www.ietf.org/proceedings/92/slides/slides-92-rtgwg-8.pdf
>>
>> There is probably additional things in there to consider for NVO3, and advice that can be reused to make it easier to move NVO3 forward.
>>
>> Regards,
>>       Erik
>>
>>
>>
>> _______________________________________________
>> nvo3 mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/nvo3
>>
>

---

From: Lucy yong <[email-masked]>
Date: Wed, 1 Apr 2015 15:35:12 +0000

Hi Anoop,

I agree with Larry.  The encapsulation for NVO3 includes the VXLAN header and the outer header (UDP/IP). VXLAN, UDP and IP headers together determine a tunnel between two VTEPs. Thus NVO3 encapsulation is transport related.

NVO3 is the technology that uses tunneling for virtual network transport.

Thanks,
Lucy

From: Larry Kreeger (kreeger) [mailto:[email-masked]]
Sent: Tuesday, March 31, 2015 9:53 PM
To: Anoop Ghanwani; Lucy yong
Cc: Erik Nordmark; [email-masked]
Subject: Re: [nvo3] Encapsulation considerations

Hi Anoop,

IMO, the "encapsulation" includes both the VXLAN header and the UDP header and even the outer IP header.  Basically, everything that is added on the the original packet to delivery it from one VTEP to another.  Using VXLAN with MPLS instead of UDP/IP makes no sense to me, but maybe I lack imagination :-)

 - Larry

From: Anoop Ghanwani <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, March 31, 2015 5:05 PM
To: Lucy yong <[email-masked]<mailto:[email-masked]>>
Cc: Erik Nordmark <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Encapsulation considerations

I'm having trouble separating encapsulations into the categories mentioned.

When we talk about an encapsulation for NVO3 does it include the VXLAN header only, or does it include the UDP header as well?  Am I allowed to use the VXLAN header on an MPLS tunnel?  I'm not proposing we do this, just trying to see if there is a better way to communicate this separation.

Anoop

On Mon, Mar 30, 2015 at 1:16 PM, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:
Hi Eric,

Here is my thought.

Regards,
Lucy

-----Original Message-----
From: Erik Nordmark [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Saturday, March 28, 2015 12:32 AM
To: Lucy yong; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Encapsulation considerations

Lucy,

Thanks,
    Erik

> From: nvo3 [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Erik Nordmark
> To: [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Lucy yong <[email-masked]>
Date: Wed, 1 Apr 2015 16:12:09 +0000

Joe pointed out confusion in following reply.(offline) I should state that one type is transport related encapsulation and another is non-transport related encapsulation. 

Lucy

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Lucy yong
Sent: Wednesday, April 01, 2015 10:27 AM
To: Erik Nordmark; [email-masked]
Subject: Re: [nvo3] Encapsulation considerations

Hi Eric,

Regards,
Lucy

-----Original Message-----
From: Erik Nordmark [mailto:[email-masked]] 
Sent: Tuesday, March 31, 2015 11:58 PM
To: Lucy yong; Erik Nordmark; [email-masked]
Subject: Re: [nvo3] Encapsulation considerations

Lucy,

Thanks,
    Erik

---

From: "Lizhong Jin" <[email-masked]>
Date: Tue, 7 Apr 2015 15:02:40 +0800

Hi Erik,
Thanks for the draft. I suggest to add one consideration: the generation of
entropy value.
1. When the node receive an UDP/TCP packet from the tenant, then entropy
value could be a hash value of 5-tuple.
2. When the node receive an IP packet from the tenant, then entropy value
could be a hash value of 2-tuple.
3. When the node receive an Ethernet packet from the tenant, then entropy
value could be a hash value of MAC address.
4. When the node receive an IP fragmented packet from the tenant, the first
fragment is UDP/TCP, and entropy will be from 5-tuple. But the second and
subsequent segments will generate entropy from 2-tuple, which will have
different entropy value with the first segment. The issue could not be
resolved currently if NVE is separated from tenant into two physical
devices.

For section 18.1.1, I suggest RFS should be analyzed. E.g., if NIC want to
do flow steering based on the inner header information, then it could use
entropy value instead of the inner header information.

Regards
Lizhong

> From: Erik Nordmark [mailto:[email-masked]]
> Sent: 2015年3月26日 5:01
> To: [email-masked]
> I presented part of this at the most recent NVO3 interim meeting.The full
12
> areas of considerations where presented at RTGWG earlier this week.
>   The draft is
> There is probably additional things in there to consider for NVO3, and
advice
> that can be reused to make it easier to move NVO3 forward.
> 
> Regards,
>     Erik
> 
> 
>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 8 Apr 2015 17:19:45 -0700

> resolved currently if NVE is separated from tenant into two physical
This is a problem common to IP fragmentation. If the NVE is doing the
fragmentation, we have the option (like in
draft-herbert-gue-fragmentation-00) of fragmenting before the packet
before encapsulating, so that each fragment still uses the same
encapsulation and entropy value.

> entropy value instead of the inner header information.
Yes, that is the intent. There is some risk in that over a single
tunnel we would only have 14 bits of entropy but possibly many
thousands of flows so that there are collisions. This shouldn't be any
issues as long as the implementation take collisions into account and
the cost of collisions is low enough. Some alternatives are use IPv6
flow label for addiotnal 20 bits of entropy, or parse the inner packet
if we really need to uniquely identify the flow.

Tom

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 08 Apr 2015 17:27:27 -0700

> resolved currently if NVE is separated from tenant into two physical

Lizhong,
I've been assuming that the above issues are well-known, since entropy 
is generated and used locally for LAGs and L3 ECMP even when there is no 
encaps. Taking that entropy result and sticking it in a field in the 
packet doesn't add to the issues AFAICT.
I don't know if there is a document we can reference that states the 
above general considerations for flow entropy/hashing. Has anybody seen 
such a document? If so we can definitely reference it.
> entropy value instead of the inner header information.
Did you mean Receive Side Scaling (RSS)? That is mentioned in the 
document - section 18.1.1 . Perhaps the RSS text can be improved.
Receive Flow Scaling is the name of a Linux software technique to take 
into account the CPU on which the application is running, which is 
different.

Thanks,
    Erik

---

From: "Lizhong Jin" <[email-masked]>
Date: Thu, 9 Apr 2015 10:55:58 +0800

Hi Tom,
See inline below.

Regards
Lizhong

> From: Tom Herbert [mailto:[email-masked]]
> Sent: 2015年4月9日 8:20
> To: Lizhong Jin
> > Thanks for the draft. I suggest to add one consideration: the
> > generation of entropy value.
> > 1. When the node receive an UDP/TCP packet from the tenant, then
> > entropy value could be a hash value of 5-tuple.
> > 2. When the node receive an IP packet from the tenant, then entropy
> > value could be a hash value of 2-tuple.
> > 3. When the node receive an Ethernet packet from the tenant, then
> > entropy value could be a hash value of MAC address.
> > 4. When the node receive an IP fragmented packet from the tenant, the
> > first fragment is UDP/TCP, and entropy will be from 5-tuple. But the
> > second and subsequent segments will generate entropy from 2-tuple,
> > which will have different entropy value with the first segment. The
> > issue could not be resolved currently if NVE is separated from tenant
> > into two physical devices.
> >
> draft-herbert-gue-fragmentation-00) of fragmenting before the packet before
> encapsulating, so that each fragment still uses the same encapsulation and
> entropy value.
[Lizhong] NVE fragmentation could work if we could prevent the tenant from
fragmenting the packet. But if NVE is separated from tenant, we need additional
mechanism to notify the tenant.

> 
> > For section 18.1.1, I suggest RFS should be analyzed. E.g., if NIC
> > want to do flow steering based on the inner header information, then
> > it could use entropy value instead of the inner header information.
> >
> Yes, that is the intent. There is some risk in that over a single tunnel we would
> only have 14 bits of entropy but possibly many thousands of flows so that there
> are collisions. This shouldn't be any issues as long as the implementation take
> collisions into account and the cost of collisions is low enough. Some
> alternatives are use IPv6 flow label for addiotnal 20 bits of entropy, or parse the
> inner packet if we really need to uniquely identify the flow.
[Lizhong] Yes, 14bit could only provide 16k flows. But I have already seen some design
of 32k or more flows for hardware flow steering.

---

From: "Lizhong Jin" <[email-masked]>
Date: Thu, 9 Apr 2015 11:11:47 +0800

> Sent: 2015年4月9日 8:27
> To: Lizhong Jin; [email-masked]
> I've been assuming that the above issues are well-known, since entropy is
> generated and used locally for LAGs and L3 ECMP even when there is no encaps.
> Taking that entropy result and sticking it in a field in the packet doesn't add to
> the issues AFAICT.
> I don't know if there is a document we can reference that states the above
> general considerations for flow entropy/hashing. Has anybody seen such a
> document? If so we can definitely reference it.
[Lizhong] If the NVE and tenant is integrated into one device, then the issue could
be solved by implementation. Because tenant know the entropy value of the first 
segment, and use the same value to the subsequent segment. So different 
implementation model could provide different entropy value. Or do we need other
mechanism to mitigate this issue, e.g., fragment on NVE in 
draft-herbert-gue-fragmentation.

> Did you mean Receive Side Scaling (RSS)? That is mentioned in the document -
> section 18.1.1 . Perhaps the RSS text can be improved.
> Receive Flow Scaling is the name of a Linux software technique to take into
> account the CPU on which the application is running, which is different.
[Lizhong] I am referring to hardware based flow steering, similar with Accelerated 
RFS (Receive Flow Steering). In the NIC I/O virtualization, the NIC will directly trigger
interrupt to the CPU where application is running by looking up a flow table. We will 
use the inner header to do the flow table lookup, if hardware could not parse the 
inner header, then entropy value in the encapsulation is required. Any design of 
hiding inner header information should consider above implementation.

Regards
Lizhong Jin

---

From: Lucy yong <[email-masked]>
Date: Thu, 9 Apr 2015 14:27:59 +0000

Lizhong,

[snip]
 [Lizhong] If the NVE and tenant is integrated into one device, then the issue could be solved by implementation. Because tenant know the entropy value of the first segment, and use the same value to the subsequent segment. So different implementation model could provide different entropy value. Or do we need other mechanism to mitigate this issue, e.g., fragment on NVE in draft-herbert-gue-fragmentation.
[Lucy] IMO: NVO3 solution SHOULD avoid packet fragmentation. Draft-herbert-gue-fragmentation provides an option for a GUE application to do fragmentation but does not require doing it. GUE application decides if the fragmentation is needed or not. We should not separate two.

Regards,
Lucy

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 09 Apr 2015 10:46:06 -0700

Lizhong,

My point was more fundamental. Today on the Internet there are routers 
which do hashing for LAG or ECMP purposes. They have the same issues 
with fragmented packets.
> hiding inner header information should consider above implementation.

I thought the purpose of RFS was to send the packet (and associated 
interrupt) to the CPU where the application process is running. That 
implies an exact flow lookup. Some hash value, whether computed by the 
receiving NIC or whether in some entropy field in the packet (computed 
by the sender or encapsulator) would not suffice for that purpose.

Of course, a hash value (e.g., from the entropy field) is useful for RSS.

Regards,
    Erik

---

From: Tom Herbert <[email-masked]>
Date: Thu, 9 Apr 2015 10:56:35 -0700

On Thu, Apr 9, 2015 at 10:46 AM, Erik Nordmark <[email-masked]> wrote:
> My point was more fundamental. Today on the Internet there are routers which
> do hashing for LAG or ECMP purposes. They have the same issues with
> fragmented packets.
>>
>>
>>> document -
>>> into
> interrupt) to the CPU where the application process is running. That implies
> an exact flow lookup. Some hash value, whether computed by the receiving NIC
> or whether in some entropy field in the packet (computed by the sender or
> encapsulator) would not suffice for that purpose.
>
RFS is technically a best effort mechanism where exact flow lookup is
not necessary, and in many cases the device won't even be able to
determine the actual transport like if the encapsulated packet was
encrypted. What we need for this to work is a very low probability of
collisions among active traffic, an occasional should be that a few
packets may be routed to the wrong CPU. It still works, but is
slightly suboptimal for those packets. There have been some related
issues reported on Linux netdev that 16-bits of indirection indexed by
hash value is not enough.

> Of course, a hash value (e.g., from the entropy field) is useful for RSS.
Same value is used for RFS. NICs provide a 32 bit hash over 5-tuple.

Tom

---

From: "Lizhong Jin" <[email-masked]>
Date: Fri, 10 Apr 2015 10:22:55 +0800

> Sent: 2015年4月10日 1:46
> To: Lizhong Jin; 'Erik Nordmark'; [email-masked]
> > [Lizhong] If the NVE and tenant is integrated into one device, then
> > the issue could be solved by implementation. Because tenant know the
> > entropy value of the first segment, and use the same value to the
> > subsequent segment. So different implementation model could provide
> > different entropy value. Or do we need other mechanism to mitigate
> > this issue, e.g., fragment on NVE in draft-herbert-gue-fragmentation.
> Lizhong,
> 
> which do hashing for LAG or ECMP purposes. They have the same issues with
> fragmented packets.
[Lizhong] agree, but if the packet is encapsulated with tunnel, the situation may
be a little different. E.g., if only two tuple hashing is applied, then the fragmentation
issue will not exist in current deployment. But if the packet is encapsulated with 
tunnel, and hardware is unable to parse the inner header (very likely in IPv6 because
of limitation), then only entropy value could be used for LAG or ECMP purposes.
Then fragmented packet will always not be optimized.

> >> Receive Flow Scaling is the name of a Linux software technique to
> >> take into account the CPU on which the application is running, which is
> different.
> > [Lizhong] I am referring to hardware based flow steering, similar with
> > Accelerated RFS (Receive Flow Steering). In the NIC I/O
> > virtualization, the NIC will directly trigger interrupt to the CPU
> > where application is running by looking up a flow table. We will use
> > the inner header to do the flow table lookup, if hardware could not
> > parse the inner header, then entropy value in the encapsulation is required.
> Any design of hiding inner header information should consider above
> implementation.
> >
> 
> interrupt) to the CPU where the application process is running. That implies an
> exact flow lookup. Some hash value, whether computed by the receiving NIC or
> whether in some entropy field in the packet (computed by the sender or
[Lizhong] right, hash value is not enough for flow steering. One possible way is
to standardize one or two basic entropy generation algorithm, and encapsulation the
information in the header.
I know the encapsulation work may be unable to make every point in the network to be
most optimized. But since it is a consideration document, it is necessary to raise
the issue. And if it could be solved, it will be beneficial for the performance of host system.

Regards
Lizhong

---

From: "Lizhong Jin" <[email-masked]>
Date: Fri, 10 Apr 2015 10:22:55 +0800

> From: Lucy yong [mailto:[email-masked]]
> Sent: 2015年4月9日 22:28
> Subject: RE: [nvo3] Encapsulation considerations
> 
> Lizhong,
> 
> [snip]
>  [Lizhong] If the NVE and tenant is integrated into one device, then the issue
> could be solved by implementation. Because tenant know the entropy value of
> the first segment, and use the same value to the subsequent segment. So
> different implementation model could provide different entropy value. Or do we
> need other mechanism to mitigate this issue, e.g., fragment on NVE in
> [Lucy] IMO: NVO3 solution SHOULD avoid packet fragmentation.
> Draft-herbert-gue-fragmentation provides an option for a GUE application to do
> fragmentation but does not require doing it. GUE application decides if the
> fragmentation is needed or not. We should not separate two.
[Lizhong] fragmentation could not be avoided, because we are unable to prevent
the tenant from fragmentation. This is the factor which makes the hashing based
load balancing unoptimized.

Regards
Lizhong

---

From: Tom Herbert <[email-masked]>
Date: Fri, 10 Apr 2015 08:15:14 -0700

> information in the header.

Please see my comments. The hash has to work with flow steering and
this is already how RFS works. If it doesn't work then the only
alternative is that devices will need to continue to do deep, protocol
specific. parsing of all the different encapsulation protocols (GUE,
Geneve, VXLAN, VXLAN-GPE, MPLS/UDP, GRE/UDP, etc.)-- this complexity
is exactly what we are trying to avoid by allowing the UDP source port
for entropy or IPv6 flow label if that is not sufficient.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 10 Apr 2015 08:32:34 -0700

> Then fragmented packet will always not be optimized.
This is solved when tenants are using IPv6 and set the flow label per
RFC6438. The flow label would be set to the same value for all
fragments and the NVE would compute the hash based on addresses and
flow label (ignore ports). This is already supported in Linux when
creating a the source port for encapsulation (udp_flow_src_port
function).

You might want to take a look at
http://people.netfilter.org/pablo/netdev0.1/papers/UDP-Encapsulation-in-Linux.pdf.
This provides a lot of insight into implementation and optimization of
encapsulation on the host side.

Thanks,
Tom

---

From: Lucy yong <[email-masked]>
Date: Fri, 10 Apr 2015 16:19:32 +0000

-----Original Message-----
From: Lizhong Jin [mailto:[email-masked]] 
Sent: Thursday, April 09, 2015 9:23 PM
To: Lucy yong; 'Erik Nordmark'; [email-masked]
Subject: RE: [nvo3] Encapsulation considerations

> Draft-herbert-gue-fragmentation provides an option for a GUE 
> application to do fragmentation but does not require doing it. GUE 
> application decides if the fragmentation is needed or not. We should not separate two.
[Lizhong] fragmentation could not be avoided, because we are unable to prevent the tenant from fragmentation. This is the factor which makes the hashing based load balancing unoptimized.
[Lucy] The fragmentation performed by Tenant is a separate issue and transparent to NVE; An NVE always receives L2 or L3 packets. IMO: NVO3 solution SHOULD avoid packet fragmentation at NVE.

Lucy

Regards
Lizhong

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 10 Apr 2015 18:55:16 +0000

On 4/9/15 7:22 PM, "Lizhong Jin" <[email-masked]> wrote:

>>the issue

I'm not very familiar with host stacks.  Do they actually fragment at the
IP layer, or is it done at the transport layer before adding the IP
header?  I'm sure TCP must break the segments up before IP would fragment,
but I'm not sure about UDP.

 - Larry

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 10 Apr 2015 12:10:27 -0700

Even for TCP it depends on what the MSS is.

Anoop

On Fri, Apr 10, 2015 at 11:55 AM, Larry Kreeger (kreeger) <[email-masked]
> wrote:

---

From: Tom Herbert <[email-masked]>
Date: Fri, 10 Apr 2015 12:18:07 -0700

> but I'm not sure about UDP.
IP fragmentation is usually done at IP layer, although there are some
devices that can do UFO (UDP fragmentation offload). TCP segmentation
offload (GSO, TSO) is done right before giving the packet to the
driver or by the device for HW offload. This includes implementation
of segmentation offload for encapsulated TCP.

Tom

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Fri, 10 Apr 2015 19:49:55 +0000

I thought Path MTU discovery was used to set the MSS in the TCP stack.  I just wasn't sure if it worked the same way for UDP.

 - Larry

From: Anoop Ghanwani <[email-masked]<mailto:[email-masked]>>
Date: Friday, April 10, 2015 12:10 PM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>
Cc: Lizhong Jin <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Erik Nordmark <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Encapsulation considerations

Anoop

On Fri, Apr 10, 2015 at 11:55 AM, Larry Kreeger (kreeger) <[email-masked]<mailto:[email-masked]>> wrote:
On 4/9/15 7:22 PM, "Lizhong Jin" <[email-masked]<mailto:[email-masked]>> wrote:

>>From: Lucy yong [mailto:[email-masked]<mailto:[email-masked]>]
>>Sent: 2015年4月9日 22:28
>>To: Lizhong Jin; 'Erik Nordmark'; [email-masked]<mailto:[email-masked]>

 - Larry

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 10 Apr 2015 19:50:34 +0000

Lucy,

> On Mar 25, 2015, at 5:23 PM, Lucy yong <[email-masked]> wrote:
> 

I do not see this distinction.

What you call tunneling vs. service is an artifact of where in the stack you are placing your reference point. If you go up to the overlay, the “service” becomes the “tunnel”. Or looking at it from the bottom, the “tunnel” is a “service”.

> Considerations for two types of encapsulations have difference. It is good for the draft to point out that and give separate considerations.

The considerations seem to apply to all encaps, instead of a grouping, a table might help (where for some encap, some consideration might be a noop).

Thanks,

— Carlos.

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 10 Apr 2015 14:47:06 -0700

Yes, you're right.  My mistake.

Anoop

On Fri, Apr 10, 2015 at 12:49 PM, Larry Kreeger (kreeger) <[email-masked]

>  I thought Path MTU discovery was used to set the MSS in the TCP stack.
> I just wasn't sure if it worked the same way for UDP.
>
>   - Larry
>
>   From: Anoop Ghanwani <[email-masked]>
> To: Larry Kreeger <[email-masked]>
> Cc: Lizhong Jin <[email-masked]>, Lucy yong <[email-masked]>,
> Erik Nordmark <[email-masked]>, "[email-masked]" <[email-masked]>
> On Fri, Apr 10, 2015 at 11:55 AM, Larry Kreeger (kreeger) <
> [email-masked]> wrote:
>

---

From: Lucy yong <[email-masked]>
Date: Fri, 10 Apr 2015 22:22:00 +0000

Hi Carlos,

-----Original Message-----
From: Carlos Pignataro (cpignata) [mailto:[email-masked]]
Sent: Friday, April 10, 2015 2:51 PM
To: Lucy yong
Cc: Erik Nordmark; [email-masked]
Subject: Re: [nvo3] Encapsulation considerations

Lucy,

> On Mar 25, 2015, at 5:23 PM, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:

[Lucy] Term "Service" makes confusion here. But at the overlay, the "service" is not the "tunnel". Maybe this is better way to distinguish the two.

2) an encapsulation for non-tunneling, i.e. transport independent encapsulation, e.g. sfc.

I don't think that you can move the reference point to make nv03 encapsulation to non-tunneling encapsulation, and make sfc encapsulation into tunneling encapsulation. The architectures determine what they are.

[Lucy] A table ends up with the pattern of two kinds.

Thanks,

Lucy

Thanks,

— Carlos.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sat, 11 Apr 2015 17:45:39 +0000

Lucy,

Please see inline.

Thumb typed by Carlos Pignataro.
Excuze typofraphicak errows

On Apr 10, 2015, at 18:22, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:

Hi Carlos,

-----Original Message-----
From: Carlos Pignataro (cpignata) [mailto:[email-masked]]
Sent: Friday, April 10, 2015 2:51 PM
To: Lucy yong
Cc: Erik Nordmark; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Encapsulation considerations

Lucy,

[Lucy] Term "Service" makes confusion here.

I think there's more confusion than that :-)

But at the overlay, the "service" is not the "tunnel". Maybe this is better way to distinguish the two.

I do not think this is a working definition. What is "tunneling purpose" and "transport related"? [see next comment]

>From an SF perspective, the SFP (SFC encap) is a Tunnel. And furthermore, the SFC encap is used to transport at the SFC graph.

And are you saying that a tunnel or the nvo3 encap are inherently "transport dependent"?

You still have not defined what is a "non-tunneling" vs. a "tunneling" encapsulation. Net net, a self-referencing definition is not helping, and neither is using examples instead of a definition. In any case, what's the point of this attempt at grouping?

It does not. Unless you can show how, say, MTU and Entropy (assuming you consider those "transport" and "tunnel" attributes) are not relevant to, say, SFC encap (that you call service). Because what you call a "service" is a tunnel and transport for the actual service. And the "tunnel" is a service to IP.

In summary you have not showed a definition or the table.

Thanks,

Carlos.

Thanks,

Lucy

Thanks,

— Carlos.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 12 Apr 2015 00:03:00 +0000

> On Apr 8, 2015, at 8:19 PM, Tom Herbert <[email-masked]> wrote:
> 

Yes, although the solution space is richer, and the same is possible without defining encapsulation-layer fragmentation.

Basically, when encapsulating, there are three potential options for fragmentation:
Fragmentation after encapsulation — what we are trying to avoid. Encap and payload info is not carried after the fragmentation function.
Fragmentation of the payload before encapsulation. The same encap fields can be used for all fragments.
Fragmentation at the encapsulation header.
#1 and #3 imply reassembly at the tunnel tailed/decapsulator (either at the IP or encap layers respectively). #2 pushes reassembly towards the receiving host. And fragmentation is cheap, what’s expensive is to reassemble.

There is one more variable — I think the most important element in this, which has not been mentioned, is figuring out the path MTU (with PMTUD, etc) — because if you want to fragment at the encapsulation layer, which value do you use?

An exemplary definition and description of these three can be found with L2TPv3:
https://tools.ietf.org/html/rfc3931#section-4.1.4 <https://tools.ietf.org/html/rfc3931#section-4.1.4> specification of #1 and #2, plus generic description
http://tools.ietf.org/html/rfc4623#section-5 <http://tools.ietf.org/html/rfc4623#section-5> specification of #3 (fragmentation at the L2TPv3 layer)
Thanks,

— Carlos.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 12 Apr 2015 00:55:19 +0000

> On Apr 10, 2015, at 3:49 PM, Larry Kreeger (kreeger) <[email-masked]> wrote:
> 

Since, as you know, UDP has no MSS and no connection, the packetization size state is maintained by the application on top of UDP, or suffer IP-level fragmentation.

See e.g, first para of https://tools.ietf.org/html/rfc1191#section-6.1 <https://tools.ietf.org/html/rfc1191#section-6.1>

Thanks,

― Carlos.

>  - Larry
> 
> From: Anoop Ghanwani <[email-masked] <mailto:[email-masked]>>
> To: Larry Kreeger <[email-masked] <mailto:[email-masked]>>
> Cc: Lizhong Jin <[email-masked] <mailto:[email-masked]>>, Lucy yong <[email-masked] <mailto:[email-masked]>>, Erik Nordmark <[email-masked] <mailto:[email-masked]>>, "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
> On Fri, Apr 10, 2015 at 11:55 AM, Larry Kreeger (kreeger) <[email-masked] <mailto:[email-masked]>> wrote:
>> On 4/9/15 7:22 PM, "Lizhong Jin" <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> >>From: Lucy yong [mailto:[email-masked] <mailto:[email-masked]>]
>> >>Sent: 2015年4月9日 22:28
>> >>To: Lizhong Jin; 'Erik Nordmark'; [email-masked] <mailto:[email-masked]>
>> [email-masked] <mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/nvo3 <https://www.ietf.org/mailman/listinfo/nvo3>
>

---

From: "Lizhong Jin" <[email-masked]>
Date: Sun, 12 Apr 2015 15:24:52 +0800

> >> >>> For section 18.1.1, I suggest RFS should be analyzed. E.g., if
> >> >>> NIC want to do flow steering based on the inner header
> >> >>> information, then it could use entropy value instead of the inner header
> information.
> >> > [Lizhong] I am referring to hardware based flow steering, similar
> >> > with Accelerated RFS (Receive Flow Steering). In the NIC I/O
> >> > where application is running by looking up a flow table. We will
> >> > use the inner header to do the flow table lookup, if hardware could
> >> > not parse the inner header, then entropy value in the encapsulation is
> required.
> >> implies an exact flow lookup. Some hash value, whether computed by
> >> the receiving NIC or whether in some entropy field in the packet
> >> (computed by the sender or
> > [Lizhong] right, hash value is not enough for flow steering. One
> > possible way is to standardize one or two basic entropy generation
> > algorithm, and encapsulation the information in the header.
> 
> Please see my comments. The hash has to work with flow steering and this is
> already how RFS works. If it doesn't work then the only alternative is that
> devices will need to continue to do deep, protocol specific. parsing of all the
> different encapsulation protocols (GUE, Geneve, VXLAN, VXLAN-GPE,
> MPLS/UDP, GRE/UDP, etc.)-- this complexity is exactly what we are trying to
> avoid by allowing the UDP source port for entropy or IPv6 flow label if that is
> not sufficient.
[Lizhong] Hi Tom, in Linux RFS, it does use rxhash value to index the flow table.
And Linux also use the same rxhash value to program the flow table after receiving
the packet. So the sequence is, first receive packet from a default CPU core, then
program the flow table, and then steer the packet to the new core for subsequent
packets.
Current hardware flow steering provides two way to program the flow table:
1. program flow table by ndo_rx_flow_steer, then it is same as current linux way, and
hash value could work in this way.
2. program by e.g., eth tool, then before receiving any packet, we could not program
the entry, because we do not know the hash value. If we could standardize the entropy
generation, then we could program before receiving any packet. This programming
is usefull if we want to fix one application to be always on one core.

Regards
Lizhong

> 
> Tom
> 
> > I know the encapsulation work may be unable to make every point in the
> > network to be most optimized. But since it is a consideration
> > document, it is necessary to raise the issue. And if it could be solved, it will be
> beneficial for the performance of host system.
> >
> 
> > Regards
> > Lizhong
> >
> >>

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Tue, 14 Apr 2015 01:05:02 +0000

Hi Carlos,

Thanks for the reference below.  So, this means the answer is "it depends" on the packetization layer above UDP, but do you know if it is common for applications running over UDP to pay attention to the PMTU and adjust its packetization size accordingly?  Are there known "problem protocols" that don't do this? (e.g. RFC 1191 that you reference mentions NFS, but this may no longer be true of newer implementations?).

Thanks, Larry

From: "Carlos Pignataro (cpignata)" <[email-masked]<mailto:[email-masked]>>
Date: Saturday, April 11, 2015 5:55 PM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>
Cc: Anoop Ghanwani <[email-masked]<mailto:[email-masked]>>, Erik Nordmark <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Lucy yong <[email-masked]<mailto:[email-masked]>>, Lizhong Jin <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Encapsulation considerations

On Apr 10, 2015, at 3:49 PM, Larry Kreeger (kreeger) <[email-masked]<mailto:[email-masked]>> wrote:

See e.g, first para of https://tools.ietf.org/html/rfc1191#section-6.1

Thanks,

― Carlos.

 - Larry

Anoop

 - Larry

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 15 Apr 2015 17:42:49 -0700

> Same value is used for RFS. NICs provide a 32 bit hash over 5-tuple.
Ah - I must have read the Linux description to quickly.
A hash works to spread the flows, and if the process is then scheduled 
on the same CPU as where the packets arrive performance improves.

My point was that if you first one to schedule/bind the process to a CPU 
and later ensure that the packets arrive on said CPU, then you need an 
exact match and not a hash.

Thanks,
    Erik

---

From: Tom Herbert <[email-masked]>
Date: Thu, 16 Apr 2015 15:03:57 -0700

On Wed, Apr 15, 2015 at 5:42 PM, Erik Nordmark <[email-masked]> wrote:
>>> implies
> A hash works to spread the flows, and if the process is then scheduled on
> the same CPU as where the packets arrive performance improves.
>
> My point was that if you first one to schedule/bind the process to a CPU and
> later ensure that the packets arrive on said CPU, then you need an exact
> match and not a hash.
>
If the hash entropy is large enough to make collisions near zero, then
the hash can be used a unique flow identifier.
With a 32 bit hash value and say 10K active connections, there are
very few collisions. The binding something like is Hash->Queue->CPU,
so host can program which queue to receive packets on that match the
hash.

In reality though, exact flow match is a fleeting concept in devices
and the hash is really want we need anyway. We really don't want
devices to need to parse every encapsulation protocol to find inner
transport headers, and besides when we start encrypting encapsulated
packets they won't be able to get to the exact flow anyway. Also, it's
important that system can adapt to seeing flows whose outer hashes
change (like when UDP source port changes). SPUD is also compelling
here as a way to provide a specific more persistent and explicit flow
identification which is needed for stateful inspection/firewalls for
encapsulated transports.

Tom

---

From: Lucy yong <[email-masked]>
Date: Wed, 22 Apr 2015 15:26:19 +0000

Hi Carlos,

Sorry for the late response due to business travel.

You ask to give the definition of tunneling encapsulation and non-tunneling encapsulation. I give a try here.

Tunneling encapsulation: the encapsulation contains tunnel end point information as well as other information that assists tunnel transport purpose. The encapsulated payload is tunneled from tunnel ingress to egress.

Non-tunneling encapsulation: the encapsulation contains the information that is used for the encapsulated payload operations. Such payload operations are independent of the architecture of a delivery network and how the payload is carried over the delivery network.

Now question is, if we will have an encapsulation that does both? If yes, the corresponding architecture makes two layers dependent each other, which, in general, is a bad design.

Regards,
Lucy

From: Carlos Pignataro (cpignata) [mailto:[email-masked]]
Sent: Saturday, April 11, 2015 12:46 PM
To: Lucy yong
Cc: Erik Nordmark; [email-masked]
Subject: Re: [nvo3] Encapsulation considerations

Lucy,

Please see inline.

Hi Carlos,

-----Original Message-----
From: Carlos Pignataro (cpignata) [mailto:[email-masked]]
Sent: Friday, April 10, 2015 2:51 PM
To: Lucy yong
Cc: Erik Nordmark; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Encapsulation considerations

Lucy,

What you call tunneling vs. service is an artifact of where in the stack you are placing your reference point. If you go up to the overlay, the "service" becomes the "tunnel". Or looking at it from the bottom, the "tunnel" is a "service".

Thanks,

Carlos.

Thanks,

Lucy

Thanks,

- Carlos.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 24 Apr 2015 01:26:31 +0000

Hi, Larry,

> On Apr 13, 2015, at 9:05 PM, Larry Kreeger (kreeger) <[email-masked]> wrote:
> 
> Hi Carlos,
> 

That’s right ― applications running over UDP can use the IP-layer info minus their overhead to set their packetization sizes, and/or register to listen to ICMPs with embedded errored packets for them.

One example of a protocol doing this, spec’ed in detail, is L2TP / L2TPv3 (e.g., RFC 3931, S4.1.4)

Thanks,

― Carlos.

> 
> Thanks, Larry
> 
> From: "Carlos Pignataro (cpignata)" <[email-masked] <mailto:[email-masked]>>
> Cc: Anoop Ghanwani <[email-masked] <mailto:[email-masked]>>, Erik Nordmark <[email-masked] <mailto:[email-masked]>>, "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>, Lucy yong <[email-masked] <mailto:[email-masked]>>, Lizhong Jin <[email-masked] <mailto:[email-masked]>>
>> On Apr 10, 2015, at 3:49 PM, Larry Kreeger (kreeger) <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 24 Apr 2015 01:26:33 +0000

Hi Lucy,

Thanks for the follow-up, please see inline.

> On Apr 22, 2015, at 11:26 AM, Lucy yong <[email-masked]> wrote:
> 
> Hi Carlos,
> 

Defining “Tunneling encapsulation” as something that "contains tunnel end point information” seems to be of marginal value. What is a tunnel?

But entertaining that definition for a bit, a corollary: MPLS LSPs are not “Tunneling encapsulation” since they do not "contains tunnel end point information”.

Further, encapsulations like SFC could be “Tunneling encapsulations” (although before you classify that as Service), because it “Transports an encapsulated payload, tunneling it, from tunnel ingress to egress.” Basically, SFC tunnels a payload from SF ingress to SF egress, passing through a number of SFF hops.

> Non-tunneling encapsulation: the encapsulation contains the information that is used for the encapsulated payload operations. Such payload operations are independent of the architecture of a delive…

GRE includes a protocol identifier field, which is used for the encapsulated payload identification. Maybe GRE is non-tunneling...

L2TPv3 contains an Adaptation Layer, the L2-Specific Sublayer, which includes for “information that is used for the encapsulated payload operations”. E.g., with payload header bits in RFC 4454 and RFC 4591. It also includes fragmentation info, RFC 4623. Based on this, L2TPv3 is a non-tunneling “L2 tunneling protocol version 3”.

The point is there is no such distinction as you are trying to make — it is all depending on whether you look from “above” or from “below”.

And the point behind the point is that attempting to make that distinction does not help solve the problem at hand anyway.

> Now question is, if we will have an encapsulation that does both? If yes, the corresponding architecture makes two layers dependent each other, which, in general, is a bad design.

This is a non-sequitur.

Thanks,

— Carlos.

> Regards,
> Lucy
> 
> 
> 
> 
> From: Carlos Pignataro (cpignata) [mailto:[email-masked] <mailto:[email-masked]>]
> Cc: Erik Nordmark; [email-masked] <mailto:[email-masked]>
> On Apr 10, 2015, at 18:22, Lucy yong <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Hi Carlos,
> 
> From: Carlos Pignataro (cpignata) [mailto:[email-masked] <mailto:[email-masked]>]
> Cc: Erik Nordmark; [email-masked] <mailto:[email-masked]>
> > On Mar 25, 2015, at 5:23 PM, Lucy yong <[email-masked] <mailto:[email-masked]>> wrote:
> >
> > From: nvo3 [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of Erik Nordmark
> > To: [email-masked] <mailto:[email-masked]>
> >    http://datatracker.ietf.org/doc/draft-rtg-dt-encap/ <http://datatracker.ietf.org/doc/draft-rtg-dt-encap/>
> >  and the slides are at
> >   http://www.ietf.org/proceedings/92/slides/slides-92-rtgwg-8.pdf <http://www.ietf.org/proceedings/92/slides/slides-92-rtgwg-8.pdf>
> >

---

From: Lucy yong <[email-masked]>
Date: Fri, 24 Apr 2015 16:03:11 +0000

Hi Carlos,

Thank you for the response, please se inline.

From: Carlos Pignataro (cpignata) [mailto:[email-masked]]
Sent: Thursday, April 23, 2015 8:27 PM
To: Lucy yong
Cc: Erik Nordmark; [email-masked]
Subject: Re: [nvo3] Encapsulation considerations

Hi Lucy,

On Apr 22, 2015, at 11:26 AM, Lucy yong <[email-masked]<mailto:[email-masked]>> wrote:

Hi Carlos,

Defining “Tunneling encapsulation” as something that "contains tunnel end point information” seems to be of marginal value. What is a tunnel?
[Lucy] The tunnel is defined by a pair of end points. The tunnel carries datagrams between the pair of end points. For a tunnel over an IP network, the end points have an IP address.

But entertaining that definition for a bit, a corollary: MPLS LSPs are not “Tunneling encapsulation” since they do not "contains tunnel end point information”.
[Lucy] LSP stands for label switch path. IMO: It is not a tunnel technology.

Further, encapsulations like SFC could be “Tunneling encapsulations” (although before you classify that as Service), because it “Transports an encapsulated payload, tunneling it, from tunnel ingress to egress.” Basically, SFC tunnels a payload from SF ingress to SF egress, passing through a number of SFF hops.
[Lucy] This does like entertaining. ☺  There are debating on SFC list above if NSH draft should remain transport independent or not. You are in favor of the independent.

GRE includes a protocol identifier field, which is used for the encapsulated payload identification. Maybe GRE is non-tunneling...
[Lucy] GRE by itself is non-tunneling. The applications using GRE such as PPTP, GRE-in-UDP, GRE-in-IP are tunnels.

L2TPv3 contains an Adaptation Layer, the L2-Specific Sublayer, which includes for “information that is used for the encapsulated payload operations”. E.g., with payload header bits in RFC 4454 and RFC 4591. It also includes fragmentation info, RFC 4623. Based on this, L2TPv3 is a non-tunneling “L2 tunneling protocol version 3”.
[Lucy] IMHO: L2TPv3 may be an example that does both. What is the reason that L2TPv3 is not widely deployed?

The point is there is no such distinction as you are trying to make — it is all depending on whether you look from “above” or from “below”.
[Lucy] IMO, it is important to be aware of the distinction from the architecture design. It makes clear the role for the encapsulator/decapsulator. BTW, where is the point to look at “above” and “below”?

And the point behind the point is that attempting to make that distinction does not help solve the problem at hand anyway.
[Lucy] Seems we have a disagreement here.  That is OK, the world is like that. ☺

Thanks,
Lucy

This is a non-sequitur.

Thanks,

— Carlos.

Regards,
Lucy

Lucy,

Please see inline.

-----Original Message-----
From: Carlos Pignataro (cpignata) [mailto:[email-masked]]
Sent: Friday, April 10, 2015 2:51 PM
To: Lucy yong
Cc: Erik Nordmark; [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Encapsulation considerations

Lucy,

Thanks,

Carlos.

Thanks,
Lucy

Thanks,

— Carlos.

---

From: "Lizhong Jin" <[email-masked]>
Date: Tue, 28 Apr 2015 16:40:58 +0800

Hi,
>From this discussion, at least I could see the hardware flow steering need to be considered in following aspects:
1. 32bit hash value will benefit the loadbalancing of CPU load when flow steering is applied. If two flows has collision to have hash value, then RFS on CPU will steer the flow to dst CPU. But that will burden the CPU load if there are too many collisions.
2. I still suggest to describe some suggestions for the hash function of the entropy value. The hash function should provide collision rate as lower as possible. Then the NIC could decide whether trust the received entropy value to do flow steering. If not trust, then revert to RSS.

BR
Lizhong

> Sent: 2015年4月17日 6:04
> To: Erik Nordmark
> Cc: Lizhong Jin; [email-masked]
> >> issues reported on Linux netdev that 16-bits of indirection indexed
> >> by hash value is not enough.
> >>
> > My point was that if you first one to schedule/bind the process to a
> > CPU and later ensure that the packets arrive on said CPU, then you
> > need an exact match and not a hash.
> >
> If the hash entropy is large enough to make collisions near zero, then the hash
> can be used a unique flow identifier.
> With a 32 bit hash value and say 10K active connections, there are very few
> collisions. The binding something like is Hash->Queue->CPU, so host can
> program which queue to receive packets on that match the hash.
> 
> In reality though, exact flow match is a fleeting concept in devices and the hash
> is really want we need anyway. We really don't want devices to need to parse
> every encapsulation protocol to find inner transport headers, and besides when
> we start encrypting encapsulated packets they won't be able to get to the
> exact flow anyway. Also, it's important that system can adapt to seeing flows
> whose outer hashes change (like when UDP source port changes). SPUD is also
> compelling here as a way to provide a specific more persistent and explicit flow
> identification which is needed for stateful inspection/firewalls for encapsulated
> transports.
> 
> Tom
> 
> > Thanks,
> >    Erik
> >
> >
> >>
> >> Tom
> >>
> >>> Regards,
> >>>     Erik
> >>>