Subject: [nvo3] draft-quinn-vxlan-gpe-00 backwards compatibility

From: Tom Herbert <[email-masked]>
Date: Tue, 18 Mar 2014 08:14:32 -0700

Hi gpe authors,

Have you verified that gpe is backwards compatible with legacy VXLAN
implementations?

Consider an HW implementation that has implemented TCP RX checksum
offload of inner packet for VXLAN. Per VXLAN draft, a device should
interpret the payload as being an Ethernet frame and ignore reserved
bits (which includes GPE bit) on receive. So when presented a GPE
packet, it seems like the HW implementation might attempt RX csum
assuming the wrong packet type. In most cases this would "work"
because the packet wouldn't appear to be a TCP packet, however if the
right data is in the right offsets (offsets of Ethertype, IP protocol)
it seems like the device might incorrectly report a checksum error. If
this is possible, it will be a very subtle bug!

Thanks,
Tom

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Tue, 18 Mar 2014 17:38:14 +0000

On 18/03/14 15:14, Tom Herbert wrote:
> offload of inner packet for VXLAN. 

And your point is?

> Per VXLAN draft, a device should
> assuming the wrong packet type.

With the current encapsulation proliferation you gotta be mad to even
try to do protocol specific RX checksum offload anywhere. Anyone with at
least a modicum of remaining sanity does full packet checksum instead.
Drivers for older (or badly designed hardware) convert the "probably
broken" checksum from your example to the protocol independent checksum
which is correct.

Once this is done the actual overlay driver in the lower layers can
utilise it. So the fact that for example the NIC from vendor Y did not
quite understand correctly the encapsulation which has been invented
after the NIC has come out no longer matters.

Example in hand - e1000 and family of drivers for Intel nics and most
overlay encaps. The kernel collects the UDP checksum, converts it to a
"CHECKSUM_COMPLETE" form (usually less than 10 pipeline-able
instructions) and passes that to lower layers. Now _ANY_ overlay can use
it by doing a block subtract of the header portion. It can even verify
and compute checksums for headers, etc in the overlay part on the way.

So if the hardware completes at least one checksum in a sane manner it
can be used - code is already there.

I could have posted an example from BSD, Open Solaris or something else
for that matter. They all use the same technique.

>  In most cases this would "work"

Can you actually provide at least one example based in reality please?

A.

>
> Thanks,
> Tom
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3
>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 18 Mar 2014 14:43:27 -0700

>> offload of inner packet for VXLAN.
I am not making a point, I am asking a question.

> Can you actually provide at least one example based in reality please?
https://github.com/adrianschroeter/kernel-source/blob/master/patches.drivers/i40e-0087-i40e-Rx-checksum-offload-for-VXLAN.patch

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Tue, 18 Mar 2014 22:28:03 +0000

> https://github.com/adrianschroeter/kernel-source/blob/master/patches.drivers/i40e-0087-i40e-Rx-checksum-offload-for-VXLAN.patch

Another person doing per-protocol special "you need to buy a new adapter
for a new protocol" magic and not reading the documentation...

Instead of actually reading how it is supposed to be done so that _ANY_
encaps further downstream can take care of it.

http://lxr.free-electrons.com/source/include/linux/skbuff.h#L40

In any case, I agree, the fact that a software developer does not know
how to convert specific offload into a generic (including covering cases
which the card [or NPU] was never intended to cover) should of course be
taken into account. We should create all protocol specs in a manner such
that people who do not read documentation and the spec towards which
they are supposed to write should not have their software broken you
know... We should also invent a new protocol every time just in case
something somewhere breaks if we augment an old one.

Or maybe not...

A.

---

From: Thomas Graf <[email-masked]>
Date: Wed, 19 Mar 2014 09:13:02 +0100

> http://lxr.free-electrons.com/source/include/linux/skbuff.h#L40

I agree with you but the reality is this:

$ git grep ' = CHECKSUM_UNNECESSARY' drivers/net/ | wc -l
102
$ git grep ' = CHECKSUM_COMPLETE' drivers/net/ | wc -l
22

Looking at the 3 specific drivers which already support UDP
encap offload (bnx2x, i40e, mlx4):

$ git grep ' = CHECKSUM_' */+(bnx2x|mlx4|i40e)
broadcom/bnx2x/bnx2x_cmn.c:  skb->ip_summed = CHECKSUM_UNNECESSARY;
broadcom/bnx2x/bnx2x_cmn.c:  skb->ip_summed = CHECKSUM_UNNECESSARY;
intel/i40e/i40e_txrx.c:      skb->ip_summed = CHECKSUM_NONE;
intel/i40e/i40e_txrx.c:      skb->ip_summed = CHECKSUM_UNNECESSARY;
mellanox/mlx4/en_rx.c:       gro_skb->ip_summed = CHECKSUM_UNNECESSARY;
mellanox/mlx4/en_rx.c:       ip_summed = CHECKSUM_UNNECESSARY;
mellanox/mlx4/en_rx.c:       ip_summed = CHECKSUM_NONE;
mellanox/mlx4/en_rx.c:       ip_summed = CHECKSUM_NONE;

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Wed, 19 Mar 2014 08:46:45 +0000

> $ git grep ' = CHECKSUM_UNNECESSARY' drivers/net/ | wc -l

I know :(

If you look at the actual code however, most cases where you will see
UNNECESSARY will be TCP checksum on non-encapsulated payloads - not
something we should care about.

> encap offload (bnx2x, i40e, mlx4):

All I can say - bad design.

One change in the encaps and it breaks (probably by design too - you get
to sell new NICs).

My old comment that we should not be bound in the architecture choices
by bad designs by any particular vendor is still valid too. Even more
valid - thanks for illustrating it.

---

From: Thomas Graf <[email-masked]>
Date: Wed, 19 Mar 2014 10:31:44 +0100

> valid - thanks for illustrating it.

If you manage to convince vendors to change direction, perfect!
Otherwise VXLAN-gpe will be "done right" but irrelevant due to
lack of backwards compatibility in the real world.

---

From: "Anton Ivanov (antivano)" <[email-masked]>
Date: Wed, 19 Mar 2014 10:04:01 +0000

> lack of backwards compatibility in the real world.

You mean - lack of backward compatibility on 3 specific first generation
specialized NICs which are presently shipping, right?

So all of us who use more generic hardware which has a checksumming algo
which can be adapted to cope, should now abandon any ideas to improve a
protocol just because three early examples of technology have been
designed in a manner which explicitly forces their buyers to upgrade
later. Right?

Interesting idea. To say the least.

A.

---

From: Tom Herbert <[email-masked]>
Date: Wed, 19 Mar 2014 08:12:57 -0700

> Interesting idea. To say the least.
Whether you like or dislike how hardware vendors implement their
features is immaterial, the fact is that what they are doing is
correct protocol-wise. It is the proposed change to VXLAN protocol
that is not backwards compatible and can lead to incorrect behavior in
existing implementations.

Also, I would point, RX csum is only one example. The general
statement is that any legacy implementation which receives a VXLAN
packet with the new format encapsulating a non-Ethernet frame
potentially misinterprets the packet and experiences bad behavior.

---

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Wed, 19 Mar 2014 17:21:03 +0000

>existing implementations.

I guess I disagree with your definition of backward compatibility.  To me,
it means new implementations can support both the new and old
(old=backward).  It doesn't mean that old implementations can somehow
magically support new extensions (new=forward).

>potentially misinterprets the packet and experiences bad behavior.

If the software on host with an old offload NIC wants to support the new
extensions, can't the offloads be turned off so they are not
misinterpreted?

---

From: "Paul Quinn (paulq)" <[email-masked]>
Date: Wed, 19 Mar 2014 18:00:37 +0000

> magically support new extensions (new=forward).

PQ>  That's my definition as well.

---

From: Tom Herbert <[email-masked]>
Date: Wed, 19 Mar 2014 11:02:11 -0700

> magically support new extensions (new=forward).
It's not an issue of old implementations supporting the new
extensions, it's a problem that the mere *presence* of the new
extensions cause incorrect or nondeterministic behavior in old
implementations. Had VXLAN specification said that the payload must
not be interpreted in the presence of unknown flags, then this change
would be safe.

It is an elementary property of protocols not to break previously
deployed versions-- this is a necessary consequence of the robustness
principle. We needed to create IPv6 as a new version because changing
the address size in IPv4 would have reeked havoc. A TCP stack must be
prepared to handle options which are unknown to it. A very similar
situation we have in TCP is the problem of a limited option space,
there is no feasible way to increase option space because any attempts
would break compatibility with deployed implementations (which is
pretty significant!) ).

> extensions, can't the offloads be turned off so they are not
I would hope. If VXLAN has seen limited deployment it might be
reasonable to absorb the change and mitigate bad effects. I suppose
the argument that VXLAN is only an I-D technically permits any
non-compatible changes, but in reality several stacks and hardware
vendors have already implemented and probably deployed the protocol.

---

From: Lucy yong <[email-masked]>
Date: Wed, 19 Mar 2014 18:21:25 +0000

[snip]
>> that is not backwards compatible and can lead to incorrect behavior 
>> in existing implementations.
> 
> I guess I disagree with your definition of backward compatibility.  To 
> me, it means new implementations can support both the new and old 

[Lucy] I disagree this definition. IMO: The backward compatibility means two things: 
1) the new implementations can support both the new and old. 
2) the old one does not support new but is able to tell that one does not match what it supports. So it can drop the packet and raise an alert.

Lucy

> If the software on host with an old offload NIC wants to support the 
> new extensions, can't the offloads be turned off so they are not 
> misinterpreted?
> 
>> 
>>> A.

---

From: Fabio Maino <[email-masked]>
Date: Mon, 24 Mar 2014 17:05:27 -0700

> potentially misinterprets the packet and experiences bad behavior.

Tom,
please note that the VXLAN-GPE draft says that a GPE device must not 
send non-ethernet frames to a VXLAN device (Section 4.2), exactly to 
avoid the problem you describe.

Also note that the draft is focusing on deployments where VXLAN is 
already in use, and GPE is introduced incrementally. Hence the use of 
the same UDP port for VXLAN and GPE.

I suspect that in the deployment you describe, one could disambiguate 
GPE from VXLAN by using two different destination UDP ports. if you 
still want to have backward compatibility the sending GPE device will 
have to know the receiver's capability (VXLAN or GPE), and pick the 
appropriate destination UDP port.

Regards,
Fabio

---

From: Tom Herbert <[email-masked]>
Date: Tue, 25 Mar 2014 08:10:00 -0700

> Tom,
> please note that the VXLAN-GPE draft says that a GPE device must not send
> non-ethernet frames to a VXLAN device (Section 4.2), exactly to avoid the
> problem you describe.
>
Unfortunately, that requirement conflicts with the robustness
principle. In a full scale deployment, it might be potentially
feasible with a whole bunch of control plane logic to enforce the rule
between communicating end hosts, but that still wouldn't account for
middleboxes somewhere in the path that have implemented VXLAN
functionality.

Note, this is not the only potential incompatibility issue with VXLAN.
Every new flag defining a new field would create another instance of
incompatibility. This is not just hypothetical, we have already
demonstrated that adding a new field to GRE breaks hashing in
switches. This problem also exists with nvgre.

> Also note that the draft is focusing on deployments where VXLAN is already
> in use, and GPE is introduced incrementally. Hence the use of the same UDP
> port for VXLAN and GPE.
>
> I suspect that in the deployment you describe, one could disambiguate GPE
> from VXLAN by using two different destination UDP ports. if you still want
> to have backward compatibility the sending GPE device will have to know the
> receiver's capability (VXLAN or GPE), and pick the appropriate destination
> UDP port.
>
At that point it becomes a different protocol.

I believe a generic and extensible encapsulation protocol needs three
fundamental elements:

1) Type-version-- so that new (incompatible) formats can be safely defined
2) Protocol type-- type of encapsulated packets
3) Header length-- offset of next header can be determined
independently of any other elements in the encapsulation.

The semantics of these elements should be invariant (just like in IP).
Protocol type always defines the type encapsulated packet, header
length is always offset of it. If an alternate interpretation of the
payload is needed which does not correspond to a protocol type (like
an OAM message) this should be in a separate type.

Please look http://tools.ietf.org/html/draft-herbert-gue-01 for reference.

Thanks,
Tom

---

From: Lucy yong <[email-masked]>
Date: Tue, 25 Mar 2014 16:06:34 +0000

Hi Tom,

I agree your statement/concern on VXLAN. NVGRE (GRE) has version field, it does not have such backward compatibility issue when enhancing it.

Cheers,
Lucy 

-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Tuesday, March 25, 2014 10:10 AM
To: Fabio Maino
Cc: [email-masked]
Subject: Re: [nvo3] draft-quinn-vxlan-gpe-00 backwards compatibility

> avoid the problem you describe.
Unfortunately, that requirement conflicts with the robustness principle. In a full scale deployment, it might be potentially feasible with a whole bunch of control plane logic to enforce the rule between communicating end hosts, but that still wouldn't account for middleboxes somewhere in the path that have implemented VXLAN functionality.

Note, this is not the only potential incompatibility issue with VXLAN.
Every new flag defining a new field would create another instance of incompatibility. This is not just hypothetical, we have already demonstrated that adding a new field to GRE breaks hashing in switches. This problem also exists with nvgre.

I believe a generic and extensible encapsulation protocol needs three fundamental elements:

1) Type-version-- so that new (incompatible) formats can be safely defined
2) Protocol type-- type of encapsulated packets
3) Header length-- offset of next header can be determined independently of any other elements in the encapsulation.

The semantics of these elements should be invariant (just like in IP).
Protocol type always defines the type encapsulated packet, header length is always offset of it. If an alternate interpretation of the payload is needed which does not correspond to a protocol type (like an OAM message) this should be in a separate type.

Thanks,
Tom

---

From: Fabio Maino <[email-masked]>
Date: Tue, 25 Mar 2014 11:28:48 -0700

> middleboxes somewhere in the path that have implemented VXLAN

right, the hard part is the incremental deployment of a new solution on 
top of the existing VXLAN implementations. The draft is just suggesting 
a way to do that incremental deployment, with some known limitations. 
GPE tries to play with the current definition of 'reserved' bits in 
VXLAN, using that as a way to transition existing VXLAN fabrics to GPE 
fabrics that will support multiple protocol encapsulation (IPx, 
ethernet) and metadata.

> switches. This problem also exists with nvgre.

In fact, one could argue that every new flag added after the protocol 
definition is indeed a bit of a new distributed 'version' field. One 
could explicitly use a couple of bits as a version, or define the new 
protocol in a way that a new flag will raise an exception in older 
implementations.

> At that point it becomes a different protocol.

Right. It is indeed a different protocol that is trying to coexist with 
the reality of networks where VXLAN (and LISP, by the way, as specified 
in the companion doc http://tools.ietf.org/html/draft-lewis-lisp-gpe) is 
already deployed.

> 1) Type-version-- so that new (incompatible) formats can be safely defined

I think Type-version doesn't buy you much in term of backward 
compatibility (compatibility of a newer device with an older device): 
you just can't change the VXLAN specification.

It helps a bit with forward compatibility (compatibility with future 
versions of the protocol) to the limited extent that older 
implementations will have to take a specific action (drop most likely) 
for newer versions.

However, I think you can do 'versioning' in various way: using a 
different UDP port, with an explicit version field, or using the 
combination of the reserved bits as a version field. Given that reserved 
bits have proven over the years to be hot real estate, GPE is not using 
an explicit version field, but does support versioning.

> an OAM message) this should be in a separate type.
or you could use a flag for the last one (OAM).

All of those features come with a cost, and can be implemented in 
different ways. GPE is trying to use an approach that is close to VXLAN 
(and LISP) so that the incremental cost of implementing GPE+VXLAN+LISP 
on the same device give it a chance to be deployed.
> Please look http://tools.ietf.org/html/draft-herbert-gue-01 for reference.

That's a great write up. Thanks especially for the appendix that 
articulates very well the motivations that are driving the effort of 
using the optimization provided by current NICs.

I think the design of the protocol would benefit from a better 
separation of the network virtualization layer and the metadata layer. 
It would allow each implementation (at the end host or in the network) 
to implement independently part of the specification, and will 
eventually help with adoption. I think with a better layering you could 
also take advantage of other well established features (such as 
security, for example) that you may want to reuse, rather than reinvent.

Fabio

---

From: Lucy yong <[email-masked]>
Date: Tue, 25 Mar 2014 18:47:53 +0000

> switches. This problem also exists with nvgre.

In fact, one could argue that every new flag added after the protocol definition is indeed a bit of a new distributed 'version' field. One could explicitly use a couple of bits as a version, or define the new protocol in a way that a new flag will raise an exception in older implementations.

[Lucy] There are tread-off b/w two methods. If you want to implement distributed "version" field, you MUST request the protocol mechanism to check all reserved bits and, if any those bit is set, you treat it as "version" not-match. This will work but the cons are hardware cost on checking every reserved bit. If you use the explicitly version field, you only check the version field to determine the compatibility but the cons are that it is hard to do a small incremental change without a new version. Both can work w/cons. Which one is better for hardware and software implementation? 

BTW, VXLAN protocol solution does not require checking those reserved bits, so it is not yet compliant with distributed "version" method.

Cheers,
Lucy    

> At that point it becomes a different protocol.

Right. It is indeed a different protocol that is trying to coexist with the reality of networks where VXLAN (and LISP, by the way, as specified in the companion doc http://tools.ietf.org/html/draft-lewis-lisp-gpe) is already deployed.

> 1) Type-version-- so that new (incompatible) formats can be safely 

I think Type-version doesn't buy you much in term of backward compatibility (compatibility of a newer device with an older device): 
you just can't change the VXLAN specification.

It helps a bit with forward compatibility (compatibility with future versions of the protocol) to the limited extent that older implementations will have to take a specific action (drop most likely) for newer versions.

However, I think you can do 'versioning' in various way: using a different UDP port, with an explicit version field, or using the combination of the reserved bits as a version field. Given that reserved bits have proven over the years to be hot real estate, GPE is not using an explicit version field, but does support versioning.

All of those features come with a cost, and can be implemented in different ways. GPE is trying to use an approach that is close to VXLAN (and LISP) so that the incremental cost of implementing GPE+VXLAN+LISP on the same device give it a chance to be deployed.
> Please look http://tools.ietf.org/html/draft-herbert-gue-01 for reference.

That's a great write up. Thanks especially for the appendix that articulates very well the motivations that are driving the effort of using the optimization provided by current NICs.

I think the design of the protocol would benefit from a better separation of the network virtualization layer and the metadata layer. 
It would allow each implementation (at the end host or in the network) to implement independently part of the specification, and will eventually help with adoption. I think with a better layering you could also take advantage of other well established features (such as security, for example) that you may want to reuse, rather than reinvent.

Fabio

---

From: Tom Herbert <[email-masked]>
Date: Tue, 25 Mar 2014 11:59:27 -0700

> That's a great write up. Thanks especially for the appendix that articulates
> very well the motivations that are driving the effort of using the
> optimization provided by current NICs.
>
> I think the design of the protocol would benefit from a better separation of
> the network virtualization layer and the metadata layer. It would allow each
> implementation (at the end host or in the network) to implement
> independently part of the specification, and will eventually help with
> adoption. I think with a better layering you could also take advantage of
> other well established features (such as security, for example) that you may
> want to reuse, rather than reinvent.
>
I think you're viewing the meta-data as being a discrete idempotent
pieces of data which are not intrinsic to network virtualization. This
true is some cases, but not in all cases. For example, my first need
for meta-data is for providing integrity and authentication of the
vni. Isolation is a hard requirement in network virtualization, and it
follows that the we need protocols to guarantee that. As I already
pointed out, we cannot rely on the network to be error free nor always
provide adequate security. So I believe that some sort of security
bits are pretty much mandatory to protect the vni in every
encapsulated packet. This becomes inseparable from the vni since the
end host cannot accept the packet without the data to verify the vni.
By the way in GUE, the vni is itself just "meta data", GUE is a
general encapsulation protocol which is not virtualization specific--
this will allow us to use a common encapsulation for different use
cases.

Packet security is also important and this is a case where we do want
to use well established mechanisms (like IPsec). This was an important
consideration in GUE: how can we use security like IPsec but still
allow visibility to the encapsulation data for consumption by devices
in the network.

After security, the next challenge will likely be congestion control.
We need congestion control which can be applied to third party traffic
to bring it into conformance with CC in the native network.
Theoretically, we can insert a DCCP header after GUE, but that seems
like a lot of overkill. This may be a case where additional meta data
is justified.

Tom

---

From: Fabio Maino <[email-masked]>
Date: Wed, 26 Mar 2014 16:44:29 -0700

Tom,
I think we're back to where this conversation started: definition of use 
cases.

Why the vni authentication tag you're proposing couldn't be part of a 
metadata header? The network may be able and willing to use the vni for 
routing, but won't certainly be able to use the vni authentication tag 
whose key is known only to the endpoints. Those kind of questions will 
drive the design. I think it's hard to ask implementations to bear the 
cost of metadata that may or may not be used without articulating for 
which use cases the metadata will be used.

Also, I'm not saying that all the use cases can be addressed by existing 
mechanisms, but I suspect that some are. You'll be looking at payload 
confidentiality, as another security use case. I guess it would be a 
good idea to look at IPsec (or maybe MACsec for a L2 overlay) for that 
one, and not reinvent the whole shebang at network virtualization 
layer... or maybe not.  One way to find out is to articulate the use case.

Fabio

---

From: Tom Herbert <[email-masked]>
Date: Wed, 26 Mar 2014 17:23:02 -0700

On Wed, Mar 26, 2014 at 4:44 PM, Fabio Maino <[email-masked]> wrote:
> routing, but won't certainly be able to use the vni authentication tag whose
> key is known only to the endpoints. Those kind of questions will drive the
> design. I think it's hard to ask implementations to bear the cost of
> metadata that may or may not be used without articulating for which use
> cases the metadata will be used.
>
I think my use case is pretty clear, we need the ability to validate a
vni before receiving at an end host. As you point out, this security
information only has meaning to end points, not to routing. I'm not
sure what "part of a metadata header" actually means, if this is a
reference to SFC, an actual example of how this could be accomplished
would be quite helpful!

> confidentiality, as another security use case. I guess it would be a good
> idea to look at IPsec (or maybe MACsec for a L2 overlay) for that one, and
> not reinvent the whole shebang at network virtualization layer... or maybe
> not.  One way to find out is to articulate the use case.
>
We need the ability to encrypt and authenticate packets sent on the
overlay network. A secondary concern is keeping the virtualization
data (e.g. vni) in clear text so we can still route based on it in the
network. Here is the text from GUE which describes a way to accomplish
this using IPsec:

"
   We expect that GUE may be used to encapsulate IPsec packets. This
   allows the benefits of deriving a flow hash for the inner,
   potentially encrypted, packet. In this case the protocol stack may
   be:

   +-----------------------------+
   |                               |
   |        UDP/IP header |
   |                               |
   |-------------------------------|
   |                               |
   |         GUE Header    |
   |                               |
   |-------------------------------|
   |                               |
   | ESP/AH/private security   |
   |                               |
   |-------------------------------|
   |                               |
   |   Encapsulated packet       |
   |                               |
   +-------------------------------+

   Note that the security does not cover the GUE header (does not
   authenticate it for instance). The GUE security field may be used to
   provide authentication or integrity of the GUE header.
"

Tom

>>> articulates

---

From: Fabio Maino <[email-masked]>
Date: Thu, 27 Mar 2014 15:30:51 -0700

> reference to SFC, an actual example of how this could be accomplished

that's not how SFC is defined today, because it's trying to address 
another use case.

> network.

One interesting aspect is that when the inner payload is IPsec 
encrypted, the integrity of the vni becomes less critical. It's not 
addressing all use cases, but in many it helps.

By the way, I'm not sure I understand where's the inner IP header in the 
picture below. Inside or outside the ESP payload?

Fabio

> Here is the text from GUE which describes a way to accomplish
> this using IPsec:
>
> "

---

From: Tom Herbert <[email-masked]>
Date: Thu, 27 Mar 2014 16:25:44 -0700

> One interesting aspect is that when the inner payload is IPsec encrypted,
> the integrity of the vni becomes less critical. It's not addressing all use
> cases, but in many it helps.
>
Somewhat, but I imagine that the SAs would most likely between
physical machines (not per vni) in which case a corrupted vni is still
a problem.

> picture below. Inside or outside the ESP payload?
Inside, next header for ESP is 0x4 or 0x29.

>>>>> separation
>>>>> of
>>>>> other well established features (such as security, for example) that
>>>>> you
>>>>> may

---

From: Fabio Maino <[email-masked]>
Date: Fri, 28 Mar 2014 10:59:22 -0700

> Inside, next header for ESP is 0x4 or 0x29.

I see, not exactly the way IPsec uses ESP, but I understand what you're 
trying to do.

In our deployments we have used IPsec tunnel mode there, that introduces 
some redundancy, but allows to reuse quite a few pieces of existing 
implementations.

If you have  a "transport independent" version of ESP you could apply 
that transformation right after the network virtualization header, and 
protect the metadata together with the payload. Using an authenticated 
encryption mode with support for Additional Authenticated Data you could 
selectively provide integrity protection or confidentiality+integrity 
protection  to different portions of the metadata.

Fabio