Subject: Re: [nvo3] Consensus call on encap proposals

From: Lizhong Jin <[email-masked]>
Date: Mon, 8 Aug 2016 15:43:52 +0800

Hi,
With my design experience of NIC and Switch, I prefer VXLAN-GPE. I have the
same concern of HW complicated logic from Fabio, and additional concern to
GENEVE and GUE is its long size of header.
1. GENEVE: 256+8=262Bytes
2. GUE: 128+4=132Bytes
In many current switch and NIC design, the parser buffer size is still
128Bytes, and some NIC/DMA has 256Bytes buffer. This is workable because:
1. IPv4 options is not widely used.
2. IPv6 extension header only support with limited number.
But if adding GENEVE/GUE header, the minimum size of buffer is 256Bytes, or
even 512Bytes. Then the question is, does the hardware need to process
these Variable Length Options/Optional Fields/Private Data. If not, then it
is not reasonable to force the hardware to increase the cost, but gain
little.

Lizhong

> ---------- Forwarded message ----------
> From: Alia Atlas <[email-masked]>
> To: NVO3 <[email-masked]>
> Cc: "Bocci, Matthew (Nokia - GB)" <[email-masked]>
> Date: Fri, 29 Jul 2016 11:13:00 -0400
> Subject: Re: [nvo3] Consensus call on encap proposals
> I'd like to have people focus on the key point of this thread.
>
> Are there serious technical objections (and specifically what are they) to
> moving forward with VXLAN-GPE as the standards-track protocol?
>
> Are there serious technical objections (and specifically what are they) to
> moving forward with GENEVE as the standards-track protocol?
>
> Are there serious technical objections (and specifically what are they) to
> moving forward with GUE as the standards-track protocol?
>
> We need to capture any relevant objections.  So far, there's been some
> discussion on extensibility - with Tom Herbert providing concrete concerns.
>
> I have concluded that almost all the authors would prefer to have no
> standards track solution if they can't guarantee that theirs is that
> standard.
>
> I do hear concerns about whether a decision will be too late.   I think
> that a decision can only be helpful.   It goes back to when is the best
> time to plant a tree - with the answer of 20 years ago or now.
>
> Regards,
> Alia
>
>
> On Fri, Jul 29, 2016 at 4:34 AM, Naoki Matsuhira <[email-masked]
> > wrote:
>
>>
>>
>> On 2016/07/21 23:56, Bocci, Matthew (Nokia - GB) wrote:
>>
>>> WG
>>>
>>> There was a discussion in the NVO3 WG meeting in Berlin following strong
>>> advice from our Area Director that we need to come to a consensus on
>>> converging on a common encapsulation. Two sets of questions were asked:
>>> (1) Should the WG move forward with one standards track encap?
>>> (2) For a given encap, do you have significant technical objections?
>>>
>>
>> I want to inform to this mailing list that I proposed ME6E-FP and ME6E-PR
>> at the yokohama meeting. I also have proposal M46E-FP and M46E-PR (past
>> called SA46T).
>>
>> These encapsulation technologies are based on address mapping. ME6E use
>> IPv6 address which mapping MAC address, and M46E use IPv6 address which
>> mapping IPv4 address.
>>
>> I understand too many encapsulation technologies, however these my
>> proposal are simple, and may contribute to the Internet.
>>
>> I believe address mapping approach is unique, so I want to propose again.
>>
>> sorry not the answer to the question.
>>
>> Naoki Matsuhira
>>
>>
>> _______________________________________________
>> nvo3 mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/nvo3
>>
>
>
>

---

From: Patrick Frejborg <[email-masked]>
Date: Tue, 9 Aug 2016 08:43:14 +0300

Hi,

perhaps we need to take a step back, look on the overall architecture
and see how challenges like this has been solved in the past. Think
that an NVO3 architecture have many similarities with a communication
systems, so what the Audio-Video Transport WG came up with was roughly
this:

SIP
- register endpoints to the system
- finding out capabilities of the endpoints
- setting up sessions between the endpoints

RTP
- streaming between endpoints
- optimized for hardware offload
- several RFCs exists
- UDP the preferred transport protocol
- extensions should not be implemented, from RFC 3550, section 5.3.1
"   Note that this header extension is intended only for limited use.
   Most potential uses of this mechanism would be better done another
   way, using the methods described in the previous section."
- SRTP, encrypted version

RTCP
- sister protocol to RTP
- control packets between the endpoints, on the data path
- SRTCP, encrypted version

Taking that approach and apply it on NVO3 would give us

Overlay Controller Protocol (OCP)
- register VTEPs to the system
- finding out capabilities of the VTEPs
- setting up tunnels between the VTEPs

Overlay Tunnel Protocol (OTP)
- "streaming" between VTEPs
- optimized for hardware offload
- UDP the preferred transport protocol
- extensions should not be implemented
- encrypted version(s)

Overlay Tunnel Control Protocol (OTCP)
- sister protocol to OTP
- control packets between the VTEPs, on the data path
- encrypted version

What do we have at the moment, please add if I miss something or got it wrong

OCP
- BESS and LISP WG on standards tracks
- some proprietary solutions
- OpenStack with OVN

OTP
- MPLS, L2TPv3, GRE etc on standards track but not implemented in NVO3
solutions.
- VXLAN/LISP, NVGRE, STT, GUE, Geneve etc

OTCP
Missing, instead each OTP is trying to have its own extension field
for various use cases. Some of those extension use cases should be
solved by the OCP but since there are several OCPs deployed today and
their architecture might not be able to handle all needed uses case,
it might be more convenient to add an extensions header for a
preferred OTP. But that interferes with hardware offloading
implementations. The question is, is it possible to leave the OTPs as
much as possible without extensions headers, optimized for hardware
offloading, and instead create a common NVO3 OTCP that all OCPs and
OTPs can leverage?
The OTP should stay as much as possible in the data forwarding path
but an OTCP packet should/can be ignored by VTEP's hardware offloading
mechanism and punted to the control plane of the VTEP. Perhaps NVO3
OAM could be one TLV subset here? As an NVO3 admin I would like to
have a NVO3 traceroute for the VTEP and underlay switches/routes, so
when a customer complains I could issue NVO3 traceroute, copying the
UDP header for the encap used by the customer, to figure out on which
ECMP path in the underlay network the customer traffic is on. And a
NVO3 ping would be useful to to check path MTU in the ECMP network.

Regards,
Patrick

On Mon, Aug 8, 2016 at 10:43 AM, Lizhong Jin <[email-masked]> wrote:
> even 512Bytes. Then the question is, does the hardware need to process these
> Variable Length Options/Optional Fields/Private Data. If not, then it is not
> reasonable to force the hardware to increase the cost, but gain little.
>
> Lizhong
>
>
>
>>
>> that a decision can only be helpful.   It goes back to when is the best time
>> to plant a tree - with the answer of 20 years ago or now.
>>
>> Regards,
>> Alia
>>
>>
>> On Fri, Jul 29, 2016 at 4:34 AM, Naoki Matsuhira
>> <[email-masked]> wrote:
>>>
>>>
>>>

---

From: Joe Touch <[email-masked]>
Date: Tue, 9 Aug 2016 08:55:28 -0700

FWIW, I don't think we should be designing ANY protocols in the IETF
based solely on the limitations of existing hardware. By the time the
doc is issued, this hardware is very likely to be in a recycling bin.

Joe

On 8/8/2016 12:43 AM, Lizhong Jin wrote:
> Hi,
> With my design experience of NIC and Switch, I prefer VXLAN-GPE. I
> have the same concern of HW complicated logic from Fabio, and
> additional concern to GENEVE and GUE is its long size of header.
> But if adding GENEVE/GUE header, the minimum size of buffer is
> 256Bytes, or even 512Bytes. Then the question is, does the hardware
> need to process these Variable Length Options/Optional Fields/Private
> Data. If not, then it is not reasonable to force the hardware to
> increase the cost, but gain little.
>
> Lizhong
>
>
>
>
>     From: Alia Atlas <[email-masked] <mailto:[email-masked]>>
>     To: NVO3 <[email-masked] <mailto:[email-masked]>>
>     Cc: "Bocci, Matthew (Nokia - GB)" <[email-masked]
>     <mailto:[email-masked]>>
>     Are there serious technical objections (and specifically what are
>     they) to moving forward with VXLAN-GPE as the standards-track
>     protocol?
>
>     Are there serious technical objections (and specifically what are
>     they) to moving forward with GENEVE as the standards-track protocol?
>
>     Are there serious technical objections (and specifically what are
>     they) to moving forward with GUE as the standards-track protocol?
>
>     We need to capture any relevant objections.  So far, there's been
>     some discussion on extensibility - with Tom Herbert providing
>     concrete concerns.
>
>     I have concluded that almost all the authors would prefer to have
>     no standards track solution if they can't guarantee that theirs is
>     that standard.
>
>     I do hear concerns about whether a decision will be too late.   I
>     think that a decision can only be helpful.   It goes back to when
>     is the best time to plant a tree - with the answer of 20 years ago
>     or now.
>
>     Regards,
>     Alia
>
>
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
>
>
>             There was a discussion in the NVO3 WG meeting in Berlin
>             following strong advice from our Area Director that we
>             need to come to a consensus on converging on a common
>             encapsulation. Two sets of questions were asked:
>         I want to inform to this mailing list that I proposed ME6E-FP
>         and ME6E-PR at the yokohama meeting. I also have proposal
>         M46E-FP and M46E-PR (past called SA46T).
>
>         These encapsulation technologies are based on address mapping.
>         ME6E use IPv6 address which mapping MAC address, and M46E use
>         IPv6 address which mapping IPv4 address.
>
>         I understand too many encapsulation technologies, however
>         these my proposal are simple, and may contribute to the Internet.
>
>         [email-masked] <mailto:[email-masked]>
>         <https://www.ietf.org/mailman/listinfo/nvo3>
>
>
>
>
>
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 9 Aug 2016 09:06:57 -0700

On 8/8/2016 10:43 PM, Patrick Frejborg wrote:
> and see how challenges like this has been solved in the past.
www.isi.edu/x-bone

> Think

FWIW, X-Bone was developed two decades ago as a kind of
"teleconferencing for routers".

> - setting up sessions between the endpoints
We had one protocol that corresponds to SIP (x-bone-ctl).

Note that we had a separate API protocol (x-bone-api).

> - SRTP, encrypted version

We used two-layer IP-in-IP encapsulation to virtualize the link and
network layers. The virtual link layer was necessary to support
"revisititation", in which a single physical node represents multiple
virtual nodes.

> - SRTCP, encrypted version
It's not clear why this is distinct. Aspects of management and reporting
of the entire overlay can use the SIP equivalent. Aspects of managing
and monitoring the channel between the endpoints should happen in-band,
within the RTP equivalent IMO.

Joe

---

From: Patrick Frejborg <[email-masked]>
Date: Tue, 9 Aug 2016 23:02:42 +0300

Whoa!
Will indeed have a look on URL you posted, thanks!

RTP is perhaps the wrong reference, I should have referred the
different overlay tunnels as NVO3 "codecs", the payload inside the
RTP, the audio/video profile identifier. Then these overlay tunnel
protocols can evolve, as the DSPs evolved/evolving in the
communication systems. What do we gain here then?
- no forklift of the whole network when a new NVO3 "codec" is introduced
- use the best "codec" for your VTEP tunnel adjacencies, e.g. between
computer nodes GUE, Geneve whatever provides the best performance in
that version of the hypervisor/NIC combination. If you are connected
to the "PSTN" and your external network is based on MPLS, then go for
MPLS over UDP or your external network is LISP capable then go for
VXLAN - these gateways will most likely do their NVO3 encaps in
hardware. Hopefully one day, the hypervisors would support MPLS (two
labels) and the OCP can setup the LSP between the hypervisor and
underlay network without running LDP between the hypervisor and
underlay network. With that we could avoid the "transcoding" scenarios
described in this draft, e.g. VXLAN-MPLS-VXLAN
https://tools.ietf.org/html/draft-ietf-bess-dci-evpn-overlay-03
And having different extension versions to get translated between
those different NVO3 "codecs" will create some hurdles with upgrades,
most likely :-).

Thus I think it would be better to keep the different NVO3 codecs as
simple as possible, optimized for the NVO3 "DSPs", and have then the
NVO3 extensions in a common sister protocol, which can evolve in its
own pace; those extensions shouldn't need hardware support anyway.
That will depend on the OCP of course, but if there is some feature
that is broken in the OCP and would require extension on every packet
(and hence changes to the NVO3 "DSP"), but there are other OCPs that
can get around this - should we allow that then, choose a single OTP
that gets us into hardware dependencies on the NVO3 "DSP" design and
requires a forklift upgrades every time a new version is released?

I don't know if this kind of scenario is possible with the three
tunnel proposals, this is a hypothetical question without any research
behind - but I am worried that we are only focusing on the overlay
tunnels and not questioning what should go to via the overlay
controller protocol, how can we have several version of hardware
supported overlay tunnels, how can these evolve and should we then
decouple the VTEP control messages from the hardware dependencies and
those control messages should be able to support several overlay
tunnel protocols?

The difference between SIP messages and RTCP is that SIP messages
between a SIP UA and a SIP proxy/registrar do take a different path in
the network what the RTP messages are using - the SIP messages are off
the data path (the signalling path) and the RTCP messages are on the
data path.

Regards,
Patrick

---

From: Joe Touch <[email-masked]>
Date: Tue, 9 Aug 2016 14:05:08 -0700

Hi, Patrick,

On 8/9/2016 1:02 PM, Patrick Frejborg wrote:
> communication systems. 
Sure, but then:

    SIP == x-bone-api
        i.e., it's the way the user indicates what to deploy

    RTP = x-bone-ctl
        i.e., the way the resources are discovered and components configured

You don't need a protocol to specifically deploy just tunnels; they're
just one part of the overall picture (which includes addressing,
routing, etc.).
...
> between a SIP UA and a SIP proxy/registrar 
Yes, as per above, SIP is closer to x-bone-api (I can't speak for the
others).

> do take a different path in

RTCP are not in the data path; they're between data endpoints. There's
no requirement that they traverse the same path as the codec'd data.

Joe

---

From: Lizhong Jin <[email-masked]>
Date: Wed, 10 Aug 2016 08:55:01 +0800

Right, but currently we are discussing which solution is better? Then it is valuable to consider the existing/potential hardware implementation, and get pros/cons.

Regards

Lizhong

On 08/09/2016 23:56, Joe Touch wrote:

FWIW, I don't think we should be designing ANY protocols in the
IETF based solely on the limitations of existing hardware. By the
time the doc is issued, this hardware is very likely to be in a
recycling bin.

Joe

On 8/8/2016 12:43 AM, Lizhong Jin
wrote:

Hi,

With my design experience of NIC and Switch, I
prefer VXLAN-GPE. I have the same concern of HW
complicated logic from Fabio, and additional
concern to GENEVE and GUE is its long size of
header.

2. GUE: 128+4=132Bytes

In many current switch and NIC design, the parser buffer
size is still 128Bytes, and some NIC/DMA has 256Bytes
buffer. This is workable because:

But if adding GENEVE/GUE header, the minimum size of buffer is
256Bytes, or even 512Bytes. Then the question is, does the
hardware need to process these Variable Length
Options/Optional Fields/Private Data. If not, then it is not
reasonable to force the hardware to increase the cost, but
gain little.

Lizhong

---------- Forwarded message
----------

To: NVO3 <[email-masked]>

Cc: "Bocci, Matthew (Nokia - GB)"
<[email-masked]>

Date: Fri, 29 Jul 2016 11:13:00
-0400

Subject: Re: [nvo3] Consensus call
on encap proposals

I'd like to have
people focus on the key point of
this thread.

Are there serious technical
objections (and specifically
what are they) to moving
forward with VXLAN-GPE as the
standards-track protocol?

Are there serious technical
objections (and specifically
what are they) to moving
forward with GENEVE as the
standards-track protocol?

Are there serious technical
objections (and specifically
what are they) to moving
forward with GUE as the
standards-track protocol?

We need to capture any
relevant objections.  So far,
there's been some discussion
on extensibility - with Tom
Herbert providing concrete
concerns.

I have concluded that
almost all the authors would
prefer to have no standards
track solution if they can't
guarantee that theirs is that
standard.

I do hear concerns about
whether a decision will be too
late.   I think that a
decision can only be helpful.
It goes back to when is the
best time to plant a tree -
with the answer of 20 years
ago or now.

Regards,

Alia

On Fri,
Jul 29, 2016 at 4:34 AM, Naoki
Matsuhira <[email-masked]>
wrote:

On 2016/07/21 23:56,
Bocci, Matthew (Nokia -
GB) wrote:

WG

There was a discussion
in the NVO3 WG meeting
in Berlin following
strong advice from our
Area Director that we
need to come to a
consensus on converging
on a common
encapsulation. Two sets
of questions were asked:

(1) Should the WG move
forward with one
standards track encap?

(2) For a given encap,
do you have significant
technical objections?

I want to inform to this
mailing list that I proposed
ME6E-FP and ME6E-PR at the
yokohama meeting. I also
have proposal M46E-FP and
M46E-PR (past called SA46T).

These encapsulation
technologies are based on
address mapping. ME6E use
IPv6 address which mapping
MAC address, and M46E use
IPv6 address which mapping
IPv4 address.

I understand too many
encapsulation technologies,
however these my proposal
are simple, and may
contribute to the Internet.

I believe address mapping
approach is unique, so I
want to propose again.

sorry not the answer to the
question.

Naoki Matsuhira

nvo3 mailing list

[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Tue, 9 Aug 2016 18:03:00 -0700

> On Aug 9, 2016, at 5:55 PM, Lizhong Jin <[email-masked]> wrote:
> 
> Right, but currently we are discussing which solution is better?

Yes, but...

> Then it is valuable to consider the existing/potential hardware implementation, and get pros/cons.

Future yes. Existing, no. We those deployed devices will be gone and we'll be left with their limitations. 

Standards should drive the future and should not be hobbled by the past. 

Joe

> FWIW, I don't think we should be designing ANY protocols in the IETF based solely on the limitations of existing hardware. By the time the doc is issued, this hardware is very likely to be in a recycling bin.
> 
> Joe
> 
>> With my design experience of NIC and Switch, I prefer VXLAN-GPE. I have the same concern of HW complicated logic from Fabio, and additional concern to GENEVE and GUE is its long size of header.
>> In many current switch and NIC design, the parser buffer size is still 128Bytes, and some NIC/DMA has 256Bytes buffer. This is workable because:
>> But if adding GENEVE/GUE header, the minimum size of buffer is 256Bytes, or even 512Bytes. Then the question is, does the hardware need to process these Variable Length Options/Optional Fields/Private Data. If not, then it is not reasonable to force the hardware to increase the cost, but gain little.
>> 
>> Lizhong
>> 
>> 
>> 
>>> 
>>> Are there serious technical objections (and specifically what are they) to moving forward with VXLAN-GPE as the standards-track protocol?
>>> 
>>> Are there serious technical objections (and specifically what are they) to moving forward with GENEVE as the standards-track protocol?
>>> 
>>> Are there serious technical objections (and specifically what are they) to moving forward with GUE as the standards-track protocol?
>>> 
>>> We need to capture any relevant objections.  So far, there's been some discussion on extensibility - with Tom Herbert providing concrete concerns.
>>> 
>>> I have concluded that almost all the authors would prefer to have no standards track solution if they can't guarantee that theirs is that standard.
>>> 
>>> I do hear concerns about whether a decision will be too late.   I think that a decision can only be helpful.   It goes back to when is the best time to plant a tree - with the answer of 20 years ago or now.
>>> 
>>> Regards,
>>> Alia
>>> 
>>> 
>>>> On Fri, Jul 29, 2016 at 4:34 AM, Naoki Matsuhira <[email-masked]> wrote:
>>>> 
>>>> 
>>>>> There was a discussion in the NVO3 WG meeting in Berlin following strong advice from our Area Director that we need to come to a consensus on converging on a common encapsulation. Two sets of questions were asked:
>>>> I want to inform to this mailing list that I proposed ME6E-FP and ME6E-PR at the yokohama meeting. I also have proposal M46E-FP and M46E-PR (past called SA46T).
>>>> 
>>>> These encapsulation technologies are based on address mapping. ME6E use IPv6 address which mapping MAC address, and M46E use IPv6 address which mapping IPv4 address.
>>>> 
>>>> I understand too many encapsulation technologies, however these my proposal are simple, and may contribute to the Internet.
>>>>

---

From: Alia Atlas <[email-masked]>
Date: Tue, 9 Aug 2016 21:28:17 -0400

Joe,

In the past 15+ years, I haven't seen this limitation going away.
It is expensive to pass the whole packet through the packet forwarding
logic.
To remove it would require significantly faster I/O to the packet
forwarding logic,
instead of being able to stream most of the packet to memory to be reunited
with
the packet header after the lookup logic.

Given that this is frequently a basic attribute of a system's architecture,
expecting
it to change drastically to handle an encapsulation without stunning
technical advantage
is rather unlikely.

If this concern is what is behind the "too extensible" comments, then it is
a significant
technical objection.  This isn't a question of it being simply slightly
harder to implement
or having different opinions on the trade-offs of different types of
extensibility.

It is one thing to have to reprogram an FPGA, write new micro-code
for a network processor or even respin an ASIC.  It is another thing to
need to change the
basic architecture of a system.

To my mind, this isn't something that should be dismissed with a "oh,
systems will just get
better".  This is and has been a common architecture approach taken as
routers have become
hardware based.  That doesn't mean that everyone with every system has the
issue.

The question is for significant technical objections - and the
encapsulation not being implementable
on many existing architectures is highly relevant.

I appreciate Lizhong being so clear and direct about the issue.

Regards,
Alia

On Tue, Aug 9, 2016 at 9:03 PM, Joe Touch <[email-masked]> wrote:

> On 08/09/2016 23:56, Joe Touch <[email-masked]> wrote:
>
> FWIW, I don't think we should be designing ANY protocols in the IETF based
> solely on the limitations of existing hardware. By the time the doc is
> issued, this hardware is very likely to be in a recycling bin.
>
> Joe
>
> With my design experience of NIC and Switch, I prefer VXLAN-GPE. I have
> the same concern of HW complicated logic from Fabio, and additional concern
> to GENEVE and GUE is its long size of header.
> But if adding GENEVE/GUE header, the minimum size of buffer is 256Bytes,
> or even 512Bytes. Then the question is, does the hardware need to process
>> Are there serious technical objections (and specifically what are they)
>> to moving forward with VXLAN-GPE as the standards-track protocol?
>>
>> Are there serious technical objections (and specifically what are they)
>> to moving forward with GENEVE as the standards-track protocol?
>>
>> Are there serious technical objections (and specifically what are they)
>> to moving forward with GUE as the standards-track protocol?
>>
>>>> There was a discussion in the NVO3 WG meeting in Berlin following
>>>> strong advice from our Area Director that we need to come to a consensus on
>>> I want to inform to this mailing list that I proposed ME6E-FP and
>>> ME6E-PR at the yokohama meeting. I also have proposal M46E-FP and M46E-PR
>>> (past called SA46T).
>>>
> nvo3 mailing [email-masked]://www.ietf.org/mailman/listinfo/nvo3
>
>
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 9 Aug 2016 18:40:49 -0700

> In the past 15+ years, I haven't seen this limitation going away.

If you want to base a limitation on a prediction of future hardware,
then do that.

But don't simply base it on existing hardware.

There's zero point to an RFC that is outdated before it is published.

> It is expensive to pass the whole packet through the packet forwarding

No disagreement, but that doesn't necessarily mean that future hardware
will be as shallow into the packet as current hardware either.

> ...
>
> Given that this is frequently a basic attribute of a system's
> architecture, expecting

But that's exactly what I do expect. Encapsulation is becoming more
prevalent, as is DPI. The deeper things get, the deeper hardware WILL
end up looking into the packet.

I understand making the system easy to implement in hardware in general,
but we really need to NOT design protocols that are obsolete even before
the ink is dry.

Joe

---

From: Alia Atlas <[email-masked]>
Date: Tue, 9 Aug 2016 21:54:28 -0400

Joe,

> but we really need to NOT design protocols that are obsolete even before

Hyperbole is great.  Recall that we're talking about a choice between 3
protocols
where the highest technical advantage is extensibility or ability to be
implemented
on many common router architectures.

There has to be a plausible reason for folks to actually implement whatever
encapsulation is done.  The IETF needs to be practical in its selections.
Running
code (or hardware) in this case does matter - not just speculative future
hardware.

Regards,
Alia

---

From: Tom Herbert <[email-masked]>
Date: Tue, 9 Aug 2016 19:27:48 -0700

> code (or hardware) in this case does matter - not just speculative future
We had many discussions in the Encapsulation Design team on the
particular issue of "HW friendliness" (draft-ietf-rtgwg-dt-encap-01).
One of the recommendations we came up with is "Keep the encap header
small. Switches and routers usually only read the first small number
of bytes into the fast memory for quick processing and easy
manipulation.". Note that this is pretty vague, we don't specify
exactly what the small number should be. This was because we could not
find any consensus among hardware vendors as to what the limit should
be; some said the limit is 128, some said 256 bytes, still others said
no limit. This theme was true for most of the hardware guidelines we
produced. For instance, I asked five different HW experts what they
thought of variable length headers and got five different answers
ranging from they're a major problem in hardware to being no problem
at all. I agree we need to be practical, but neither does it seem like
we should constrain protocols to the least common denominator of
current hardware support.

Tom

---

From: Alia Atlas <[email-masked]>
Date: Tue, 9 Aug 2016 22:41:15 -0400

Tom,

> current hardware support.

Yes, it isn't clear what the limits are because it varies based on vendor
and even particular system.   Clearly, some limit is necessary and a concern
for a common standard.   Powers of 2 are always popular with hardware
and software designers.

Regards,
Alia

---

From: Patrick Frejborg <[email-masked]>
Date: Wed, 10 Aug 2016 09:19:47 +0300

Hi Joe,

I should have used Overlay Controller Schemes instead of OCP, all
current overlay solution do have some "signalling" mechanisms to setup
and tear down the tunnels, a bunch of protocols. Probably you could
make extensions to SIP as well and have that as an Overlay Controller
Scheme (OCS), that would be on standards tracks.

You are right about RTCP, it is not in the data path/inside the stream
- it is between endpoints and de facto in the data path, otherwise it
would be useless collecting statistics for the codec to detect
transmissions faults and jitter etc.

Why not do the similar for NVO3, design OTCP so that the OAM is active
on all ECMP paths between two VTEPs though there is only one active
end system stream between the VTEPs. And the OAM packets do not need
to carry the header&payload of whatever encapsulation protocol is
used, decouple that so you can use whatever encapsulation protocol fit
your needs. Have mandatory TLVs here (OAM, traceroute, ping etc) and
have optional TLVs so if there is some OCS that have problems to send
their own extension over the signalling path between VTEPs then that
OCS can use this protocol to send state or whatever information is
needed between the VTEPs - without introducing new hardware encoding
offloading tweaks for everyone or breaking them by some new strange
information.

Regards,
Patrick

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Aug 2016 08:15:03 -0700

On 8/9/2016 11:19 PM, Patrick Frejborg wrote:
> end system stream between the VTEPs. ...
RTCP needs a new separate mechanism because teleconferencing needs
different measurements than would typically be gathered to run a net.

Overlays don't. Why not just run whatever measurement/monitoring that
could run in the underlay?

If you do have a need for a different mechanism, you need to decide how
much is specific to what makes an overlay different and whether that
belongs in the controller level or within individual tunnels (who would
run automatically and be monitored as an entity).

Finally, based on the design of our control protocol in X-Bone, I don''t
think using SIP as anything but general inspiration is useful here.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Aug 2016 08:20:04 -0700

On 8/9/2016 6:54 PM, Alia Atlas wrote:
...
> Recall that we're talking about a choice between 3 protocols
> where the highest technical advantage is extensibility or ability to
> be implemented

IMO, the highest advantage is widest participation.

Extensibility reduces flux that can impede participation, but allows for
long-term stability that improves participation.

Ease of implementation increases participation at the outset but can
require ossifying aspects that impede later participation.

I agree these two tensions should be balanced.

> encapsulation is done.  The IETF needs to be practical in its
> selections.  Running
> code (or hardware) in this case does matter - not just speculative
> future hardware.

Not truly speculative, but reasonable predictions are useful.

Had we limited ourselves to only protocols that were efficiently
implemented in the existing hardware, we never would have proposed IPv4
or TCP.

Joe

---

From: Pat Thaler <[email-masked]>
Date: Wed, 10 Aug 2016 10:08:31 -0700

+1

On Tue, Aug 9, 2016 at 6:28 PM, Alia Atlas <[email-masked]> wrote:

> instead of being able to stream most of the packet to memory to be
> reunited with

I agree. Furthermore, for data center use cases, this limitation has gotten
stronger rather than weaker due to strong requirements on reducing latency
for those use cases.  Handling more bytes in the forwarding logic increases
the latency.

> If this concern is what is behind the "too extensible" comments, then it
> is a significant
>> based solely on the limitations of existing hardware. By the time the doc
>> is issued, this hardware is very likely to be in a recycling bin.
>>
>> Joe
>>

---

From: Patrick Frejborg <[email-masked]>
Date: Wed, 10 Aug 2016 20:47:14 +0300

Hi Joe,

I have not stated that we should use RTCP/SIP as such, as you said
"using SIP as anything but general inspiration is useful here" is
totally correct. The VTEPs will more and more be deployed inside
hypervisors, hence running only this RTCP stylish monitoring protocol
(or other underlay monitoring solutions) only inside the underlay is
not good enough - it has to be deployed between VTEPs. If you
integrate that inside overlay tunnels and there are a lot of tunnels
between the VTEPs, then you most likely have a lot of OAMs going
between the VTEPs, doing the duplicate job for the same path because
they are integrated into the streams. That OAM has to be managed by
the VTEPs - I have no idea how much that will burden the VTEP's
resources. Would it be more resource efficient to figure out a
mechanism so that you can setup OAM sessions between the hypervisor
VTEPs, covering all possibly ECMP paths through the underlay and no
more, regardless of how many overlay tunnels exists between the VTEPs?
And if that can be done, then you could add mandatory and optional
TLVs for other functions that can not be handled by the overlay
controller to be sent between the VTEPs - all this shouldn't interfere
hardware offloading for the end system streams between the VTEPs.

Regards,
Patrick

---

From: Joe Touch <[email-masked]>
Date: Wed, 10 Aug 2016 16:28:02 -0700

On 8/10/2016 10:47 AM, Patrick Frejborg wrote:
> totally correct. 
My misinderstanding; glad we're on the same page.

> The VTEPs will more and more be deployed inside
> not good enough - it has to be deployed between VTEPs. 

I'm not clear on how it matters where you run any of this - the network
should never care.

I'm not suggesting you run the monitoring only in the underlay; I'm
suggesting that whatever monitoring you run in the overlay can be
basically the same as what you would run in an underlay. I.e., I'm not
clear that overlays need new kinds of overlay-specific monitoring -
except for endpoint liveness, which the control protocol ought to be
doing anyway (FWIW, this is how we solved this issue in the X-Bone
system - the control protocol had heartbeats for liveness and all other
monitoring was done on the overlay itself).
> If you
> they are integrated into the streams. 
Yes, you'd have lots of OAMs between the endpoints, but each could be
for a different tunnel. Each tunnel might be treated differently by the
underlay, so that kind of per-tunnel OAM is required IMO anyway. And the
overhead of per-tunnel OAM ought to be small compared to the tunnel
traffic (by design) anyway.

> That OAM has to be managed by
You can't do that unless you know how to control ECMP path
differentiation in the underlay, which is rarely known.

> hardware offloading for the end system streams between the VTEPs.
OAM is control plane and thus typically in the software path anyway. All
you need is a way to know whether to pass the traffic to the controller
or not at the endpoints, and the encapsulation tells you that (you use
the same mechanism that would apply in the underlay to know the
difference between OAM traffic and user traffic).

--

Overall, IMO if you can tell you're on an overlay from within the
overlay, you did it wrong (because someone WILL eventually want to run
an overlay over the overlay too) - our system was capable of doing so
over 15 levels deep. That's why we didn't treat OAM as integral to the
tunneling mechanism.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Wed, 10 Aug 2016 16:34:39 -0700

On Tue, Aug 9, 2016 at 7:41 PM, Alia Atlas <[email-masked]> wrote:
>> > selections.
>> > future
The power of 2 property is a nice simplification, based on that I
believe the minimum length of headers we should expect hardware to be
able to deal with is 256 bytes. I posted an I-D proposing that in
https://tools.ietf.org/html/draft-herbert-int-area-limits-00. Since
this limit applies to all protocol layers and options (e.g. IPv6
extension headers) and not just items in nvo3, this might be more of a
discussion to have in int-area?

Thanks,
Tom

---

From: Patrick Frejborg <[email-masked]>
Date: Thu, 11 Aug 2016 11:57:49 +0300

Hi Joe,

think we are on the same page - your experience regarding OAM with
X-bone seems to match what I think on how it should be used in NVO3.
And if that is the case, then we have a need of a control protocol
between the VTEPs, here we can then add different extensions and keep
the overlay streams free from becoming cumbersome to implement for the
hardware architects/designers.

Continuing to think out of the box, exploring if we could make the
ECMP underlay network less cloudy for the VTEPs and provide some more
visibility of the paths within the underlay network by this overlay
tunnel control plane protocol (OTCP).
Lets say you have underlay network with for spine switches, we have
this OTCP defined and since it is a control plane solution the current
underlay routers and hw VTEPs could be upgraded with a sw release
supporting OTCP (no hw upragde is required). If the underlay network
is hashing by the transport protocol's source port, then perhaps the
VTEPs could discover the four different paths in a two-tier leaf-spine
network with four spines by sending OTCP traceroutes through the
network with different UDP source ports. Once four paths are
discovered, start the OAM over these four paths, if so desired. If you
need some CoS in the network to be provided for the VTEPs - e.g  you
have end systems streams that have subscribed for less overbooked
paths in the underlay and other end systems streams subscribed for
best effort then perhaps the OTCP could have some affinity fields, set
by the spine routers on the ports, to distinguish which paths in the
underlay network are for premium or best effort end systems. If there
is congestion on some of the path, the VTEP discover that by the OAM
mechanism and then it could change the UDP source port and the traffic
is switched to another path. Sure that there could be more use cases -
if this is doable or needed at all, early ideas, to be explored if the
group finds it interesting.

It seems that it would be a good idea to separate the streaming
overlay tunnel encapsulation, optimized for hardware offloading, so
the stream can stay in the data plane (NIC, ASICs), have several
encapsulation schemes supported so we are not closing the innovation
door for forthcoming encapsulation schemes either. OAM needs to be in
the control plane of the VTEPs, should be generic for all
encapsulation schemes, here we could also add extensions and have
other VTEP control plane innovations supported.

Regards,
Patrick

---

From: Joe Touch <[email-masked]>
Date: Thu, 11 Aug 2016 10:29:38 -0700

Some basic notes below...

> tunnel control plane protocol (OTCP).
...
Again, my basic rule is "nothing in the overlay that can't be done in an
underlay".

So I ask: where are the tools by which you can derive this information
for ECMP in the current Internet? If you don't already have them, there
are only two options:
   
    1) ask the underlay to export that info (which IMO isn't going to
happen)

    2) start some research on how to accomplish this.

I don't see this as a WG task either way.

My personal view is that trying to optimize the way in which overlays
use paths in an underlay is a lot like optimizing how virtual memory
uses physical memory. That was a very active area of research in the
1970s when VM was becoming more prevalent, and the community eventually
gave up - realizing that virtualization has its own merits, and that
optimizing the underlying physical system can be nearly impossible,
e.g., when you run software VM in as a virtual machine inside another, etc.

We decided to avoid that issue in the X-Bone for that reason - we chose
to focus only on "empirically visible path properties", e.g.,
reordering, loss, bandwidth, latency, and jitter. Everything else is a
matter of intent - and we didn't care what was intended, only what ended
up happening.

> door for forthcoming encapsulation schemes either.

I agree. I don't see a benefit to requiring a single particular
encapsulation.

>  OAM needs to be in

I agree.

Joe

---

From: Jesse Gross <[email-masked]>
Date: Thu, 11 Aug 2016 23:59:03 +0000

Alia,

This is a self-fulling prophesy. If there are no users of something then, of course, it will never advance. However, you can’t then use this to argue that it should not be used because there have been no advancements.

The most common example given in this area is the use of IP options. At the time that routing was moving to hardware implementations, options were not widely used and so were not implemented. However, imagine that options were in common use – do you think that router vendors would have decided that IP was too difficult to implement and abandoned that market? Or would we now be accepting that this is a common element of protocols?

It’s hard for me to believe that the capabilities we’re discussing here are unimplementable. Certainly in the case of NICs, support for Geneve already exists in shipping products, as I recently pointed out. Switching ASICs take a bit longer to mature but, again, there have already been numerous announcements of Geneve support.

Regardless, I don’t believe that protocols should be designed based on the lowest common denominator. At this point, the goal seems to be unification and broadest applicability, however, this does not seem to be useful without focusing on what we are trying to accomplish. As I and others have said in the past, extensibility is a core need and without it, the protocol is equally unimplementable. If there is serious debate about what is required and technically possible then I don’t believe that IETF should attempt to create a standard. Both of these can only be shown through evidence of what actually happens.

Jesse

On 8/9/16, 6:28 PM, "nvo3 on behalf of Alia Atlas" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

Joe,

In the past 15+ years, I haven't seen this limitation going away.
It is expensive to pass the whole packet through the packet forwarding logic.
To remove it would require significantly faster I/O to the packet forwarding logic,
instead of being able to stream most of the packet to memory to be reunited with
the packet header after the lookup logic.

Given that this is frequently a basic attribute of a system's architecture, expecting
it to change drastically to handle an encapsulation without stunning technical advantage
is rather unlikely.

If this concern is what is behind the "too extensible" comments, then it is a significant
technical objection.  This isn't a question of it being simply slightly harder to implement
or having different opinions on the trade-offs of different types of extensibility.

It is one thing to have to reprogram an FPGA, write new micro-code
for a network processor or even respin an ASIC.  It is another thing to need to change the
basic architecture of a system.

To my mind, this isn't something that should be dismissed with a "oh, systems will just get
better".  This is and has been a common architecture approach taken as routers have become
hardware based.  That doesn't mean that everyone with every system has the issue.

The question is for significant technical objections - and the encapsulation not being implementable
on many existing architectures is highly relevant.

Regards,
Alia

On Tue, Aug 9, 2016 at 9:03 PM, Joe Touch <[email-masked]<mailto:[email-masked]>> wrote:

On Aug 9, 2016, at 5:55 PM, Lizhong Jin <[email-masked]<mailto:[email-masked]>> wrote:
Right, but currently we are discussing which solution is better?

Yes, but...

Joe

Regards
Lizhong
On 08/09/2016 23:56, Joe Touch<mailto:[email-masked]> wrote:

Joe

Lizhong

---------- Forwarded message ----------
From: Alia Atlas <[email-masked]<mailto:[email-masked]>>
To: NVO3 <[email-masked]<mailto:[email-masked]>>
Cc: "Bocci, Matthew (Nokia - GB)" <[email-masked]<mailto:[email-masked]>>
Date: Fri, 29 Jul 2016 11:13:00 -0400
Subject: Re: [nvo3] Consensus call on encap proposals

Regards,
Alia

On Fri, Jul 29, 2016 at 4:34 AM, Naoki Matsuhira <[email-masked]<mailto:[email-masked]>> wrote:

Naoki Matsuhira

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

nvo3 mailing list

[email-masked]<mailto:[email-masked]>

---

From: Alia Atlas <[email-masked]>
Date: Thu, 11 Aug 2016 20:20:13 -0400

Jesse,

It's a question of the benefit of the change versus the cost to modify the
hardware.
Changing one FPGA or ASIC or reprogramming a network processor is
significantly cheaper
than redesigning a system architecture.

I certainly agree with and share the frustrations of seeing extensibility
built in and then abandoned
- over and over again.

Regards,
Alia

On Thu, Aug 11, 2016 at 7:59 PM, Jesse Gross <[email-masked]> wrote:

> Alia,
>
>
>
> This is a self-fulling prophesy. If there are no users of something then,
> of course, it will never advance. However, you can’t then use this to argue
> that it should not be used because there have been no advancements.
>
>
>
> The most common example given in this area is the use of IP options. At
> the time that routing was moving to hardware implementations, options were
> not widely used and so were not implemented. However, imagine that options
> were in common use – do you think that router vendors would have decided
> that IP was too difficult to implement and abandoned that market? Or would
> we now be accepting that this is a common element of protocols?
>
>
>
> It’s hard for me to believe that the capabilities we’re discussing here
> are unimplementable. Certainly in the case of NICs, support for Geneve
> already exists in shipping products, as I recently pointed out. Switching
> ASICs take a bit longer to mature but, again, there have already been
> numerous announcements of Geneve support.
>
>
>
> Regardless, I don’t believe that protocols should be designed based on the
> lowest common denominator. At this point, the goal seems to be unification
> and broadest applicability, however, this does not seem to be useful
> without focusing on what we are trying to accomplish. As I and others have
> said in the past, extensibility is a core need and without it, the protocol
> is equally unimplementable. If there is serious debate about what is
> required and technically possible then I don’t believe that IETF should
> attempt to create a standard. Both of these can only be shown through
> evidence of what actually happens.
>
>
>
> Jesse
>
>
>
> On 8/9/16, 6:28 PM, "nvo3 on behalf of Alia Atlas" <[email-masked]
> on behalf of [email-masked]> wrote:
>
>
>
> Joe,
>
>
>

---

From: Jesse Gross <[email-masked]>
Date: Fri, 12 Aug 2016 02:58:53 +0000

Alia,

As I mentioned, we are already seeing numerous implementations of Geneve across all types of platforms. I agree that there is always a cost vs. benefit tradeoff (on all types of implementations!) but I think the facts on the ground are that this is possible today.

Jesse

On 8/11/16, 5:20 PM, "Alia Atlas" <[email-masked]<mailto:[email-masked]>> wrote:

Jesse,

It's a question of the benefit of the change versus the cost to modify the hardware.
Changing one FPGA or ASIC or reprogramming a network processor is significantly cheaper
than redesigning a system architecture.

I certainly agree with and share the frustrations of seeing extensibility built in and then abandoned
- over and over again.

Regards,
Alia

On Thu, Aug 11, 2016 at 7:59 PM, Jesse Gross <[email-masked]<mailto:[email-masked]>> wrote:
Alia,

Jesse

Joe,

Regards,
Alia

Yes, but...

Joe

Regards
Lizhong

Joe

Lizhong

---------- Forwarded message ----------
From: Alia Atlas <[email-masked]<mailto:[email-masked]>>
To: NVO3 <[email-masked]<mailto:[email-masked]>>
Cc: "Bocci, Matthew (Nokia - GB)" <[email-masked]<mailto:[email-masked]>>
Date: Fri, 29 Jul 2016 11:13:00 -0400
Subject: Re: [nvo3] Consensus call on encap proposals

Regards,
Alia

Naoki Matsuhira

nvo3 mailing list

---

From: Tom Herbert <[email-masked]>
Date: Fri, 12 Aug 2016 09:56:21 -0700

> built in and then abandoned
In a large datacenter deployment we need extensibility to deal with
changing conditions and threats. For example, maybe we need add some
bits to the headers that are needed to address a newly discovered
security vulnerability. The worse case scenario is that if we add
those bits it pushes us beyond the limits of HW processing and
extending the protocol is no longer feasible. If a vendor tells us
that we just need to swap out a few hundred thousand parts to get a
larger parsing buffer to move forward it would NOT be an acceptable
answer.

We need to know upfront exactly what the capabilities of the hardware
and the protocol are. If the hardware can't give us a reasonable
guarantee of extensibility, then we'll simply avoid using those
features and don't commit to a hardware solution. Extensibility is a
requirement in the datacenter, hardware support is not-- at best the
latter is just an optimization.

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Aug 2016 10:21:12 -0700

On 8/11/2016 4:59 PM, Jesse Gross wrote:
> The most common example given in this area is the use of IP options.
> At the time that routing was moving to hardware implementations,
> options were not widely used and so were not implemented. However,
> imagine that options were in common use – do you think that router
> vendors would have decided that IP was too difficult to implement and
> abandoned that market? Or would we now be accepting that this is a
> common element of protocols?

A good example from history here are TCP options. In the beginning, they
were considered only a complexity overhead.

Another example is IPsec.

Both, are examples of TLV options (TCP within the TCP header; IPsec as a
'next layer' protocol).

Both are widely supported in hardware.

That's not to suggest that we should focus on TLV solutions, but it goes
to prove that even TLV-format options do not prevent widespread hardware
support.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Fri, 12 Aug 2016 10:54:15 -0700

On Fri, Aug 12, 2016 at 10:21 AM, Joe Touch <[email-masked]> wrote:
> The most common example given in this area is the use of IP options. At the
> time that routing was moving to hardware implementations, options were not
> widely used and so were not implemented. However, imagine that options were
> in common use – do you think that router vendors would have decided that IP
> was too difficult to implement and abandoned that market? Or would we now be
> accepting that this is a common element of protocols?
>
>
> That's not to suggest that we should focus on TLV solutions, but it goes to
> prove that even TLV-format options do not prevent widespread hardware
> support.
>
Joe,

I don't know sure that parsing of TCP options is really all that
widespread, but I do know that the inability of middleboxes to
properly implement support for both IP options and IPv6 extension
headers have pretty much made those non-starters for widespread
deployment.

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Aug 2016 10:59:19 -0700

On 8/12/2016 10:54 AM, Tom Herbert wrote:
> widespread, 
In hardware at the endsystems, it is.

> but I do know that the inability of middleboxes to

IPsec is a counterexample.

For many years, the network device community pushed back on options
because it undercuts their profit model (build for the 80% case and toss
the rest to software slow-path).

The tide there is turning; even the OPS groups are starting to admit
that we can't simply deprecate IP options, but we can do things to
encourage their more widespread support (e.g., limit the chain, etc.).

Middleboxes ignore or misimplement options by design; they''re never
useful as a constraint for protocol design.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Fri, 12 Aug 2016 13:05:39 -0700

> encourage their more widespread support (e.g., limit the chain, etc.).
I hope you're right and extension headers do become deployable someday...

> useful as a constraint for protocol design.

Wouldn't placing a limit on the header length chain as you alluded to
above be an example of such a constraint on protocol design?

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 12 Aug 2016 13:20:52 -0700

> above be an example of such a constraint on protocol design?

The proposal is to change the bound on the chain length in the requirements.

That's different from limiting chain length because middleboxes are
broken or ignore requirements.

Joe