Subject: Re: [tsvwg] [nvo3] VXLAN-GPE Support for ECN propagation between IP headers

From: Bob Briscoe <[email-masked]>
Date: Mon, 19 Jun 2017 12:37:34 +0100

Joe,

[Cutting the distr to nvo3 and adding tsvwg.]

On 16/06/17 20:38, Joe Touch wrote:
>
> Hi, Bob,
>
> This approach seems to assume an integrated implementation of multiple 
> protocol layers.
>
> There are other cases where the "shim" is added indirectly, as a 
> packet is "tunneled" independently over multiple protocols.
>

So what aspect of the following para (in S.3.1) does not include the 
case(s) you are thinking of:

    In some cases a tunnel adds an outer IP header and a tightly coupled
    shim header to an inner header that is not an IP header, but that in
    turn encapsulates an IP header (or might encapsulate an IP header).
    For instance an inner Ethernet (or other link layer) header might
    encapsulate an inner IP header as its payload.  We call this a
    tightly coupled shim over an encapsulating header.

Can you be more specific? Or perhaps give an example? Or suggest an edit 
to the above text?

> I scanned the draft and didn't see this addressed explicitly (maybe I 
> missed it). It's implied throughout  3.2 but never stated as a MAY, 
> e.g., systems MAY support ECN across shim layers but are NOT REQUIRED 
> to do so. Can that sort of explicit caveat be included? This case 
> might be added as a final bullet in the list at the end of section 3.2.
>
Section 3.2 solely gives the cases where ECN is not or cannot be 
supported, so any ECN field in an outer IP header will have to be zeroed 
(typically by configuration because the implementation will not include 
any ECN logic).

If none of the cases listed in S.3.2 applies, then you would be in the 
territory of section 3.1 (i.e. supporting ECN would already be covered 
by RFC6040).

S.3.2 is solely there to plug an important hole that was left in 
RFC6040. RFC6040 said what implementers have to do to support RFC6040, 
but it did not include what /network operators/ have to do if they have 
an implementation that does not support any ECN tunnelling spec 
(RFC6040, RFC4301 or full functionality mode of RFC3168).

Have I misunderstood your point? Could I have made the text clearer?

I only realized S.3.2 was needed when I discovered that there is still 
equipment being built that is copying the 8-bit ToS byte from inner to 
outer as if the Internet Protocol is still as it was before Diffserv was 
standardized 19 years ago (Dec 1998).

I suspect lazy implementers search for "Internet Protocol header" and 
follow the many outdated articles you can find on the Web, rather than 
checking the RFCs. For instance, the 2nd hit from Google (ranked just 
below Wikipedia) erroneously tells us that the 8-bit ToS field contains 
3 precedence bits (that are ignored), 4 ToS bits and 1 currently unused 
bit. I am not going to cite the link here, which will just reinforce its 
ranking.

Bob

> Joe
>
>
> On 6/16/2017 7:25 AM, Bob Briscoe wrote:
>> Fabio, Larry, Uri,
>>
>> I notice VXLAN-GPE is on the IETF standards track.
>> It will need to specify support for propagation of the ECN field 
>> between inner and outer for cases when there is (or might be) an IP 
>> header (v4 or v6) within the L2 encapsulation, and when the outer is 
>> IP (v4 or v6).
>>
>> I believe ECN propagation between inner and outer is already in the 
>> Linux code for VXLAN, but it's not specified in RFC7348, so other 
>> implementations might overlook it. Given that ECN is heavily used in 
>> DCs (e.g. for DCTCP), this is a significant omission.
>>
>> Please refer to draft-ietf-tsvwg-rfc6040update-shim 
>> <https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim> and 
>> RFC6040.
>>
>> I will warn you that, unless you believe that all existing VXLAN 
>> implementations already support ECN,  this will not be just a 
>> straightforward case of referencing the appropriate RFCs. If there 
>> might be non-ECN VXLAN decapsulators out there, the tunnel ingress 
>> will have to know (by config or capability negotiation) whether the 
>> egress supports ECN propagation, and if not, switch to 'compatibility 
>> mode' (zeroing the outer ECN field). That could require definition of 
>> a capability parameter during tunnel set-up (unless you envisage 
>> tunnel set up will always be by administrative config).
>>
>> Cheers
>>
>>
>>
>> Bob
>>
>>
>> -- 
>> ________________________________________________________________
>> Bob Briscoehttp://bobbriscoe.net/
>>
>>
>> _______________________________________________
>> nvo3 mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/nvo3
>
>
>
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Joe Touch <[email-masked]>
Date: Mon, 19 Jun 2017 05:45:21 -0700

Hi, Bob,

On 6/19/2017 4:37 AM, Bob Briscoe wrote:
>> This approach seems to assume an integrated implementation of
>> multiple protocol layers.
>>
> Can you be more specific? Or perhaps give an example? Or suggest an
> edit to the above text?

The issue is that the paragraph defines "tightly coupled" in terms of
"tightly coupled".

AFAICT, "tightly coupled" depends on whether the layers are added all in
the same software context or some "peeking" occurs after the shim layer
is added but before the final IP.

> supported, so any ECN field in an outer IP header will have to be
> zeroed (typically by configuration because the implementation will not
> include any ECN logic).
>
> but it did not include what /network operators/ have to do if they
> have an implementation that does not support any ECN tunnelling spec

My point is that the propagation of ECN across shim layers needs to be
more clearly explained as optional (even, IMO, for "tightly coupled"
cases). Right now, it reads more like "OK, so if you're layering has
failed to support ECN propagation, here's how to at least make it safe".

> outer as if the Internet Protocol is still as it was before Diffserv
> was standardized 19 years ago (Dec 1998).
>
> below Wikipedia) erroneously tells us that the 8-bit ToS field
> contains 3 precedence bits (that are ignored), 4 ToS bits and 1
> currently unused bit. I am not going to cite the link here, which will
> just reinforce its ranking.
Agreed, but again you're implying that there's something wrong with code
that can't propagate ECN, rather than saying that ECN operates at the IP
layer in which it is inserted by routers. Expecting ECN to propagate to
tunnels is an optimization that in one sense is useful when it works,
but in another is the antithesis of the point of tunneling.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 19 Jun 2017 18:41:43 +0100

Joe,

On 19/06/17 13:45, Joe Touch wrote:
> AFAICT, "tightly coupled" depends on whether the layers are added all 
> in the same software context or some "peeking" occurs after the shim 
> layer is added but before the final IP.
So I think you are not objecting to the para I quoted above, but to 
something in the previous para, quoted here:

    In many cases the shim header(s) and the outer IP header are always
    added (or removed) as part of the same process.  We call this a
    tightly coupled shim header.  Processing the shim and outer together
    is often necessary because the shim(s) are not sufficient for packet
    forwarding in their own right; not unless complemented by an outer
    header.

Is the phrase "as part of the same process" the one you are objecting to?

If so, I have no objection to using an alternative to the word 'process' 
to avoid the implication that this means specifically one 'process' as 
defined for the OS in question. It wasn't intended to use the word 
'process' so strictly. The more general meaning of 'procedure' or 
'routine' was intended.

The main point was that the definition of a shim is something that 
cannot be used alone to forward a packet to another network address, 
unless it is complemented by an outer PSN header.

Whatever, these paragraphs only give context to the subsequent normative 
text, which (deliberately) doesn't actually rely on these definitions 
anyway.

>
>>
>>> I scanned the draft and didn't see this addressed explicitly (maybe 
>>> I missed it). It's implied throughout  3.2 but never stated as a 
>>> MAY, e.g., systems MAY support ECN across shim layers but are NOT 
>>> REQUIRED to do so. Can that sort of explicit caveat be included? 
>>> This case might be added as a final bullet in the list at the end of 
>>> section 3.2.
>>>
>> zeroed (typically by configuration because the implementation will 
>> not include any ECN logic).
>>
>> If none of the cases listed in S.3.2 applies, then you would be in 
>> the territory of section 3.1 (i.e. supporting ECN would already be 
>> covered by RFC6040).
>>
>> RFC6040. RFC6040 said what implementers have to do to support 
>> RFC6040, but it did not include what /network operators/ have to do 
>> if they have an implementation that does not support any ECN 
>> tunnelling spec (RFC6040, RFC4301 or full functionality mode of RFC3168).
>>
>> I only realized S.3.2 was needed when I discovered that there is 
>> still equipment being built that is copying the 8-bit ToS byte from 
>> inner to outer as if the Internet Protocol is still as it was before 
>> Diffserv was standardized 19 years ago (Dec 1998).
>>
>> follow the many outdated articles you can find on the Web, rather 
>> than checking the RFCs. For instance, the 2nd hit from Google (ranked 
>> just below Wikipedia) erroneously tells us that the 8-bit ToS field 
>> currently unused bit. I am not going to cite the link here, which 
>> will just reinforce its ranking.
> Agreed, but again you're implying that there's something wrong with 
> code that can't propagate ECN, rather than saying that ECN operates at 
> the IP layer in which it is inserted by routers. Expecting ECN to 
> propagate to tunnels is an optimization that in one sense is useful 
> when it works, but in another is the antithesis of the point of tunneling.
The difficulty being addressed in this section is that blindly copying 
an inner ECN field to an outer is unsafe (whether or not you want to 
support ECN).

There is so much text in S.3.2 that says ECN is optional that I'm not 
sure how I can say it any more times without making the reader die of 
boredom. S.3.2. starts with

    Even when ECN propagation is not implemented or is not being used,

and the proposed new normative text starts:

       When the implementation of a tunnel ingress does not support
       [RFC6040 <https://tools.ietf.org/html/rfc6040>] or one of its compatible predecessors ([RFC4301 <https://tools.ietf.org/html/rfc4301>] or the
       full functionality mode of [RFC3168 <https://tools.ietf.org/html/rfc3168>]) and when the outer tunnel
       header is IP (v4 or v6),

And the section ends with all this:

    Permanently zeroing the outer ECN field is safe, but it is not
    sufficient to claim compliance withRFC 6040 <https://tools.ietf.org/html/rfc6040>  because it does not meet
    the aim of introducing ECN support to tunnels (seeSection 4.3 of [RFC6040] <https://tools.ietf.org/html/rfc6040#section-4.3>).  Developers and network operators are encouraged to
    implement and deploy tunnel endpoints compliant withRFC 6040 <https://tools.ietf.org/html/rfc6040>  (as
    updated by the present specification) in order to provide the
    benefits of wider ECN deployment [RFC8087 <https://tools.ietf.org/html/rfc8087>].  Nonetheless, propagation
    of ECN between IP headers, whether separated by shim headers or not,
    has to be OPTIONAL to implement and to use, because:

    o  Legacy implementations of tunnels without any ECN support already
       exist

    o  A network might be designed so that there is usually no bottleneck
       within the tunnel

    o  If the tunnel endpoints would have to search within an L2 header
       to find an encapsulated IP header, it might not be worth the
       potential performance hit

What more do you want me to say, to emphasize that ECN is optional?

Preferably, please suggest specific edits.

Cheers

Bob

---

From: Joe Touch <[email-masked]>
Date: Wed, 21 Jun 2017 15:46:51 -0700

Hi, Bob,

> Is the phrase "as part of the same process" the one you are objecting to?
Yes.

The doc is written a bit backwards; it presumes an optimization rather
than presuming the safe default.

Also, the doc is written based on an implementation hint ('process')
rather than a protocol requirement.

IMO, the recommendations should be:

ECN bits MUST be set to a known safe default when an IP layer is added,
EXCEPT if a tunnel is specified as the integrated insertion of multiple
layers, then jumping layers to coordinate ECN MAY be valid

> If so, I have no objection to using an alternative to the word
> 'process' to avoid the implication that this means specifically one
> 'process' as defined for the OS in question. It wasn't intended to use
> the word 'process' so strictly. The more general meaning of
> 'procedure' or 'routine' was intended.
>

Even when a shim is enough to accomplish such fowarding, if the shim
lacks ECN support then ECN information should not be crossing the shim
layer (unless the shim isn't really defined as a separate layer, e.g.,
the example I gave above would state that the tunnel encapsulation isn't
shim THEN IP, but shim AND IP as an integrated step).

> an inner ECN field to an outer is unsafe (whether or not you want to
Then IMHO that should be your title and focus.

The doc reads a bit backwards, more focusing on the optimization of how
and when you can do a smart copy across a shim than the basic idea that
"this isn't how ECN works".

It's not that you don't say this, it's that a reader would get the
reverse impression (i.e., that they should find a way to integrate shim
processing to enable this leap, rather than the fact that this is a true
exception and not the focus).

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 22 Jun 2017 02:13:38 +0100

Joe,

On 21/06/17 23:46, Joe Touch wrote:
>>>> So what aspect of the following para (in S.3.1) does not include 
>>>> the case(s) you are thinking of:
>>> The issue is that the paragraph defines "tightly coupled" in terms 
>>> of "tightly coupled".
>>>
>>> AFAICT, "tightly coupled" depends on whether the layers are added 
>>> all in the same software context or some "peeking" occurs after the 
>>> shim layer is added but before the final IP.
[BB] Your concern holds if one believes that every RFC is implicitly 
preceded be the statement "The IETF recommends that you comply with this 
RFC". In contrast, I believe there is nothing implicit around an RFC. I 
see an RFC solely as a specification of what an implementer can 
voluntarily choose to comply with in order to interoperate with others 
all collectively and voluntarily choosing to add a particular feature to 
the Internet.

I think some of our previous disagreement have hinged on this difference 
of opinion on the status of RFCs. If you know of any IETF document that 
resolves this, pls point to it. Otherwise, discussion of every I-D is 
going to tend to divert into an argument over this point.

> rather than a protocol requirement.
[BB] I explained (still in the thread below) that the normative protocol 
requirements text a little later in the draft is (deliberately) not 
dependent on this scene-setting paragraph. A common way to write an RFC 
is to say "We have this sort of stuff in the world. Now here's some 
normative text that is written more generically so that it applies to 
that sort of stuff we have just described, as well as more general cases."

> EXCEPT if a tunnel is specified as the integrated insertion of 
> multiple layers, then jumping layers to coordinate ECN MAY be valid
[BB] See above. This would be a good way to write this RFC, if it was 
implicitly surrounded by "The IETF recommends you comply with this RFC". 
But If implementers reading this RFC are a self-selecting set who are 
trying to implement ECN tunnelling, then the way it is written is a good 
way.

Neither is incorrect. But given RFC6040 is already immutably published. 
and it is written in the latter style, the only style that can be used 
for an update is the same style.

>> 'process' as defined for the OS in question. It wasn't intended to 
>> use the word 'process' so strictly. The more general meaning of 
> the example I gave above would state that the tunnel encapsulation 
> isn't shim THEN IP, but shim AND IP as an integrated step).
[BB] I don't disagree. I believe we are still only disagreeing about the 
implicit status of RFCs ("Commandments" vs. "Points to Voluntarily 
congregate around").
>
>> The difficulty being addressed in this section is that blindly 
>> copying an inner ECN field to an outer is unsafe (whether or not you 
>> want to support ECN).
[BB] It is the title, but only of the section. The document as a whole 
is about what it says in the title of the document: "Propagating 
Explicit Congestion Notification Across IP Tunnel Headers Separated by a 
Shim"
>
> The doc reads a bit backwards, more focusing on the optimization of 
> how and when you can do a smart copy across a shim than the basic idea 
> that "this isn't how ECN works".
>
> reverse impression (i.e., that they should find a way to integrate 
> shim processing to enable this leap, rather than the fact that this is 
> a true exception and not the focus).
That is indeed the impression I would expect a reader to get, given I 
see readers as a self-selected set who are already motivated to find a 
way to enable this leap.

BTW, I have not come across an implementation yet that intrinsically 
cannot do this smart copy to enable ECN (or Diffserv) to leap across a 
shim. However, I would be surprised if there was not one.

Bob

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 Jun 2017 09:35:40 -0700

Hi, Bob,

I think we're on the same page here. I'll suggest that it would be
useful to clarify "tightly coupled" as referring to "is required by a
specification (not just as an implementation issue)" and replace "part
of the same process" with "as defined in an integrated manner in a
specification requirement".

FWIW, I do think nearly every RFC comes with "the IETF recommends you do
this", which is at least partly why the IESG puts a strong wording to
the contrary for individual submissions where they don't agree with that
sentiment, so it would be useful to recheck the wording throughout in
that light. However, I don't have any specific suggestions.

I.e., although we agree, I'm not sure someone reading the document in
its current form would get the same impression.

Joe

On 6/21/2017 6:13 PM, Bob Briscoe wrote:
>>>>> Can you be more specific? Or perhaps give an example? Or suggest
>>>>> an edit to the above text?
>>>>
>> The doc is written a bit backwards; it presumes an optimization
>> rather than presuming the safe default.
> preceded be the statement "The IETF recommends that you comply with
> this RFC". In contrast, I believe there is nothing implicit around an
> RFC. I see an RFC solely as a specification of what an implementer can
> all collectively and voluntarily choosing to add a particular feature
> to the Internet.
>
> I think some of our previous disagreement have hinged on this
> difference of opinion on the status of RFCs. If you know of any IETF
> document that resolves this, pls point to it. Otherwise, discussion of
> every I-D is going to tend to divert into an argument over this point.
>
>>
> [BB] I explained (still in the thread below) that the normative
> protocol requirements text a little later in the draft is
> (deliberately) not dependent on this scene-setting paragraph. A common
> way to write an RFC is to say "We have this sort of stuff in the
> world. Now here's some normative text that is written more generically
> so that it applies to that sort of stuff we have just described, as
> well as more general cases."
>
>>
> implicitly surrounded by "The IETF recommends you comply with this
> RFC". But If implementers reading this RFC are a self-selecting set
> who are trying to implement ECN tunnelling, then the way it is written
> is a good way.
>
> Neither is incorrect. But given RFC6040 is already immutably
> published. and it is written in the latter style, the only style that
> can be used for an update is the same style.
>
>>
>> lacks ECN support then ECN information should not be crossing the
>> shim layer (unless the shim isn't really defined as a separate layer,
>> e.g., the example I gave above would state that the tunnel
>> encapsulation isn't shim THEN IP, but shim AND IP as an integrated step).
> [BB] I don't disagree. I believe we are still only disagreeing about
> the implicit status of RFCs ("Commandments" vs. "Points to Voluntarily
> congregate around").
>>
> Explicit Congestion Notification Across IP Tunnel Headers Separated by
> a Shim"
>>
>> how and when you can do a smart copy across a shim than the basic
>> idea that "this isn't how ECN works".
>>
>> shim processing to enable this leap, rather than the fact that this
>> is a true exception and not the focus).