Subject: [tsvwg] Some tunnel-oriented comments on draft-ietf-tsvwg-rfc5405bis-03

From: "Black, David" <[email-masked]>
Date: Wed, 8 Jul 2015 19:02:26 +0000

<WG chair hat OFF>

I'm going to do two rounds of review on the UDP Guidelines (rfc5405bis) draft.

This email is the first round, on tunnel-related concerns.  I had every intention
of getting this done in April or May, but life is what happens while one is busy
making other plans ...

I have three tunnel-related concerns that I think could use more attention:

-1- Preprovisioned or Reserved Capacity

Section 3.1.7 says:

   Some UDP applications are only expected to be deployed over network
   paths that use preprovisioned capacity or capacity reserved using
   dynamic provisioning, e.g., through the Resource Reservation Protocol
   (RSVP).  Multicast applications are also used with preprovisioned
   capacity (e.g., IPTV deployments within access networks).  These
   applications MAY choose not to implement any congestion control
   mechanism and instead rely on transmitting only on paths where the
   capacity is provisioned and reserved for this use.  This might be an
   acceptable choice for a subset of restricted networking environments,
   but is by no means a safe practice for operation over the wider
   Internet.

This would be a good place to point out that any IETF specification of such
an application is expected to have a strong applicability statement that
restricts the application to such network environments, preferably coupled
with mechanisms to discourage or prevent escape from such environments and/
or usage over the Internet in general.  Section 5 of RFC 7510 (MPLS/UDP)
may be a useful example to mention.  This is particularly important when
preprovisioned capacity is expected to be the common case.

The last paragraph of Section 3.1.1 is also on this topic, and could at
least use a pointer to Section 3.1.7, and might want to be moved to and
merged into Section 3.1.7. 

-2- MTU effects

Section 3.2 on Message Size Guidelines should discuss the MTU-reduction
effects of tunnel-based encapsulation.  The subtleties stem from a
combination of this fine recommendation:

   an application SHOULD NOT send UDP datagrams
   that result in IP packets that exceed the MTU of the path to the
   destination.  

with a strong desire for tunnel encapsulators (the application in this case)
to send one encapsulated packet for each unencapsulated packet received.
For example, an encapsulator should not be fragmenting or reassembling either
IPv6 packets or IPv4 packets with DF=1, and many implementations aren't
in a position to do anything in this area.

In essence, the text in Section 3.2 appears to assume an ability of a tunnel
ingress to adjust the size of transmitted packets - many tunnel ingresses
won't or can't do this for implementation reasons, so a discussion of what to
do instead is in order.  There's some "e.g." text in 3.1.9 that alludes to
what's involved, but it's too brief:

   Designing a tunneling mechanism requires significantly more expertise
   than needed for many other UDP applications, because tunnels are
   usually intended to be transparent to the endpoints transmitting over
   them, so they need to correctly emulate the behavior of an IP link,
   e.g., handling fragmentation, treatment of DSCP values, generating
   and responding to ICMP messages, etc.

I think more needs to be said on MTU considerations.  There was a
general tunneling considerations draft over in intarea, but it seems
to have been abandoned.

I've seen an argument made over on the intarea list that tunnel ingress-egress
MTU discovery (PMTUD or PLPMTUD) ought to be a "MUST use" so that a tunnel
carrying IP traffic effectively functions like a virtual link with its own
MTU - the ingress then drops packets that don't fit in the MTU and is the
"right place" to generate ICMP responses (as appropriate).

-3- UDP Zero Checksums, especially for IPv6 outer headers

Well, RFC 7510 (MPLS/UDP) tried to follow the instructions in RFC 6935 and
RFC 6936 for what to do when omitting the UDP checksum when IPv6 is the outer
header in a tunnel.  To describe that experience as "painful" would be an
understatement, and we had some very good people involved in the work.  The
result is that RFC 7510 contains about *4* pages of text (sections 3.1 and 3.2)
on this minor aspect of that tunnel protocol.

I think a *strong* recommendation (a semi-polite form of "expect to have to
explain yourself to your AD if you don't do this) is in order to keep others
out of this tarpit.  The recommendation would be that any UDP-based tunnel
encapsulation that wants to zero out the UDP checksum **SHOULD** add a
checksum to the encapsulation that covers the UDP header and the outer IPv6
pseudo-header.  For consistency, that checksum should also be used for at
least the UDP header over IPv4 to protect against port corruption-based
misdelivery in the UDP header (and covering the outer IPv4 pseudo-header for
consistency is fine), but the IPv4 outer-header case is not as crucial as
the IPv6 outer header case.

I might point to what happened in RFC 7510 as an indication of the gauntlet that
awaits those who want to avoid outer checksums of any form.

</WG chair hat OFF>

Thanks,
--David

> -----Original Message-----
> From: tsvwg [mailto:[email-masked]] On Behalf Of Internet-
> [email-masked]
> Sent: Tuesday, July 07, 2015 12:36 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: [tsvwg] I-D ACTION:draft-ietf-tsvwg-rfc5405bis-03.txt
>  
> A new Internet-Draft is available from the on-line Internet-Drafts
> directories.
> This draft is a work item of the Transport Area Working Group Working Group of
> the IETF.
> 
>     Title         : UDP Usage Guidelines
>     Author(s)     : L. Eggert, et al
>     Filename      : draft-ietf-tsvwg-rfc5405bis
>     Pages         : 46
>     Date          : 2015-07-07
> 
>    The User Datagram Protocol (UDP) provides a minimal message-passing
>    transport that has no inherent congestion control mechanisms.  This
>    document provides guidelines on the use of UDP for the designers of
>    applications, tunnels and other protocols that use UDP.  Congestion
>    control guidelines are a primary focus, but the document also
>    provides guidance on other topics, including message sizes,
>    reliability, checksums, middlebox traversal, the use of ECN, DSCPs,
>    and ports.
> 
>    Because congestion control is critical to the stable operation of the
>    Internet, applications and other protocols that choose to use UDP as
>    an Internet transport must employ mechanisms to prevent congestion
>    collapse and to establish some degree of fairness with concurrent
>    traffic.  They may also need to implement additional mechanisms,
>    depending on how they use UDP.
> 
>    Some guidance is also applicable to the design of other protocols
>    (e.g., protocols layered directly on IP or via IP-based tunnels),
>    especially when these protocols do not themselves provide congestion
>    control.
> 
>    If published as an RFC, this document will obsolete RFC5405.
> 
> A URL for this Internet-Draft is:
> https://www.ietf.org/internet-drafts/draft-ietf-tsvwg-rfc5405bis-03.txt
> 
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
> 
> Below is the data which will enable a MIME compliant mail reader
> implementation to automatically retrieve the ASCII version of the
> Internet-Draft.

---

From: [email-masked]
Date: Thu, 9 Jul 2015 09:11:25 +0100

Thanks for the review comments.

Please see below.

Gorry

> of getting this done in April or May, but life is what happens while one
> is busy making other plans ...
>
> restricts the application to such network environments,
>
I am fine with this addition.

> preferably coupled with mechanisms to discourage or prevent escape
> from such environments and/
GF: I've often said the same myself, so I think this also is OK to add.

> merged into Section 3.1.7.
GF: We'll look at this.

> "right place" to generate ICMP responses (as appropriate).
GF: But then... do you think these ICMP messages make it back to the
source? And how does this handle tunnels-within-tunnels? (but that's an
ICMP problem, rather than a UDP app problem).

GF: We could also add:
PLPMTUD is not susceptible to "black holing", but requires explicit 
participation by the receiver. This type of feedback is not typically
present for unidirectional applications.

GF: So, there's only one line on UDP tunnels, and we could say more if
there was more guidance. Joe worked on draft-ietf-intarea-tunnels,
draft-ietf-rtgwg-dt-encap says things, but I can't see obvious
recommendations. I'm not sure what you would like to see added to describe
tunnel MTU usage.

> result is that RFC 7510 contains about *4* pages of text (sections 3.1 and
> 3.2) on this minor aspect of that tunnel protocol.
>
> I think a *strong* recommendation (a semi-polite form of "expect to have
> to explain yourself to your AD if you don't do this) is in order to keep
> others out of this tarpit.
>
GF: Intention understood.

>The recommendation would be that any UDP-based tunnel
> checksum to the encapsulation that covers the UDP header and the outer
> IPv6 pseudo-header.
>
GF: I think this says use UDP-Lite or provide an equivalent header
integrity check. That is indeed one way to be safe, and could be made
clearer.

GF: I note, this creates an inability to traverse NAT or NAPT on any other
middle box that could otherwise update the checksum, unless
the payload also caries the original IP pseudo-checksum (as proposed in GUE).
Should this concern us for IPv6?

> For consistency, that checksum should also be used for at
> misdelivery in the UDP header (and covering the outer IPv4 pseudo-header
> for consistency is fine), but the IPv4 outer-header case is not as
crucial as
> the IPv6 outer header case.
GF: I can see the logic. I'd be happier checking community consensus on
whether to add this though.

> I might point to what happened in RFC 7510 as an indication of the
> gauntlet that awaits those who want to avoid outer checksums of any form.
>
> </WG chair hat OFF>
>
> Thanks,
> --David
>
>

---

From: "Black, David" <[email-masked]>
Date: Thu, 9 Jul 2015 15:04:11 +0000

> > -2- MTU effects

[... snip ...]

> ICMP problem, rather than a UDP app problem).

For tunnels-within-tunnels, the immediate consumer of the MTU-related ICMP
messages would be the ingress for the outermost tunnel (i.e., the node that
applied the IP header against which the ICMP message was generated).  At that
ingress node, the ICMP message is primarily input to the "virtual link" PMTUD
logic, and that ingress node can (SHOULD?) remove a level of encap to pass
the ICMP message upstream.

The ICMP problem is that with enough nested tunnels, one eventually runs
out of ICMP payload space, and can't propagate an ICMP message generated
deep inside a set of nested tunnels all the way back out.  This is where
"virtual link" MTU maintenance helps - the ICMP message updates the MTUs
as far out as it gets, and the next packet hits am MTU limit at a tunnel
ingress that isn't as deeply nested, so the ICMP message propagates further
out ... lather, rinse, repeat ... and eventually, everybody figures it out
and the too-large packets get ICMP too-big responses from the initial
tunnel ingress.  Not perfect ... but then, neither is ICMP.

> present for unidirectional applications.

For PLPMTUD, what I meant to suggest is adding tunnel ingress to tunnel
egress PLPMTUD probes in addition to the tunneled traffic.
 
> recommendations. I'm not sure what you would like to see added to describe

I'd definitely like to hear from Joe on this topic.  As a strawman, I'd
suggest at least a lower case "should"  for tunnels maintaining/monitoring
MTU from ingress to egress and enforcing that MTU at ingress (drops, ICMP
responses if/as appropriate), as opposed to always leaving it to some node
within the tunnel.  The result drops undeliverable traffic earlier and
yields a structure in which ICMP has some chance of working with deep
tunnel nesting.

> integrity check. That is indeed one way to be safe, and could be made

+1, and please add a warning that many middleboxes black-hold UDP-Lite :-(.

> the payload also carries the original IP pseudo-checksum (as proposed in GUE).

Inability to traverse NAT or NAPT might be a "feature" for some tunnels
that depend on pre-provisioned capacity.  Beyond that, this could use
more thought/discussion.

> whether to add this though.

+1, on community input/consensus on this whole topic. 

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]]
> Sent: Thursday, July 09, 2015 4:11 AM
> To: Black, David
> Cc: [email-masked]
> Subject: Re: [tsvwg] Some tunnel-oriented comments on draft-ietf-tsvwg-
> rfc5405bis-03
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 09 Jul 2015 10:56:35 -0700

> I have three tunnel-related concerns that I think could use more attention:
...
>    that result in IP packets that exceed the MTU of the path to the

It is very important to understand what is meant by the "MTU of the path
to the destination".

Some might (IMO) misinterpret that as the "MTU of the tunnel path, minus
the encapsulation overhead".

I interpret it as the "largest payload that the tunnel can transit from
the ingress to the egress." That is defined by the egress reassembly
maximum minus the encapsulation overhead.

Here's the difference:

- in the first case, IPv6 can never be compliantly be tunneled over a
compliant IPv6

	Consider a tunnel whose path supports 1280B packets (the
	minimum IPv6 requires).

	As a tunnel, that path's MTU is 1280-(40+IP options). Even
	with no options, the path's MTU is now 1240 and cannot be
	considered a viable IPv6 tunnel.

- in the second case, IPv6 can tunnel over IPv6:

	Tunnel supports a path of 1280B but is required to be able
	to reassemble 1500B packets at the egress.

	That path's tunnel MTU is 1500-(40+IP options), which
	gives plenty of room to support 1280B packets.

Note that this is consistent with the pending revision to the "Tunnels
in the Internet Architecture" document in INTAREA.

> in a position to do anything in this area.

Agreed, but it's also required that both IPv4 and IPv6 egresses be
capable of reassembling fragments, i.e., when the fragments are created
by the ingress.

> general tunneling considerations draft over in intarea, but it seems

It has been slow due to a lack of cycles by many parties, but it's
currently open in another window on my desktop right now and a draft
will be posted when the submission window reopens.

FWIW, I'll send you a copy of that off-list in a few minutes so you can
see the current state.

> "right place" to generate ICMP responses (as appropriate).

MUST use if the tunnel wants to avoid ingress fragment generation when
the incoming packets are valid but larger than the value the tunnel can
assume will transit unfragmented (e.g., 1240 for IPv6).

Joe

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 09 Jul 2015 19:12:45 +0100

See end...

> assume will transit unfragmented (e.g., 1240 for IPv6).

On MTU, it seems like it may be best that the actual requirements of how 
to use ICMP are best kept in the tunnels draft - with a reference from 
RFC5405.bis - since these are not specifically "UDP" requirements. 
However draft-ietf-intarea-tunnels-00 is stated as INFO backgroundand 
the guidelines are BCP, so if we wish for normative txt it probably 
needs to come here.

..."MUST use PMTU or PLPMTUD if the tunnel wants to avoid ingress 
fragment generation when the incoming packets are valid but larger than 
the value the tunnel can assume will transit unfragmented (e.g., 1240 
for IPv6)."

I can of course try to make sure the language is improved when 
referencing MTU for UDP tunnels.

Gorry

---

From: Joe Touch <[email-masked]>
Date: Thu, 09 Jul 2015 11:21:42 -0700

>> ICMP problem, rather than a UDP app problem).

It's actually a UDP app problem when you use UDP as the encapsulation.

Consider IP(outer)-UDP-IP(inner)

The tunnel should look at only IP(outer) [and maybe UDP for ECMP].

When it generates an ICMP error, it generates it back to the ingress.

That ICMP is received by the ingress IP processing and RFC1122 requires
that it be sent on to the ingress UDP processing.

At that point, it's in the hands of the UDP processing as to what to do.
IP is done with it.

So what can UDP do? As the "app layer" it could could fragment the
incoming messages, but UDP doesn't have header information that supports
fragmentation.

If UDP receives an IPv4 DF=0 packet it could fragment the packet, but
technically that should have already been done by the "router" that
attaches to the tunnel (because a tunnel looks like a link).

So UDP can adjust the MTU of the tunnel, but only so far. It can't set
it below 1280 and still be considered a compliant IPv6 link. For IPv4,
the number is lower but somewhat absurdly so - 68.

So how does a tunnel tell a router that it's MTU changed? Links don't
typically have variable MTUs, but let's assume that's possible.

At that point it would be the NEXT PACKET arriving at the router that is
too larger which would cause the *router* to generate the ICMP back to
the source.

Trying to expect that there's a relay of the inner tunnel ICMP back to
the source is not recursive and doesn't need to happen.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 09 Jul 2015 11:24:57 -0700

> RFC5405.bis - since these are not specifically "UDP" requirements.

Agreed.

> However draft-ietf-intarea-tunnels-00 is stated as INFO background and

We started as INFO for the tunnels doc because it was quite early and it
wasn't clear whether there was momentum for it being BCP. I hope it will
be considered for BCP once we converge on text. Until that happens, I
agree that it's useful for this doc to make 2119-level assertions and
explain how they relate back to the INFO doc.

> referencing MTU for UDP tunnels.

Please wait until -01 is posted if possible; the text is open in another
window, and it'd be useful to cite it accurately.

Joe

---

From: "Black, David" <[email-masked]>
Date: Thu, 9 Jul 2015 20:19:41 +0000

More near end ...

> RFC5405.bis - since these are not specifically "UDP" requirements.

+1, MTU and path MTU discovery should be the focus here.

> referencing MTU for UDP tunnels.

I think it's somewhat more than that - the ingress has to actually use
the discovered MTU - I'm sure that was intended for the improved language ;-).

OTOH, the tunnels draft is (IMHO) a much better place to discuss whether
to try to de-encap ICMP and pass it upstream from the tunnel ingress vs.
just update the MTU to deal with future too-big packets and likewise for
details of what too-big means for IPv4 and IPv6 with and without
fragmentation/segmentation.

Thanks,
--David

---

From: [email-masked]
Date: Fri, 10 Jul 2015 07:25:28 +0100

You can expect the next revision to include some updates to address this.

I'll note this in the slides for the meeting.

Gorry