Subject: RE: Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

From: "Wen Zhang (wzhang)" <[email-masked]>
Date: Tue, 7 Jan 2014 03:59:06 +0000

Hi,

Here are a few comments/feedback on the draft:

1. Section 3. Encapsulation in UDP

I'd like to see the source port allocation requirements stated in a more explicit manner. The draft seems to imply the udp source port should vary on a per-flow basis in order to achieve the load-
sharing goal, so maybe we can state exactly that? ie., I would prefer the following sentence to read:

Source Port of UDP

     This field contains a 16-bit entropy value that is
     generated by the ingress PE router to uniquely identify
     an MPLS flow. What algorithm...

2. Section 6 Security Considerations

For the security aspects, I'd like to see a better problem description. To me the problem is not so much of how to address integrity and privacy for MPLS-over-UDP, but rather what _new_ problem is introduced with MPLS-over-UDP that native mpls or other mpls encapsulation solutions doesn't have. In other words, is integrity/privacy a bigger problem for mpls-over-udp than plain mpls,
and why? or how the solution makes security an easier or harder problem? On the surface, the security concern should be no more or less than what's already stated in RFC-4023 (MPLS in IP or GRE).

3. I think the IPSec vs. DTLS discussion can use a bit more clarity without going into too much protocol discussion between the two. I would suggest the following change in the text:

====
In the case where any of the above security issues is concerned, the MPLS-in-UDP tunnels SHOULD be secured with IPsec or DTLS. Both protocols should satisfy the authentication, integrity, and confidentiality security requirements. IPSec was designed as network security mechanism for Internet Protocols, and therefore resides at the network layer. As such, if the tunnel is secured with IPsec, the UDP header would not be visible to P routers anymore in either IPSec tunnel or transport mode. As a result, the meaning of adopting MPLS-in-UDP encapsulation format as an alternative to MPLS-in-GRE and MPLS-in-IP encapsulation formats is lost. By comparison, DTLS is better suited for application security and can better preserve network and transport layer protocol information.
Specifically, if DTLS is used, the destination port of the UDP header will be filled with a value (TBD2) indicating MPLS with DTLS and the source port can still be used as an entropy field for load sharing purposes. 
====

Thanks,
Wen

-----Original Message-----
From: IETF-Announce [mailto:[email-masked]] On Behalf Of The IESG
Sent: Thursday, January 02, 2014 10:14 AM
To: IETF-Announce
Cc: [email-masked]
Subject: Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

The IESG has received a request from the Multiprotocol Label Switching WG
(mpls) to consider the following document:
- 'Encapsulating MPLS in UDP'
   <draft-ietf-mpls-in-udp-04.txt> as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits final comments on this action. Please send substantive comments to the [email-masked] mailing lists by 2014-01-16. Exceptionally, comments may be sent to [email-masked] instead. In either case, please retain the beginning of the Subject line to allow automated sorting.

Abstract

    This document specifies an IP-based encapsulation for MPLS, called
    MPLS-in-UDP (User Datagram Protocol).

The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-mpls-in-udp/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-mpls-in-udp/ballot/

The following IPR Declarations may be related to this I-D:

    http://datatracker.ietf.org/ipr/1941/
    http://datatracker.ietf.org/ipr/2198/

---

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 8 Jan 2014 10:21:37 +0000

Hi,

>  <draft-ietf-mpls-in-udp-04.txt> as Proposed Standard

this document needs to describe how it addresses the issues raised in BCP145 (RFC5405). It already contains some text about messages sizes and congestion considerations, which is great. Unfortunately, the text about congestion considerations is not fully in line with RFC5405.

Lars

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 10 Jan 2014 08:47:35 +0000

Hi,

that sounds good. What congestion control are you going to be specifying for your tunnel?

Lars

On 2014-1-10, at 4:46, Xuxiaohu <[email-masked]> wrote:

> Hi Lars,
> 
> Thanks a lot for your comments.
> 
> I wonder whether the following modified text for Congestion Consideration section is OK from your point of view:
> 
> Since the MPLS-in-UDP encapsulation causes MPLS packets to be forwarded through "UDP tunnels", the congestion control guidelines for UDP tunnels as defined in Section 3.1.3 of [RFC5405] SHOULD be followed. Specifically, MPLS can carry a number of different protocols as payloads. When the payload traffic is IP-based and congestion-controlled, the UDP tunnel SHOULD NOT employ its own congestion control mechanism, because congestion losses of tunneled traffic will already trigger an appropriate congestion response at the original senders of the tunneled traffic. When the payload traffic is not known to be IP-based, or is known to be IP-based but not congestion-controlled, the UDP tunnel SHOULD employ an appropriate congestion control mechanism. Furthermore, because UDP tunnels are usually bulk-transfer applications as far as the intermediate routers are concerned, the guidelines as defined in Section 3.1.1 of [RFC5405] SHOULD apply.
> 
> Best regards,
> Xiaohu
> 
>> -----邮件原件-----
>> 发件人: mpls [mailto:[email-masked]] 代表 Eggert, Lars
>> 发送时间: 2014年1月8日 18:22
>> 收件人: IETF
>> 抄送: [email-masked]
>> 主题: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS
>> in UDP) to Proposed Standard
>> 
>> Hi,
>> 
>> this document needs to describe how it addresses the issues raised in BCP145
>> (RFC5405). It already contains some text about messages sizes and congestion
>> considerations, which is great. Unfortunately, the text about congestion
>> considerations is not fully in line with RFC5405.
>> 
>> Lars

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 10 Jan 2014 09:15:53 +0000

RFC3985/6.5.  Congestion Considerations
may have some useful text. 

We have a lot of deployment experience with
PWs and frankly I have never seen a congestion
complain raised from the field.

Stewart

On 10/01/2014 03:46, Xuxiaohu wrote:
> _______________________________________________
> mpls mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mpls

-- 
For corporate legal information go to:

http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 10 Jan 2014 10:36:35 -0500

Maybe I am completely missing things, but this looks wrong.
If the MPLS LSP is carrying fixed rate pseudo-wires, adding congestion
control will make it more likely that the service won't work.  Is that
really the goal?

We do not perform congestion control on MPLS LSPs.
Assuming that a UDP tunnel is carrying just MPLS and was established
just for MPLS, why would we expect it to behave differently than an MPLS
LSP running over the exact same path, carrying the exact same traffic?

Yours,
Joel

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 10 Jan 2014 16:09:45 +0000

Hi,

> LSP running over the exact same path, carrying the exact same traffic?

we've been rehashing this discussion several times over the years, e.g., for PWE, AMT, etc. In order to carry fixed-rate or otherwise non-congestion-controlled traffic over unprovisioned general Internet paths, there needs to be some sort of basic congestion control mechanism, like a circuit breaker.

The whole point of running MPLS is to create networks in which paths are provisionable, so this is usually not an issue. But if you start sticking MPLS inside of UDP, those packets can go anywhere on the net, so you need mechanisms to control the rate of that traffic if it causes congestion, or at the very least you need to be able to stop the traffic if it creates severe congestion.

Lars

---

From: Scott Brim <[email-masked]>
Date: Fri, 10 Jan 2014 11:32:10 -0500

I don't think it's right to try to solve this in MPLS, because MPLS is
not a forwarding protocol - it's a connectivity protocol. In any use
of UDP, congestion control is either left to something above UDP or
ignored (left to queue management). Similarly, you want the client of
MPLS to be responsible for managing its traffic. MPLS gives you paths,
it doesn't push packets over them.

Scott

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 10 Jan 2014 03:46:07 +0000

Hi Lars,

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 10 Jan 2014 10:06:41 +0000

Hi Lars,

> -----邮件原件-----
> 发件人: Eggert, Lars [mailto:[email-masked]]
> 发送时间: 2014年1月10日 16:48
> 收件人: Xuxiaohu
> 抄送: IETF; [email-masked]
> 主题: Re: Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP)
> to Proposed Standard
> 
> Hi,
> 

IMHO, this is a common issue with any other encapsulations for MPLS and even other applications of UDP tunnels . Of course, the following text could be added if we believe it's necessary: 

   "...applications that uses the encapsulation as specified in this
   document SHOULD monitor the packet loss rate to ensure that it is
   within acceptable parameters. Packet loss is considered acceptable
   if a TCP flow across the same network path under the same network
   conditions would achieve an average throughput, measured on a
   reasonable timescale, that is not less than that of the MPLS-in-UDP flow.
   The comparison to TCP cannot be specified exactly, but is intended as
   an "order-of-magnitude" comparison in timescale and throughput.

   In essence, this requirement states that it is
   not acceptable to deploy an application using the encapsulation
   specified in this document on the best-effort Internet, which
   consumes bandwidth arbitrarily and does not compete fairly with TCP
   within an order of magnitude. One method of determining an
   acceptable bandwidth is described in [RFC3448]. "

Best regards,
Xiaohu

> > Since the MPLS-in-UDP encapsulation causes MPLS packets to be forwarded
> through "UDP tunnels", the congestion control guidelines for UDP tunnels as
> defined in Section 3.1.3 of [RFC5405] SHOULD be followed. Specifically, MPLS
> can carry a number of different protocols as payloads. When the payload traffic
> is IP-based and congestion-controlled, the UDP tunnel SHOULD NOT employ its
> own congestion control mechanism, because congestion losses of tunneled
> traffic will already trigger an appropriate congestion response at the original
> senders of the tunneled traffic. When the payload traffic is not known to be
> IP-based, or is known to be IP-based but not congestion-controlled, the UDP
> tunnel SHOULD employ an appropriate congestion control mechanism.
> Furthermore, because UDP tunnels are usually bulk-transfer applications as far
> as the intermediate routers are concerned, the guidelines as defined in Section
> 3.1.1 of [RFC5405] SHOULD apply.
> >
> > Best regards,
> > Xiaohu
> >
> >> -----邮件原件-----
> >> 主题: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt>
> >> (Encapsulating MPLS in UDP) to Proposed Standard
> >>
> >> Hi,
> >>
> >> this document needs to describe how it addresses the issues raised in
> >> BCP145 (RFC5405). It already contains some text about messages sizes
> >> and congestion considerations, which is great. Unfortunately, the
> >> text about congestion considerations is not fully in line with RFC5405.
> >>
> >> Lars

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 10 Jan 2014 10:07:13 +0000

Hi Stewart,

Thanks for that information and I will look at it.

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: Stewart Bryant [mailto:[email-masked]]
> 发送时间: 2014年1月10日 17:16
> 收件人: Xuxiaohu; Eggert, Lars; IETF
> 抄送: [email-masked]
> We have a lot of deployment experience with PWs and frankly I have never seen
> a congestion complain raised from the field.
> 
> Stewart
> 
> 
>

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 10 Jan 2014 18:24:02 +0000

Hi Lars,
I think that " The whole point of running MPLS is to create networks in which paths are provisionable, so this is usually not an issue." is only partially correct. LDP-based MPLS network is not provisionable and LSPs follow IP best route selection. Explicit signaling of LSP is achievable in (G)MPLS by using RSVP(-TE) signaling.

	Regards,
		Greg

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Eggert, Lars
Sent: Friday, January 10, 2014 8:10 AM
To: Joel Halpern
Cc: [email-masked]; IETF
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Hi,

On 2014-1-10, at 16:36, Joel M. Halpern <[email-masked]> wrote:
> just for MPLS, why would we expect it to behave differently than an 
> MPLS LSP running over the exact same path, carrying the exact same traffic?

Lars

---

From: Scott Brim <[email-masked]>
Date: Fri, 10 Jan 2014 15:41:47 -0500

On Fri, Jan 10, 2014 at 3:14 PM, Edward Crabbe <[email-masked]> wrote:
> +1
>
> Let the encapsulated transport protocols take care of the congestion
> end-to-end.

Xu Xiaohu proposed a good paragraph:
https://www.ietf.org/ibin/c5i?mid=6&rid=49&gid=0&k1=933&k2=75392&tid=1389386424

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 10 Jan 2014 15:47:49 -0500

The problem with taht paragraph is that it assumes that the UDP 
encapsulator knows what the payload is.
It seems to me that one could easily be applying the UDP in any one of a 
number of cases where the payload will not be known.
So the requirement in the paragraph seems at best difficult to meet.

And has been noted, this seems to place an expectation on UDP 
encapsulated MPLS that is not present for MPLS itself, when the traffic 
is not known to be IP.

Yours,
Joel

---

From: Scott Brim <[email-masked]>
Date: Fri, 10 Jan 2014 16:32:02 -0500

On Fri, Jan 10, 2014 at 3:47 PM, Joel M. Halpern <[email-masked]> wrote:
> The problem with taht paragraph is that it assumes that the UDP encapsulator
> knows what the payload is.
> And has been noted, this seems to place an expectation on UDP encapsulated
> MPLS that is not present for MPLS itself, when the traffic is not known to
> be IP.

OK good point - so we invoke the end-to-end argument on MPLS's behalf.

---

From: "Eggert, Lars" <[email-masked]>
Date: Sat, 11 Jan 2014 06:23:20 +0000

Hi,

> OK good point - so we invoke the end-to-end argument on MPLS's behalf.

look at it the other way. From the viewpoint of the rest of the net, you are an application using UDP. Such applications need to follow a set of principles we have IETF consensus on (RFC5405).

By encapsulating MPLS in UDP, you are changing the game. That traffic can now appear on any Internet path, and not just inside provisioned networks. Because of that, you need a mechanism to detect if you are causing congestion, and a mechanism to react to it.

And it *is* a requirement on the encapsulator, because from the perspective of the rest of the net, that is the application that generates the UDP traffic.

Lars

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 13 Jan 2014 08:50:06 +0000

Hi,

On 2014-1-13, at 4:40, Xuxiaohu <[email-masked]> wrote:
>> I don't think it's right to try to solve this in MPLS, because MPLS is not a
>> forwarding protocol - it's a connectivity protocol.

right, MPLS is the wrong place to address is. The UDP encaps/decaps function needs to have this functionality.

>> In any use of UDP, congestion
>> control is either left to something above UDP or ignored (left to queue
>> management).

There are several cases, see Section 3.1.3 of RFC5405. MPLS-over-UDP can fall into any of the three cases, depending on what traffic is inside the LSP being encapsulated. 

You'll notice that RFC5405 for the first case - encapsulation of IP-based congestion controlled "normal" Internet traffic - even says that the tunnel SHOULD NOT employ any congestion control scheme of its own. Having layered control loops fighting is not productive.

The issue with MPLS-in-UDP (and GRE-in-UDP, and any other encaps scheme that can carry non-IP traffic) are with cases two and three. When the workload that is being encapsulated isn't known to be congestion controlled by its endpoints, it is the obligation of the tunnel to detect congestion and react to it by reducing the traffic volume. Because for the rest of the network, that tunnel is the UDP sender, and we have IETF consensus that we don't want UDP senders that don't react to congestion on the net. (That's one of the main reasons for the existence of the RMCAT WG - we don't want non-congestion-controlled RTP media traffic on the net.)

The key difference between putting MPLS e.g. into IP compared to putting it into UDP is that once it's in UDP, it can go pretty much anywhere on the net, because UDP traverses NATs and firewalls much more easily than IP traffic with a rare protocol number does.

>> Similarly, you want the client of MPLS to be responsible for
>> managing its traffic. MPLS gives you paths, it doesn't push packets over them.

Right. However, once you slap a UDP header on a packet during encapsulation, you now subjected yourself to the rules for Internet UDP senders. Those are documented in RFC5405, and require the tunnel to implement some sort of congestion detection and control. I'd personally consider a circuit breaker mechanism sufficient, like RTP and I think PWE are using.

> Fully agree. The congestion control should be performed either by the UDP tunnel itself or the client of MPLS. In the former case, it'd better to specify the practical congestion control mechanisms (if there were any) in a generic draft (e.g., RFC5405bis) and then any use of the UDP tunnel could refer to that generic draft with regard to congestion control.

The general concept of a circuit breaker is easy enough that it doesn't really need to be written down. And it wouldn't be possible to describe it in a generic fashion, because congestion detection is typically specific to the protocol being encapsulated (e.g., RTP uses RTCP feedback to derive loss information, etc.) And the reaction to congestion is also dependent on the protocol being encapsulated (does it support multiple rates or only on/off, what timescales are OK for reaction, etc.)

> In the latter case, if the client of MPLS is TCP-friendly, that is great. Otherwise (e.g., circuit emulation service), it shouldn't be deployed on the Internet at all, just as has been pointed out in RFC3985, therefore there is no need for any specific congestion control mechanism on the client.
> 
>  "... In essence, this requirement states that it is
>   not acceptable to deploy an application (using PWE3 or any other
>   transport protocol) on the best-effort Internet, which consumes
>   bandwidth arbitrarily and does not compete fairly with TCP within an
>   order of magnitude." (quoted from Section 6.5 of RFC3985)
> 
> The above choice seems no conflict with the following congestion control guidelines as quoted from Section 3.1.1 of RFC5405, as those non-TCP-friendly traffic would be transported over a provisioned path, rather than on the Internet.
> 
>   "...Finally, some bulk transfer applications may choose not to implement
>   any congestion control mechanism and instead rely on transmitting
>   across reserved path capacity.  This might be an acceptable choice
>   for a subset of restricted networking environments, but is by no
>   means a safe practice for operation in the Internet."

How is that in conflict? Both quotes say that Internet traffic needs congestion control, which is a restatement of RFC2914.

Lars

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 13 Jan 2014 09:37:20 +0000

Hi,

On 2014-1-13, at 10:16, Xuxiaohu <[email-masked]> wrote:
> No conflict at all. What I meant is: for those clients of MPLS which are not TCP-friendly (case 2&3 as described in Section 3.1.3 of RFC5405), they should never be transported over the unprovisioned path (e.g., the Internet). Insteads, they should only be transported over a provisioned path in a restricted networking environment. As a result, there is no need for the congestion control mechanism for them.

ah, sorry! I misread "The above choice seems no conflict" in your sentence as "The above choice seems TO conflict". Oops.

Lars

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 13 Jan 2014 09:42:12 +0000

Hi,

> No conflict at all. What I meant is: for those clients of MPLS which are not TCP-friendly (case 2&3 as described in Section 3.1.3 of RFC5405), they should never be transported over the unprovisione…

I agree, but I think we need a safety mechanism when such traffic does end up on the general Internet (because operators may not read the RFC, or there may be configuration errors, etc.)

Even when running inside a provisioned domain, you probably want some sort of safety net, like a circuit breaker that detects if your tunnel is experiencing/causing severe congestion, and shut it down.

Lars

---

From: Alexander Vainshtein <[email-masked]>
Date: Sun, 12 Jan 2014 09:42:13 +0000

Lars, Scott and all,

First, a piece of history.

Congestion control considerations have been a major point of contention between the (then) leadership of the Transport Area and the PWE3 WG (that has started in the Transport Area, then migrated to Internet Area and finally arrived to the Routing Area).  A congestion control framework document is still listed as one of the goals of this WG even if the target date has passed and the draft that was supposed to deal with the subject (https://datatracker.ietf.org/doc/draft-ietf-pwe3-congestion-frmwk) has expired almost 4 years ago.

TDM PWs which, for historical reasons, have always included encapsulations over UDP/IP, have been one of the focal points of this contention with DISCUSS by one of the then Transport Area directors in the process of the IESG approval of the first of these documents. Eventually, this contention has been resolved by adding a dedicated Congestion Control section to RFC 4553. This section introduced several guards against congestion being created by excessive deployment of TDM PWs and provided specific guidelines for  implementing a congestion prevention mechanism (specific to TDM PWs). 

Going back to the draft in question...

Encapsulating  MPLS in UDP looks to me like a far-going extension of running TDM PWs over UDP/IP. The main differences, as I see them, are:

- General MPLS-in-UDP flows can be "BW-greedy" while TDM PWs are not
- Congestion detection for these flows by the encapsulation endpoints (presumably called "applications using MPLS/UDP encapsulation" in the ) is much more problematic, nor is it clear to me, how backpressure  between these endpoints can operate even if congestion were detected

The text I have found in the Congestion Considerations section of the draft recognizes potential for congestion created by MPLS/UDP flows, and even calls for a mandatory additional congestion control mechanism. However, I could not find any guidelines (or even hints) that would help an implementer to create such a mechanism.  So I'd say that with regard to congestion control this draft does not meet the standard that has been expected (years ago from now) from the TDM PW drafts. 

My 2c,
       Sasha 
Email: [email-masked]
Mobile: 054-9266302

> Sent: Saturday, January 11, 2014 8:23 AM
> To: Scott Brim
> Cc: [email-masked]; IETF
> look at it the other way. From the viewpoint of the rest of the net, you are an
> application using UDP. Such applications need to follow a set of principles we
> have IETF consensus on (RFC5405).
> 
> By encapsulating MPLS in UDP, you are changing the game. That traffic can
> now appear on any Internet path, and not just inside provisioned networks.
> Because of that, you need a mechanism to detect if you are causing
> congestion, and a mechanism to react to it.
> 
> And it *is* a requirement on the encapsulator, because from the perspective
> of the rest of the net, that is the application that generates the UDP traffic.
> 
> Lars

---

From: Edward Crabbe <[email-masked]>
Date: Fri, 10 Jan 2014 12:14:24 -0800

+1

On Fri, Jan 10, 2014 at 8:32 AM, Scott Brim <[email-masked]> wrote:

> > we've been rehashing this discussion several times over the years, e.g.,
> for PWE, AMT, etc. In order to carry fixed-rate or otherwise
> non-congestion-controlled traffic over unprovisioned general Internet
> paths, there needs to be some sort of basic congestion control mechanism,
> like a circuit breaker.
> >
> > The whole point of running MPLS is to create networks in which paths are
> provisionable, so this is usually not an issue. But if you start sticking
> MPLS inside of UDP, those packets can go anywhere on the net, so you need
> mechanisms to control the rate of that traffic if it causes congestion, or
> at the very least you need to be able to stop the traffic if it creates
> severe congestion.
> >
> > Lars

---

From: Mark Tinka <[email-masked]>
Date: Sun, 12 Jan 2014 14:18:11 +0200

On Friday, January 10, 2014 06:09:45 PM Eggert, Lars wrote:

> The whole point of running MPLS is to create networks in
> which paths are provisionable, so this is usually not an
> issue. But if you start sticking MPLS inside of UDP,
> those packets can go anywhere on the net, so you need
> mechanisms to control the rate of that traffic if it
> causes congestion, or at the very least you need to be
> able to stop the traffic if it creates severe
> congestion.

What most networks do is just police/shape traffic at 
whatever rate you can afford to pay for, as it enters/leaves 
the provider's network.

As MPLS is in UDP, all the provider sees is IP, as they 
should. I don't think there will be any "special treatment" 
to UDP traffic carrying MPLS, vs. UDP traffic carrying other 
payloads.

Mark.

---

From: Mark Tinka <[email-masked]>
Date: Sun, 12 Jan 2014 14:44:30 +0200

On Friday, January 10, 2014 08:24:02 PM Gregory Mirsky 
wrote:

> Hi Lars,
> I think that " The whole point of running MPLS is to
> create networks in which paths are provisionable, so
> this is usually not an issue." is only partially
> correct. LDP-based MPLS network is not provisionable and
> LSPs follow IP best route selection. Explicit signaling
> of LSP is achievable in (G)MPLS by using RSVP(-TE)
> signaling.

Greg, I think we can delineate provisioning from whether it 
is dynamic or explicit.

MPLS LSP's are all pre-provisioned, because of the FEC's 
created. Whether those FEC's are dynamically or explicitly 
provisioned is a separate issue.

Mark.

---

From: Xuxiaohu <[email-masked]>
Date: Mon, 13 Jan 2014 02:12:38 +0000

> -----邮件原件-----
> 发件人: mpls [mailto:[email-masked]] 代表 Gregory Mirsky
> 发送时间: 2014年1月11日 2:24
> 收件人: Eggert, Lars; Joel Halpern
> 抄送: [email-masked]; IETF
> I think that " The whole point of running MPLS is to create networks in which
> paths are provisionable, so this is usually not an issue." is only partially correct.
> LDP-based MPLS network is not provisionable and LSPs follow IP best route
> selection. Explicit signaling of LSP is achievable in (G)MPLS by using RSVP(-TE)
> signaling.

+1. With the replacement of a LDP-based LSP tunnel by an IP-based tunnel (e.g., MPLS-in-GRE, MPLS-in-IP or MPLS-in-UDP), the path that the traffic travels through is not changed at all.

Best regards,
Xiaohu

> we've been rehashing this discussion several times over the years, e.g., for PWE,
> AMT, etc. In order to carry fixed-rate or otherwise non-congestion-controlled
> traffic over unprovisioned general Internet paths, there needs to be some sort of
> basic congestion control mechanism, like a circuit breaker.
> 
> provisionable, so this is usually not an issue. But if you start sticking MPLS inside
> of UDP, those packets can go anywhere on the net, so you need mechanisms to
> control the rate of that traffic if it causes congestion, or at the very least you
> need to be able to stop the traffic if it creates severe congestion.
> 
> Lars

---

From: Xuxiaohu <[email-masked]>
Date: Mon, 13 Jan 2014 03:40:17 +0000

> -----邮件原件-----
> 发件人: Scott Brim [mailto:[email-masked]]
> 发送时间: 2014年1月11日 0:32
> 收件人: Eggert, Lars
> 抄送: Joel Halpern; [email-masked]; Xuxiaohu; IETF
> forwarding protocol - it's a connectivity protocol. In any use of UDP, congestion
> management). Similarly, you want the client of MPLS to be responsible for

Fully agree. The congestion control should be performed either by the UDP tunnel itself or the client of MPLS. In the former case, it'd better to specify the practical congestion control mechanisms (if there were any) in a generic draft (e.g., RFC5405bis) and then any use of the UDP tunnel could refer to that generic draft with regard to congestion control. In the latter case, if the client of MPLS is TCP-friendly, that is great. Otherwise (e.g., circuit emulation service), it shouldn't be deployed on the Internet at all, just as has been pointed out in RFC3985, therefore there is no need for any specific congestion control mechanism on the client.

Best regards,
Xiaohu

> >> If the MPLS LSP is carrying fixed rate pseudo-wires, adding
> >> congestion control will make it more likely that the service won't
> >> work.  Is that really the goal?
> >>
> > we've been rehashing this discussion several times over the years, e.g., for
> PWE, AMT, etc. In order to carry fixed-rate or otherwise
> non-congestion-controlled traffic over unprovisioned general Internet paths,
> there needs to be some sort of basic congestion control mechanism, like a
> circuit breaker.
> >

---

From: Xuxiaohu <[email-masked]>
Date: Mon, 13 Jan 2014 09:16:30 +0000

> 发送时间: 2014年1月13日 16:50
> 抄送: Scott Brim; Joel Halpern; [email-masked]; IETF
> >> I don't think it's right to try to solve this in MPLS, because MPLS
> >> is not a forwarding protocol - it's a connectivity protocol.
> 
> >> control is either left to something above UDP or ignored (left to
> >> queue management).
> 
> There are several cases, see Section 3.1.3 of RFC5405. MPLS-over-UDP can fall
> into any of the three cases, depending on what traffic is inside the LSP being
> encapsulated.
> 
> You'll notice that RFC5405 for the first case - encapsulation of IP-based
> congestion controlled "normal" Internet traffic - even says that the tunnel
> SHOULD NOT employ any congestion control scheme of its own. Having layered
> control loops fighting is not productive.
> 
> The issue with MPLS-in-UDP (and GRE-in-UDP, and any other encaps scheme
> that can carry non-IP traffic) are with cases two and three. When the workload
> that is being encapsulated isn't known to be congestion controlled by its
> endpoints, it is the obligation of the tunnel to detect congestion and react to it
> by reducing the traffic volume. Because for the rest of the network, that tunnel
> is the UDP sender, and we have IETF consensus that we don't want UDP senders
> that don't react to congestion on the net. (That's one of the main reasons for
> the existence of the RMCAT WG - we don't want non-congestion-controlled RTP
> media traffic on the net.)
> 
> The key difference between putting MPLS e.g. into IP compared to putting it
> into UDP is that once it's in UDP, it can go pretty much anywhere on the net,
> because UDP traverses NATs and firewalls much more easily than IP traffic with
> a rare protocol number does.
> 
> >> Similarly, you want the client of MPLS to be responsible for managing
> >> its traffic. MPLS gives you paths, it doesn't push packets over them.
> 
> Right. However, once you slap a UDP header on a packet during encapsulation,
> you now subjected yourself to the rules for Internet UDP senders. Those are
> documented in RFC5405, and require the tunnel to implement some sort of
> congestion detection and control. I'd personally consider a circuit breaker
> mechanism sufficient, like RTP and I think PWE are using.
> 
> > Fully agree. The congestion control should be performed either by the UDP
> tunnel itself or the client of MPLS. In the former case, it'd better to specify the
> practical congestion control mechanisms (if there were any) in a generic draft
> (e.g., RFC5405bis) and then any use of the UDP tunnel could refer to that
> generic draft with regard to congestion control.
> 
> The general concept of a circuit breaker is easy enough that it doesn't really
> need to be written down. And it wouldn't be possible to describe it in a generic
> fashion, because congestion detection is typically specific to the protocol being
> encapsulated (e.g., RTP uses RTCP feedback to derive loss information, etc.) And
> the reaction to congestion is also dependent on the protocol being
> encapsulated (does it support multiple rates or only on/off, what timescales are
> OK for reaction, etc.)
> 
> > In the latter case, if the client of MPLS is TCP-friendly, that is great. Otherwise
> (e.g., circuit emulation service), it shouldn't be deployed on the Internet at all,
> just as has been pointed out in RFC3985, therefore there is no need for any
> specific congestion control mechanism on the client.
> >
> > The above choice seems no conflict with the following congestion control
> guidelines as quoted from Section 3.1.1 of RFC5405, as those non-TCP-friendly
> traffic would be transported over a provisioned path, rather than on the
> Internet.
> >
> How is that in conflict? Both quotes say that Internet traffic needs congestion
> control, which is a restatement of RFC2914.

Hi Lars,

Best regards,
Xiaohu

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 13 Jan 2014 10:51:45 -0500

Lars, are you really asking that the document say:
1) That the UDP encapsulation of MPLS can only be used when the devices
performing the encapsulation and decapsulation know what protocol (above
IP, if IP is being carried; above Ethernet and IP if the frame is
Ethernet carrying IP) is being used
2) That those devices need to engage in a congestion control protocol if
the carried packets are not TCP, SCTP, or DCCP?

Those both look like excessive and difficult requirements.  Which is
fine if our goal is to pretend we are telling folks not to use UDP
encapsulation of MPLS.  (I had not thought that was our goal.)

In practice, I simply do not see how anyone implementing this will pay
any attention to either requirement.

Yours,
Joel

On 1/13/14 3:50 AM, Eggert, Lars wrote:
>>> I don't think it's right to try to solve this in MPLS, because
>>> MPLS is not a forwarding protocol - it's a connectivity
>>> protocol.
> 
> right, MPLS is the wrong place to address is. The UDP encaps/decaps
> function needs to have this functionality.
> 
>>> In any use of UDP, congestion control is either left to something
>>> above UDP or ignored (left to queue management).
> 
> There are several cases, see Section 3.1.3 of RFC5405. MPLS-over-UDP
> can fall into any of the three cases, depending on what traffic is
> inside the LSP being encapsulated.
> 
> You'll notice that RFC5405 for the first case - encapsulation of
> IP-based congestion controlled "normal" Internet traffic - even says
> that the tunnel SHOULD NOT employ any congestion control scheme of
> its own. Having layered control loops fighting is not productive.
> 
> The issue with MPLS-in-UDP (and GRE-in-UDP, and any other encaps
> scheme that can carry non-IP traffic) are with cases two and three.
> When the workload that is being encapsulated isn't known to be
> congestion controlled by its endpoints, it is the obligation of the
> tunnel to detect congestion and react to it by reducing the traffic
> volume. Because for the rest of the network, that tunnel is the UDP
> sender, and we have IETF consensus that we don't want UDP senders
> that don't react to congestion on the net. (That's one of the main
> reasons for the existence of the RMCAT WG - we don't want
> non-congestion-controlled RTP media traffic on the net.)
> 
> The key difference between putting MPLS e.g. into IP compared to
> putting it into UDP is that once it's in UDP, it can go pretty much
> anywhere on the net, because UDP traverses NATs and firewalls much
> more easily than IP traffic with a rare protocol number does.
> 
> Right. However, once you slap a UDP header on a packet during
> encapsulation, you now subjected yourself to the rules for Internet
> UDP senders. Those are documented in RFC5405, and require the tunnel
> to implement some sort of congestion detection and control. I'd
> personally consider a circuit breaker mechanism sufficient, like RTP
> and I think PWE are using.
> 
>> Fully agree. The congestion control should be performed either by
>> the UDP tunnel itself or the client of MPLS. In the former case,
>> it'd better to specify the practical congestion control mechanisms
>> (if there were any) in a generic draft (e.g., RFC5405bis) and then
>> any use of the UDP tunnel could refer to that generic draft with
>> regard to congestion control.
> 
> The general concept of a circuit breaker is easy enough that it
> doesn't really need to be written down. And it wouldn't be possible
> to describe it in a generic fashion, because congestion detection is
> typically specific to the protocol being encapsulated (e.g., RTP uses
> RTCP feedback to derive loss information, etc.) And the reaction to
> congestion is also dependent on the protocol being encapsulated (does
> it support multiple rates or only on/off, what timescales are OK for
> reaction, etc.)
> 
>> In the latter case, if the client of MPLS is TCP-friendly, that is
>> great. Otherwise (e.g., circuit emulation service), it shouldn't be
>> deployed on the Internet at all, just as has been pointed out in
>> RFC3985, therefore there is no need for any specific congestion
>> control mechanism on the client.
>> 
>> "... In essence, this requirement states that it is not acceptable
>> to deploy an application (using PWE3 or any other transport
>> protocol) on the best-effort Internet, which consumes bandwidth
>> arbitrarily and does not compete fairly with TCP within an order of
>> magnitude." (quoted from Section 6.5 of RFC3985)
>> 
>> The above choice seems no conflict with the following congestion
>> control guidelines as quoted from Section 3.1.1 of RFC5405, as
>> those non-TCP-friendly traffic would be transported over a
>> provisioned path, rather than on the Internet.
>> 
>> "...Finally, some bulk transfer applications may choose not to
>> implement any congestion control mechanism and instead rely on
>> transmitting across reserved path capacity.  This might be an
>> acceptable choice for a subset of restricted networking
>> environments, but is by no means a safe practice for operation in
>> the Internet."
> 
> How is that in conflict? Both quotes say that Internet traffic needs
> congestion control, which is a restatement of RFC2914.
> 
> Lars
>

---

From: Scott Brim <[email-masked]>
Date: Mon, 13 Jan 2014 14:09:40 -0500

On Mon, Jan 13, 2014 at 10:51 AM, Joel M. Halpern <[email-masked]> wrote:
> Those both look like excessive and difficult requirements.

I'm concerned about TCP-over-X.25 scenarios. If the lower layer
encapsulation does not know what congestion management is being used
above, and it does what would be best assuming there isn't any,
performance can truly suck.

How was this handled in other cases?

Saying UDP is a user of IP and thus must provide congestion control
would be fine if our layering were simple and UDP was Transport, but
in this case UDP is "lower layer", an encapsulation to extend the
reach of MPLS and whatever is running over it.

> Which is

Yes.

My inclination is that this is not fundamental architecture, and we
can afford to see if there's a problem before we solve it.

Scott

---

From: <[email-masked]>
Date: Tue, 14 Jan 2014 08:28:05 +0000

It stands to reason that if tunnelers can turn off udp checksums
because their performance is degraded, they can turn off
congestion control because it will degrade their performance.

Rest of the internet getting congested and getting
misdelivered corrupted packets? Really not their problem.

There are important vendors trying to sell products here,
and they need performance to do so.
Get with the program!

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: ietf [[email-masked]] On Behalf Of Joel M. Halpern [[email-masked]]
Sent: 10 January 2014 15:36
To: Eggert, Lars; Xuxiaohu
Cc: [email-masked]; IETF
Subject: Re: Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Yours,
Joel

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 14 Jan 2014 11:00:41 +0000

On 13/01/2014 19:09, Scott Brim wrote:
> I'm concerned about TCP-over-X.25 scenarios.
... and how many b/s of that exist in the universe!

Stewart

---

From: Scott Brim <[email-masked]>
Date: Tue, 14 Jan 2014 08:11:38 -0500

> ... and how many b/s of that exist in the universe!

Stewart: none that I know of, of course, but it was in production at a
significant time in Internet history and was one of our first experiences
with multiple layers each trying to provide transport control and thereby
destroying goodput. I'll never forget it. When I think of two layers each
trying to do congestion management,  with no way to coordinate with each
other, that's the first example that comes to mind.

Scott

---

From: Scott Brim <[email-masked]>
Date: Tue, 14 Jan 2014 08:52:33 -0500

Now that I'm at a real keyboard, and since at least one person didn't
get what I was trying to say with my reference, let me try to explain
more clearly.

Transport is pretty arcane. We sometimes get it more or less right
when we're dealing with a single instance of it in endpoints and
routers/switches. In this case, if we add congestion management to the
encapsulating UDP, we have two possible instances of the same function
stacked on top of each other, where each has no way of knowing whether
the other exists, if so what it's doing, or if there's any way to
communicate with it. We have examples in the past where we have got
this badly wrong. IMHO this is more likely to be a problem than not.

The best architectural answer I can think of in this case is the one
with the least surprises built in: treat the lower level UDP as just
an encapsulation, not an intelligent transport protocol. Yes there
should be scope for congestion management but that is higher up, where
the endpoints come in to play.

Scott

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 14 Jan 2014 14:20:34 +0000

Yes, the inner (real) transport header is the only meaningful place
to apply congestion avoidance.

Stewart

On 14/01/2014 13:52, Scott Brim wrote:
> .

---

From: "Eggert, Lars" <[email-masked]>
Date: Tue, 14 Jan 2014 15:20:17 +0000

> to apply congestion avoidance.

But what if the inner traffic isn't congestion controlled?

Lars

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 14 Jan 2014 10:23:59 -0500

Isn't that basically the problem of the inner traffic sender, not the 
problem of the tunnel that is carrying the traffic?
Asking tunnel's to solve the problem of applications with undesirable 
behavior seems backwards.

Yours,
Joel

---

From: "Eggert, Lars" <[email-masked]>
Date: Tue, 14 Jan 2014 15:29:38 +0000

Hi,

On 2014-1-14, at 16:23, Joel M. Halpern <[email-masked]> wrote:
> Isn't that basically the problem of the inner traffic sender, not the problem of the tunnel that is carrying the traffic?

no, because the sender of the inner traffic may be blasting some L2 traffic, for an L2 where that is OK behavior. But that traffic is now being encapsulated inside UDP and can hence go anywhere on the net *without the sender being aware of this*.

> Asking tunnel's to solve the problem of applications with undesirable behavior seems backwards.

It is the *tunnel* that performs the encapsulation and allows that traffic to go places it couldn't before. And so it's the tunnel's responsibility to make sure that the traffic it injects into the Internet complies with the BCPs we have on congestion control.

Lars

---

From: Scott Brim <[email-masked]>
Date: Tue, 14 Jan 2014 10:39:00 -0500

Lars, I know we're repeating arguments from the last decade. The
choice is between (1) specifying congestion control around the
substrate UDP that can be turned off if it causes problems, or (2)
specifying nothing at this time and adding it later if operators want
it.

I guess if this can be written as a SHOULD, up to the implementor's
discretion, then okay.

Scott

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 14 Jan 2014 10:48:44 -0500

So a customer buys an Ethernet service from an operator.  And it is 
delivered via MPLS.  Possibly across some other operators (Internet as a 
Service.)
If it is tunneled in IP, or GRE (with or without MPLS), then that is fine.
But if I add a UDP header I must suddenly add congestion control at the 
point I add the UDP header?

I can understand (disagree with, but understand) Llyod's argument that 
the UDP header may reach a host, so the checksum might be an issue.  I 
think taht is dealt with by the fact that the host will drop packets 
with 0 UDP checksums.  But I understand the increased reach concern there.

For congestion control?  Adding an in-network control loop?  When the 
UDP encapsulator may not even know what the service is being used for? 
And so is probably more likely to apply an inner congestion control to a 
TCP stream running over IP over Ethernet over MPLS over UDP as to some 
unconstrained flow?

Yours,
Joel

---

From: "Eggert, Lars" <[email-masked]>
Date: Tue, 14 Jan 2014 16:06:21 +0000

Hi,

> I guess if this can be written as a SHOULD, up to the implementor's

I don't think we can leave this up to implementors discretion. We've had IETF consensus that Internet communication requires congestion control at least since RFC2914. A circuit breaker mechanisms seems straightforward to implement.

As is, I object to this document going forward. The minor benefits of getting some better load balancing for MPLS are far outweighed by the risks.

(I'm also going to shut up now, and let others speak. I think I've said my bit.)

Lars

---

From: Alexander Vainshtein <[email-masked]>
Date: Tue, 14 Jan 2014 16:00:09 +0000

Lars, Scott and all,
IMHO and FWIW the problem is not between congestion control requirements being specified as SHOULD or MUST.
Specifying some functionality as SHOULD without describing (even in a non-normative way) how this functionality could be implemented because the protocol specified does not provide any hooks for it is indeed meaningless.
This is the way to guarantee that this functionality will never be implemented even if we discover that in some cases it is needed.

In the case of congestion control the minimal required hooks are:

1. Ability to detect congestion at the tail-end of the tunnel
2. Ability to pass this information to the head-end of the tunnel.

To the best of my understanding, proposed protocol does not provide any of these hooks. Hence specifying any requirements for congestion control does not make any sense to me regardless of the level of these requirements (MUST, SHOULD or MAY) - these requirements cannot be implemented without some redesign of the protocol.

I would also like to note that the declared purpose of the protocol is simplification of ECMP. One of the reasons to use ECMP is that the "fat" flow cannot get the required BW on any specific link... 

Regards,
       Sasha 

> From: mpls [mailto:[email-masked]] On Behalf Of Scott Brim
> Sent: Tuesday, January 14, 2014 5:39 PM
> To: Eggert, Lars
> Cc: [email-masked]; IETF discussion list
> Lars, I know we're repeating arguments from the last decade. The choice is
> between (1) specifying congestion control around the substrate UDP that can
> be turned off if it causes problems, or (2) specifying nothing at this time and
> adding it later if operators want it.
> 
> > no, because the sender of the inner traffic may be blasting some L2 traffic,
> for an L2 where that is OK behavior. But that traffic is now being
> encapsulated inside UDP and can hence go anywhere on the net *without
> the sender being aware of this*.
> >
> > It is the *tunnel* that performs the encapsulation and allows that traffic to
> go places it couldn't before. And so it's the tunnel's responsibility to make
> sure that the traffic it injects into the Internet complies with the BCPs we
> have on congestion control.
> >
> > Lars

---

From: Alexander Vainshtein <[email-masked]>
Date: Tue, 14 Jan 2014 16:05:21 +0000

Joel,
You have said (and I fully agree with you) that the UDP encapsulator does not even know (in some cases) what the service is for.
If this is the case, how will the UDP encapsulator assign entropy source UDP ports to specific MPLS packets it encapsulates without breaking reordering-sensitive micro-flows within this service?

Regards,
       Sasha 

> From: mpls [mailto:[email-masked]] On Behalf Of Joel M. Halpern
> Sent: Tuesday, January 14, 2014 5:49 PM
> To: Eggert, Lars
> Cc: [email-masked]; Scott Brim; IETF discussion list
> So a customer buys an Ethernet service from an operator.  And it is delivered
> via MPLS.  Possibly across some other operators (Internet as a
> Service.)
> But if I add a UDP header I must suddenly add congestion control at the point
> I add the UDP header?
> 
> I can understand (disagree with, but understand) Llyod's argument that the
> UDP header may reach a host, so the checksum might be an issue.  I think
> taht is dealt with by the fact that the host will drop packets with 0 UDP
> checksums.  But I understand the increased reach concern there.
> 
> For congestion control?  Adding an in-network control loop?  When the UDP
> encapsulator may not even know what the service is being used for?
> And so is probably more likely to apply an inner congestion control to a TCP
> stream running over IP over Ethernet over MPLS over UDP as to some
> unconstrained flow?
> 
> Yours,
> Joel
>

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 14 Jan 2014 11:22:02 -0500

> (I'm also going to shut up now, and let others speak. I think I've said my bit.)

I'm in basic agreement.

Assuming we might all agree that there are conceivable scenarios where
a circuit breaker mechanism would be useful, is the real issue that
we don't all agree that it could be implemented in a way that's not
burdensome and doesn't degrade performance unnecessarily?

Or is there still fundamental disagreement about whether the scenarios
where the circuit breaker is useful are even valid?

-- 
Wes Eddy
MTI Systems

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 14 Jan 2014 03:32:09 +0000

Hi Curtis and Joel,

Thanks a lot for your detailed comments.

Hi Lars,

I wonder whether the following compromised text for the Congestion Consideration section of this doc is roughly acceptable to you.

Since the MPLS-in-UDP encapsulation causes MPLS packets to be forwarded through "UDP tunnels", the congestion control guidelines for UDP tunnels as defined in Section 3.1.3 of [RFC5405] SHOULD be followed. Specifically, MPLS can carry a number of different protocols as payloads. When the MPLS payload traffic is IP-based and congestion-controlled, the UDP tunnel SHOULD NOT employ its own congestion control mechanism, because congestion losses of tunneled traffic will already trigger an appropriate congestion response at the original senders of the tunneled traffic. When the MPLS payload traffic is not known to be IP-based, or is known to be IP-based but not congestion-controlled, the UDP tunnel SHOULD employ an appropriate congestion control mechanism which is outside the scope of this document.

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: Curtis Villamizar [mailto:[email-masked]]
> 发送时间: 2014年1月14日 9:33
> 收件人: Joel M. Halpern
> 抄送: Eggert, Lars; Xuxiaohu; [email-masked]; Scott Brim; IETF
> In message <[email-masked]>
> "Joel M. Halpern" writes:
> 
> > 1) That the UDP encapsulation of MPLS can only be used when the
> > devices performing the encapsulation and decapsulation know what
> > protocol (above IP, if IP is being carried; above Ethernet and IP if
> > the frame is Ethernet carrying IP) is being used
> > 2) That those devices need to engage in a congestion control protocol
> > if the carried packets are not TCP, SCTP, or DCCP?
> >
> I agree that the requirement for the equipment to "know" what is in the MPLS
> traffic is ridiculous.  A provider generally does know whether the traffic will be
> mostly IP or mostly PW and if PW mostly IP inside the PW.
> 
> If stating that implementations SHOULD support a form of congestion control
> satisfies Lars, then its fine to add it.  And then ignore it.
> 
> Even if the document said MUST it is likely to be ignored in practice through a
> non-RFC compliant configuration knob.
> 
> As to whether MPLS tunneled traffic in UDP could end up on the plain old
> Internet, that is exactly what most providers in effect do today with MPLS.  The
> high margin legacy L2 protocols, such as TDM, ATM, FR, all ride on PW and
> MPLS over the very same infrastructure as plain IP.
> The low traffic volume high paying traffic gets queueing priority based on
> paying so much for it and an expectation of good or perfect service and the
> traffic volume is so low that the plain IP traffic doesn't notice.  If a UDP and IP
> header got stuck in there, not much would change.  And certainly any
> congestion control would be disabled.
> 
> If OTOH, some end customer put their NFSv4 over UDP stream or their
> bittorrent stream over MPLS over UDP, it would behave just the same (just as
> badly but no worse) as plain NFSv4 over UDP over IP or plain bittorrent.  No
> change.  And the network hostile bittorrent site isn't going to enable
> congestion control on the UDP tunnel either.
> 
> So this whole discussion is over nothing.  But if putting a SHOULD in a
> document ends the discussion, it might be worth it.  The technical sticky point
> is knowing that the congestion is occuring.  Perhaps a LM like packet on
> another UDP port could be exchanged with the same caveats about inaccuracy
> if implemented in software and inaccuracy if reordering occurs, and a
> recommendation can be made to do something to introduce drop if congestion
> is indicated by this method.  How to introduce drop should be an
> implementation detail, a form of AQM or leaky bucket being possible choices,
> but entirely a local matter.
> Since it is a SHOULD and if no real customer ever wants it we can all just ignore
> it.
> 
> Curtis
> 
> 
> > >>> MPLS is not a forwarding protocol - it's a connectivity protocol.
> > >
> > > typically specific to the protocol being encapsulated (e.g., RTP
> > > uses RTCP feedback to derive loss information, etc.) And the
> > > reaction to congestion is also dependent on the protocol being
> > > encapsulated (does it support multiple rates or only on/off, what
> > > timescales are OK for reaction, etc.)
> > >

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 13 Jan 2014 15:31:04 -0500

RFC5405

   However, if the IP traffic in the tunnel is known to not be
   congestion-controlled, additional measures are RECOMMENDED in order
   to limit the impact of the tunneled traffic on other traffic
   sharing the path.

RFC5405

   | for tunnels carrying IP Traffic,                        | 3.1.3   |
   | SHOULD NOT perform congestion control                   |         |
   |                                                         |         |
   | for non-IP tunnels or rate not determined by traffic,   | 3.1.3   |
   | SHOULD perform congestion control                       |         |

RECOMMENDED == SHOULD

IMHO the intended usage of MPLS over UDP, like the widespread usage of
PW, will not be deployed in a situation where congestion control is
applicable and therefore will not be deployed with congestion control.
And nothing bad will happen.  If we want to define an optional
congestion control in MPLS over UDP, that would meet the RFC5405
recommendation above.

Curtis

ps - Keep in mind that even for the case where Ethernet (L2VPN) over
PW over MPLS over UDP over IP over some L2 is run, the thing run over
the L2VPN Ethernet is predominantly IP.

In message <[email-masked]>
"Eggert, Lars" writes:

--===============1602757442944542782==
Content-Language: en-US
Content-Type: multipart/signed;
	boundary="Apple-Mail=_F966CFC8-CF7E-4540-8EA2-FE7CAF7DBF1D";
	protocol="application/pgp-signature"; micalg=pgp-sha1

--Apple-Mail=_F966CFC8-CF7E-4540-8EA2-FE7CAF7DBF1D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=GB2312

Hi,

On 2014-1-13, at 4:40, Xuxiaohu <[email-masked]> wrote:
>> I don't think it's right to try to solve this in MPLS, because MPLS =
is not a
>> forwarding protocol - it's a connectivity protocol.

right, MPLS is the wrong place to address is. The UDP encaps/decaps =
function needs to have this functionality.

>> control is either left to something above UDP or ignored (left to =
queue
>> management).

There are several cases, see Section 3.1.3 of RFC5405. MPLS-over-UDP can =
fall into any of the three cases, depending on what traffic is inside =
the LSP being encapsulated.=20

You'll notice that RFC5405 for the first case - encapsulation of =
IP-based congestion controlled "normal" Internet traffic - even says =
that the tunnel SHOULD NOT employ any congestion control scheme of its =
own. Having layered control loops fighting is not productive.

The issue with MPLS-in-UDP (and GRE-in-UDP, and any other encaps scheme =
that can carry non-IP traffic) are with cases two and three. When the =
workload that is being encapsulated isn't known to be congestion =
controlled by its endpoints, it is the obligation of the tunnel to =
detect congestion and react to it by reducing the traffic volume. =
Because for the rest of the network, that tunnel is the UDP sender, and =
we have IETF consensus that we don't want UDP senders that don't react =
to congestion on the net. (That's one of the main reasons for the =
existence of the RMCAT WG - we don't want non-congestion-controlled RTP =
media traffic on the net.)

The key difference between putting MPLS e.g. into IP compared to putting =
it into UDP is that once it's in UDP, it can go pretty much anywhere on =
the net, because UDP traverses NATs and firewalls much more easily than =
IP traffic with a rare protocol number does.

>> managing its traffic. MPLS gives you paths, it doesn't push packets =
over them.

Right. However, once you slap a UDP header on a packet during =
encapsulation, you now subjected yourself to the rules for Internet UDP =
senders. Those are documented in RFC5405, and require the tunnel to =
implement some sort of congestion detection and control. I'd personally =
consider a circuit breaker mechanism sufficient, like RTP and I think =
PWE are using.

> Fully agree. The congestion control should be performed either by the =
UDP tunnel itself or the client of MPLS. In the former case, it'd better =
to specify the practical congestion control mechanisms (if there were =
any) in a generic draft (e.g., RFC5405bis) and then any use of the UDP =
tunnel could refer to that generic draft with regard to congestion =
control.

The general concept of a circuit breaker is easy enough that it doesn't =
really need to be written down. And it wouldn't be possible to describe =
it in a generic fashion, because congestion detection is typically =
specific to the protocol being encapsulated (e.g., RTP uses RTCP =
feedback to derive loss information, etc.) And the reaction to =
congestion is also dependent on the protocol being encapsulated (does it =
support multiple rates or only on/off, what timescales are OK for =
reaction, etc.)

> In the latter case, if the client of MPLS is TCP-friendly, that is =
great. Otherwise (e.g., circuit emulation service), it shouldn't be =
deployed on the Internet at all, just as has been pointed out in =
RFC3985, therefore there is no need for any specific congestion control =
mechanism on the client.
>=20
>=20
> The above choice seems no conflict with the following congestion =
control guidelines as quoted from Section 3.1.1 of RFC5405, as those =
non-TCP-friendly traffic would be transported over a provisioned path, =
rather than on the Internet.
>=20
>   "...Finally, some bulk transfer applications may choose not to =
implement
>   means a safe practice for operation in the Internet."

How is that in conflict? Both quotes say that Internet traffic needs =
congestion control, which is a restatement of RFC2914.

Lars

--Apple-Mail=_F966CFC8-CF7E-4540-8EA2-FE7CAF7DBF1D
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="signature.asc"
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Message signed with OpenPGP using GPGMail

--Apple-Mail=_F966CFC8-CF7E-4540-8EA2-FE7CAF7DBF1D--

--===============1602757442944542782==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

--===============1602757442944542782==--

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 13 Jan 2014 20:32:46 -0500

Joel,

I agree that the requirement for the equipment to "know" what is in
the MPLS traffic is ridiculous.  A provider generally does know
whether the traffic will be mostly IP or mostly PW and if PW mostly IP
inside the PW.

If stating that implementations SHOULD support a form of congestion
control satisfies Lars, then its fine to add it.  And then ignore it.

Even if the document said MUST it is likely to be ignored in practice
through a non-RFC compliant configuration knob.

As to whether MPLS tunneled traffic in UDP could end up on the plain
old Internet, that is exactly what most providers in effect do today
with MPLS.  The high margin legacy L2 protocols, such as TDM, ATM, FR,
all ride on PW and MPLS over the very same infrastructure as plain IP.
The low traffic volume high paying traffic gets queueing priority
based on paying so much for it and an expectation of good or perfect
service and the traffic volume is so low that the plain IP traffic
doesn't notice.  If a UDP and IP header got stuck in there, not much
would change.  And certainly any congestion control would be disabled.

If OTOH, some end customer put their NFSv4 over UDP stream or their
bittorrent stream over MPLS over UDP, it would behave just the same
(just as badly but no worse) as plain NFSv4 over UDP over IP or plain
bittorrent.  No change.  And the network hostile bittorrent site isn't
going to enable congestion control on the UDP tunnel either.

So this whole discussion is over nothing.  But if putting a SHOULD in
a document ends the discussion, it might be worth it.  The technical
sticky point is knowing that the congestion is occuring.  Perhaps a LM
like packet on another UDP port could be exchanged with the same
caveats about inaccuracy if implemented in software and inaccuracy if
reordering occurs, and a recommendation can be made to do something to
introduce drop if congestion is indicated by this method.  How to
introduce drop should be an implementation detail, a form of AQM or
leaky bucket being possible choices, but entirely a local matter.
Since it is a SHOULD and if no real customer ever wants it we can all
just ignore it.

Curtis

---

From: <[email-masked]>
Date: Tue, 14 Jan 2014 21:44:29 +0000

The HDLC part here is last link, not the scope of the whole path.
Any 'low' bit error rate given actually becomes quite high once
you consider no of bits per packet and line rate...

Do read Jonathan Stone's papers on where errors creep in -
not just in the link, by at any point along the path, including
regeneration

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Curtis Villamizar [[email-masked]]
Sent: 14 January 2014 20:54
To: Wood L  Dr (Electronic Eng)
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

> http://about.me/lloydwood

OK, perhaps if you are running MPLS/UDP/IP over HDLC and the HDLC
configuration is set to count FCS errors but not drop you will still
*really* need the UDP checksum.  Otherwise its isn't going to do much
for you.  Any checksum is really bad for some types of errors such as
chunk reordering and multiple bit errors.

Maybe on HDLC or PPP with 16 bit CRC you may see a low error rate, but
in theory that would be much less than 10^-5 since few multiple bit
errors will be coincidence match the CRC, even for a 16 bit CRC.

I suspect most routers would be able to do the checksum anyway and for
modern links if they come up with a zero error count that's fine.

<ot>

Modern OTN based transport networks use forward error correction FEC
which accounts for a fair amount of overhead and a lot of processing
gates on the receiving end.  The measure of effectiveness of given FEC
is in dB with 10 dB being a reduction of a factor of 10 in bit errors
and typical FEC in the high tens of dB.  The target corrected error
rate is often 10^-15 or one bit error in 24 hours for 10 Gb/s, one bit
error in 2.5 hours for 100 Gb/s.  Any link with corrected bit error
rates approaching 10^-12 is taken out of service.  This is roughly
equivalent to the old ES (errored seconds) and SES (severely errored
seconds) metric where a ES is one second with any bit errors and an
SES is one second with 10 or more errors (I think its 10).  More than
some number of ES or SES and a link is taken down.  The uncorrected
errors are passed through.

A packet may traverse an entire continent with 2-3 such links
separated by regeneration or could stop at a number of routers along
the way.  Typically today the router uses 10GbE or 100GbE (growing
use) which are then passed as a bit stream in the transport network.
At the other end the uncorrected errors from transport are picked up
by Ethernet 32 bit FCS.  Since a 32 bit FCS picks up 100% of single
bit errors and most instances where a small number of bits are in
error, and all but 1 in 2^32 where many bits are in error, few errors
are going to get through.  If GFP is used, the per packet FCS is
checked at each hop and for GFP-T also checked end to end.

A bad local ethernet is more likely to contribute an error (again
better than 1 in 2^32 detection is expected) due to something like a
bad CAT-{5,5e,6} connection or too many sharp turns.  A DSL or DOCSIS
link is also more likely to contribute an error.  With CRC32 on all
links and no bad hardware in between (ie: circa 1990s equipment with
no parity RAM and no correction on DMA, buses, etc) you would expect
on the order of 10^-8 errors (10^-9 per hop, a few errored hops).

For example, two hosts on my home LAN had non-zero tcp checksums.
Each had < 10^-6 packet error rate.  It is hard to tell if this is
host errors at the other end.  The only hosts I have with non-zero are
on the service provider DMZ LAN so that would include any bot attacks,
etc, where sending hosts could be old junk.  Host behind those have
zero UDP and TCP checksum errors.  This seems similar to Stewart's
quick check.

In the T1/T3 days the transport layer just had parity and just counted
parity errors.  Providers in those days were notorious for ignoring ES
and SES counters until the customer complained.  HDLC then had its 16
bit CRC, optional 32 bit.  If an ISP wasn't paying attention to their
HDLC error counters then it was up to the IP end customer to complain
and hope the problem got escallated rather than dropped.

</ot>

As to whether congestion control is in practice needed see
http://www.ietf.org/mail-archive/web/mpls/current/msg11222.html

Its fine to make them both optional and to make congestion control
mechanisms out of scope and the topic of a later document if needed.

Curtis

> >>> -----ÓÊ¼þÔ­¼þ-----
> >>> ·¢¼þÈË: mpls [mailto:[email-masked]] ´ú±í Eggert, Lars
> >>> ·¢ËÍÊ±¼ä: 2014Äê1ÔÂ8ÈÕ 18:22
> >>> ÊÕ¼þÈË: IETF
> >>> ³ËÍ: [email-masked]
> >>> Ö÷Ìâ: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS

---

From: <[email-masked]>
Date: Wed, 15 Jan 2014 05:39:59 +0000

Curtis

http://lmgtfy.com/?q=Jonathan+Stone+CRC+checksum

HDLC is just whatever is over the last hop. You said HDLC, I reused that as
an example.

Any link technology could be substituted - 10Mbps Ethernet, say, though
you'd criticise that as not being 10Gbps Ethernet and therefore out of date.

Again, the point is that the link check is not end-to-end, and that errors
can creep in from the most unexpected places. By analogy with security,
if I have security across each hop, why would I need security end-to-end?
I already have it across each hop! Each link is highly and absolutely
unrbreakably secure! What is this end-to-end of which you speak?

If you don't get that point because it's a bit abstract and timeless, that's
fine. (and if you have to explain the joke, the joke wasn't funny.)
The link CRC doesn't apply across the entire path; do the maths for
the path and a series of concatenated links.

[As it happens, I'm familiar with UDP/IP/HDLC internet infrastructure installed
within the decade and  in daily operational use to deliver imagery from orbit.
But the paper I wrote on that dates from 2007, so is old and
won't be of interest to you.]

Wait, this thread is all about putting 90s MPLS technology over UDP
technology specified in 1980. Clearly, if MPLS has to rely on an older
technology in this way, the MPLS crowd should give up and go home. 

We've learned repeatedly that zero checksums are a bad idea. IPv6
RFC2460:

        Unlike IPv4, when UDP packets are originated by an IPv6 node,
         the UDP checksum is not optional.  That is, whenever
         originating a UDP packet, an IPv6 node must compute a UDP
         checksum over the packet and the pseudo-header, and, if that
         computation yields a result of zero, it must be changed to hex
         FFFF for placement in the UDP header.  IPv6 receivers must
         discard UDP packets containing a zero checksum, and should log
         the error.

which RFC6935 simply rewote as inconvenient to tunnelers without
considering how it affected everything else in the network.

Those that do not learn from history are doomed to repeat it.
(Unless it's recent history, in which case they're doomed to reject it
as irrelevant.)

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Curtis Villamizar [[email-masked]]
Sent: 15 January 2014 01:42
To: Wood L  Dr (Electronic Eng)
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

In message <[email-masked]>
[email-masked] writes:

> The HDLC part here is last link, not the scope of the whole path.  Any
> 'low' bit error rate given actually becomes quite high once you
> consider no of bits per packet and line rate...
>
> Do read Jonathan Stone's papers on where errors creep in - not just in
> the link, by at any point along the path, including regeneration
>
> Lloyd Wood

Lloyd,

There is no HDLC hop.  No one has used HDLC for internet
infrastructure in ages.  It was a joke, like Scott's comment on
wanting to use X.25.  HDLC was disappearing when the Stone/Partridge
Sigcomm 2000 paper was written.

Links please.  And how old is that paper?  Not another 15 year old
work is it?

If you have one bit error per day, how many packets do you lose that
day?  (hint: one).

If you have one bit error per day, how many undetactable packet errors
do you have?  (hint: crc32 gets all one bit errors, therefore zero).

10^-12 bit errors is one per 10 second on 100 Gb/s, one per 100 second
on 10 Gb/s and is generally considered high enough to take a link down
immediately.  A 1500 byte packet is 12,000 bits, about ~10^4.  That
would yield a packet rate as high as 10^-8 if bit errors were mostly
one bit error per packet.  In that case all errors would be
detectable.  It is only when there are a lot of bit errors or more per
packet that the CRC can be defeated and then its about 10^-9 chance.

So at an error rate much less than 10^-8 packets (tightly bunched
errors with multiple bit errors per packet) some 10^-9 might be
undetectable with a CRC32.  One packet every 10^6 seconds at 100 Gb/s
could have an undetectable error.  About one undetectable error a day
or one a week for continuous full out 100 Gb/s link.

Note that the same low error rate does not apply to a GbE or 10GbE
over colored optics over ROADM in the metro since there is no FEC
there.  It also may not apply to the enterprise or campus Ethernets.
In those hops the error rate is likely to be higher.  Needless to say,
wireless hops can have very high error rates.

This is why it could make sense to have the UDP checksum optional in
MPLS over UDP.  It wouldn't hurt to provide the checksums but in some
cases it might be OK to disable them.  That is what SHOULD is for in
an IETF document.

Curtis

---

From: <[email-masked]>
Date: Wed, 15 Jan 2014 05:45:43 +0000

this draft should be about mpls in TCP - a TCP tunnel.

That will fix all congestion concerns.

I look forward to reading justification of why TCP checksums can be turned off.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: mpls [[email-masked]] On Behalf Of Curtis Villamizar [[email-masked]]
Sent: 15 January 2014 01:00
To: Eggert, Lars
Cc: [email-masked]; Scott Brim; IETF discussion list
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt>  (Encapsulating MPLS in UDP) to Proposed Standard

In message <[email-masked]>
"Eggert, Lars" writes:

> no, because the sender of the inner traffic may be blasting some
> L2traffic, for an L2 where that is OK behavior. But that traffic is
> nowbeing encapsulated inside UDP and can hence go anywhere on the
> net*without the sender being aware of this*.

That application would be a PW application and it would be more
appropriate to fix that in PW if there is consensus for a need to do
so, which afaik there is not.

> > Asking tunnel's to solve the problem of applications with
> > undesirablebehavior seems backwards.
>
> It is the *tunnel* that performs the encapsulation and allows
> thattraffic to go places it couldn't before. And so it's the
> tunnel'sresponsibility to make sure that the traffic it injects into
> theInternet complies with the BCPs we have on congestion control.
>
> Lars

If it is a service provider encapsulating traffic within their own
network, then they know what they are doing.  That is the anticipated
use and among that community there is no consensus for need for
congestion control.

If it is some hostile hosts trying to send MPLS over UDP over IP,
they, being hostile, are going to disable any congestion control.
Besides, no hostile host has a T1 to tunnel over the Internet so they
would be sending the same traffic they would normally just send of UDP
over IP.

Anything made up of frames (Ethernet, ATM, FR) over PW over MPLS is
carrying IP and if frames drop, the IP applications see the drop and
behave just as they would for any drop.  (ATM shreadding thread to
/dev/null please).

If congestion aware or using a congestion aware transport, the top
level applications are still congestion aware.  If congestion
ignoreant, they are still congestion ignoreant.  If hostile, they are
still hostile.

Back to draft-ietf-mpls-in-udp.  I think the most recent text proposed
by the author is fine.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 12:25:23 -0500

In message <[email-masked]>
Xuxiaohu writes:
 
> I wonder whether the following compromised text for the Congestion
> Consideration section of this doc is roughly acceptable to you.
>  
>    Since the MPLS-in-UDP encapsulation causes MPLS packets to be
>    forwarded through "UDP tunnels", the congestion control guidelines
>    for UDP tunnels as defined in Section 3.1.3 of [RFC5405] SHOULD be
>    followed. Specifically, MPLS can carry a number of different
>    protocols as payloads. When the MPLS payload traffic is IP-based
>    and congestion-controlled, the UDP tunnel SHOULD NOT employ its own
>    congestion control mechanism, because congestion losses of tunneled
>    traffic will already trigger an appropriate congestion response at
>    the original senders of the tunneled traffic. When the MPLS payload
>    traffic is not known to be IP-based, or is known to be IP-based but
>    not congestion-controlled, the UDP tunnel SHOULD employ an
>    appropriate congestion control mechanism which is outside the scope
>    of this document.
> 
> Best regards,
> Xiaohu

This works for me.  If deployments indicate a need for congestion
control another document can be written.  I doubt it will be needed.
If someone (Lars perhaps) feels a pressing need to wirte a congestion
control for MPLS over UDP document (or for any tunneling over UDP)
regardless of deployment experience, then they can go ahead.

Whether it works for Lars may be the issue.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 15:54:10 -0500

<ot>

</ot>

Curtis

> >>> -----�ʼ�ԭ��-----
> >>> ������: mpls [mailto:[email-masked]] ���� Eggert, Lars
> >>> ����ʱ��: 2014��1��8�� 18:22
> >>> �ռ���: IETF
> >>> ����: [email-masked]
> >>> ����: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 19:40:34 -0500

In message <[email-masked]>
 
> But what if the inner traffic isn't congestion controlled?

Lars,

The exact same thing will happen in all of the following cases:

  NON-congestion controlled application --over--
  UDP --over-- IP --over-- L2

  NON-congestion controlled application --over--
  UDP --over-- IP --over-- MPLS --over-- L2

  NON-congestion controlled application --over--
  UDP --over-- IP --over-- MPLS --over-- UDP --over-- IP --over-- L2

The non-congestion controlled application is what needs fixing.

It would be wrong to try to put congestion control at every layer
underneath the non-congestion controlled application except perhaps as
a transparent replacement for UDP and that is out of scope for a
draft-ietf-mpls-in-udp discussion.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 19:46:26 -0500

In message <[email-masked]>
 
> behavior seems backwards.

Or perhaps the job of some form of very fair AQM like SFQ.

Any tunneling complicates the AQM notion of what a flow is and
therefore may affect its assessment of what is fair in bad ways.

This too is out of scope for a draft-ietf-mpls-in-udp discussion.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 20:00:12 -0500

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 20:13:33 -0500

> where the circuit breaker is useful are even valid?

A circuit breaker (drop all traffic on sign of congestion) at the MPLS
over UDP level would be the worst thing you could specify and IMO it
would guarentee zero implementations.

A circuit breaker might be appropriate for TDM over PW but nothing
else.  This makes sense because TDM cannot change its bandwidth so the
only choice is to shut it off.  TDM over PW is an application that can
run over MPLS (or GRE or L2TP).  The best place to put that circuit
breaker (if anywhere at all) is in TDM over PW.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 20:42:31 -0500

Lloyd,

Curtis

---

From: Edward Crabbe <[email-masked]>
Date: Tue, 14 Jan 2014 12:28:14 -0800

+100

the same problem would exist with or without the additional encapsulation
(this includes any L2 encaps that may exist in the case of L2 tunelling.)
 The only difference is /reach/.  This same concern exists *most* common
tunnels types in the internet today.

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 15 Jan 2014 03:06:44 +0000

> -----邮件原件-----
> 发件人: mpls [mailto:[email-masked]] 代表 Curtis Villamizar
> 发送时间: 2014年1月15日 9:14
> 收件人: Wesley Eddy
> 抄送: Scott Brim; Eggert, Lars; IETF discussion list; [email-masked]
> > >> specifying nothing at this time and adding it later if operators
> > >> want it.
> > >>
> > > I don't think we can leave this up to implementors discretion. We've had
> IETF consensus that Internet communication requires congestion control at least
> since RFC2914. A circuit breaker mechanisms seems straightforward to
> implement.
> > >
> > > As is, I object to this document going forward. The minor benefits of getting
> some better load balancing for MPLS are far outweighed by the risks.
> > >
> > a circuit breaker mechanism would be useful, is the real issue that we
> > don't all agree that it could be implemented in a way that's not
> A circuit breaker (drop all traffic on sign of congestion) at the MPLS over UDP
> level would be the worst thing you could specify and IMO it would guarentee
> zero implementations.
> 
> A circuit breaker might be appropriate for TDM over PW but nothing else.  This
> makes sense because TDM cannot change its bandwidth so the only choice is to
> shut it off.  TDM over PW is an application that can run over MPLS (or GRE or
> L2TP).  The best place to put that circuit breaker (if anywhere at all) is in TDM
> over PW.

I couldn't agree more with Curtis's point. Since RFC3985 has already made a detailed description of congestion considerations about PWs, it seems that the simplest way is just to add a reference to that RFC in this doc, in addition to the text I have proposed in a previous email. The advantage of this choice is: since there is only one place where the congestion control mechanism for MPLS/PW traffic is specified, it's much easier for operators and implementors to refer. In addition, once a more practical congestion control mechanism was found in the future, it only needs to update one RFC, rather than updating many RFCs.

Best regards,
Xiaohu

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 15 Jan 2014 13:35:20 -0500

> http://lmgtfy.com/?q=Jonathan+Stone+CRC+checksum

That is the Sigcomm 2000 paper that a number of people have said is no
longer relevant, in some cases giving quite a bit of detail about the
error causes in the paper and why these will be rare today.  I thought
you were refering to something more recent.

> HDLC is just whatever is over the last hop. You said HDLC, I reused that as

I gave HDLC as an example because in 2000 it was in use and a source
of errors but in 2014 it is likely not to be in use just about
anywhere in North America and Europe at least.

> you'd criticise that as not being 10Gbps Ethernet and therefore out of date.

My point was that if you substituted 10Mbps Ethernet with a 32 bit
FCS, for HDLC which often just counted and passed along errored
packets, you would have far fewer errors, almost none.

> unrbreakably secure! What is this end-to-end of which you speak?

You have security end to end because someone may have a motivation to
monitor or alter your packet.

If you have robust L2 FCS, then there is no one with a motivation to
disable that FCS and corrupt your data.  If they did for MPLS carrying
IP, then there is a check, albiet not a great one, that might catch
it.  If the MPLS payload is PW carrying a L2 carrying IP, same
applies.

Note that if the PW payload is TDM, Ethernet, or most other L2, at
least a checksum if not more is available in the payload so the errors
would be detected and the customer of the PW would complain.  Those
types of complaints have been a complete non-issue.

> the path and a series of concatenated links.

The joke is that you didn't realize any mention of HDLC being used
now, just like X.25 being used now, was a joke.  It actually isn't
funny at all, except perhaps until someone doesn't get it.

> won't be of interest to you.]

That is clearly the exception.  TDM is still alive and well in the
third world and in access parts of the developed world that have not
been upgraded but elsewhere alternative exist for terestrial Internet
and TDM is gone from any non-third world network core that I am aware
of.

So don't run MPLS over UDP without a UDP checksum *on that
infrastructure* if you can avoid it.  But for most modern
infrastructures MPLS over UDP without a UDP checksum would be fine.

We are asking for a SHOULD, not a MUST NOT wrt UDP checksum.

> technology in this way, the MPLS crowd should give up and go home. 

And MPLS carried nothing but IP in those days so from a error
detection standpoint it was and still is a NOOP.

> considering how it affected everything else in the network.

How it affects other everything else in the network should be
considered before ignoring a SHOULD, as is always the case.

> http://about.me/lloydwood

Lloyd.  There are a lot of papers that are still relevant.  Almost all
of the causes of errors studied in the 2000 network.  Apparently an
exception is your HDLC infrastructure, and that would be gone too as
an issue if 32 bit CRC is enabled for HDLC and packets dropped on
error, not just counted.

That paper reported:

   2,209 M packets

   468,434 errors
   389,934 ACK-of-FIN bug (old Window NT bug, fixed)
    78,500 remaining
   Other errors cited but often not quantified
     CRLF replacement (thought to be Solaris bug)
     VJHC or other header comprssion bug
     possible PowerMac OS-X bug (fixed by publication)
     router memory error (ECC is used now)
     bad host DMA (PCIe has CRC32 today)

The paper has a breakdown by type of error and discussion of the
causes.  In some cases a particular type of error came mostly from a
small set of hosts but the cause is only likely to be host related
and the entire type of error category can't with certainty be
attributed to host error.

   Bad Hosts

      Another surprise in our traces was the large fraction of errors
      where are due to persistantly-misbehaving hosts.

In fact the paper says:

   In general, link errors should be caught by the CRC.  However there
   are cases where the link level protocol can interact to cause
   higher level checksum errors.  The most notable situation is header
   compression and we looked vigorously for errors of this sort.

Link errors are really not considered in the paper as a significant
source of errors.  Router memory and other hardware errors are cited
but with today's hardware those types of errors should also be gone.

I discussed the sources of errors in this paper previously in
  http://www.ietf.org/mail-archive/web/mpls/current/msg11247.html
You did respond to that but only by top posting and ignoring the
discussion in that email of the causes of errors in the paper.

"No longer relevant" does not mean "not good work".  It would be worth
it for the Stone/Partridge study to be repeated.  For this context it
would have to be with cooperation of a provider and over the type of
infrastructure that MPLS is intended to be run.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 15 Jan 2014 14:19:56 -0500

Lloyd,

Suggesting MPLS over TCP brings us back to the X.25 comment.

You can do PPP in TCP.  It has the benefit of getting a tunnle past
firewalls.  It has the drawback of TCP over IP over PPP over TCP where
the upper and lower TCPs don't know about each other and interact,
with the upper TCP doing redundant retransmits and a lot of
unnecessary retransmits when the lower TCP is stalled.  As a host
solution this particular tunnel over TCP serves a purpose.

As a router solution, a tunnel over TCP would cause massive redundant
or unnecessary retransmits due to lots of TCP running over it unaware
that retransmits are occurring at a lower layer.

Another problem is the TCP state that would have to be held in
hardware.  Normally the router functions that use TCP are in the
control plane and in software.  All packets would also have to be
bufferred in hardware until acknowledged.

There are the save feasibility issues with a TCP checksum in most high
end hardware.

MPLS over TCP would likely be only feasible if done in software and
therefore is not an appropriate solution for the deployment scenarios
considered for MPLS over UDP.

There are congestion control mechanisms that would be feasible for
MPLS over UDP but mostly involve feedback, the programming of a leaky
bucket, aka traffic shaper, and (preferably) some form of AQM on the
shaper queue.  This would involve no retransmissions and use the types
of hardware building blocks available in forwarding chips.

Curtis

---

From: joel jaeggli <[email-masked]>
Date: Thu, 16 Jan 2014 09:06:42 -0800

On 1/14/14, 7:29 AM, Eggert, Lars wrote:
>> Isn't that basically the problem of the inner traffic sender, not
>> the problem of the tunnel that is carrying the traffic?
> 
> no, because the sender of the inner traffic may be blasting some L2
> traffic, for an L2 where that is OK behavior. But that traffic is now
> being encapsulated inside UDP and can hence go anywhere on the net
> *without the sender being aware of this*.
> 
>> undesirable behavior seems backwards.
> 
> It is the *tunnel* that performs the encapsulation and allows that
> traffic to go places it couldn't before. And so it's the tunnel's
> responsibility to make sure that the traffic it injects into the
> Internet complies with the BCPs we have on congestion control.

There seems assertion on the part of transport that tunnel endpoints
should be aware of congestion on intermediate hops. This seems to be the
assertion here, it certainly is with respect to AMT.  This  certainly
not a property that we demand of routers in other contexts.

my observations

  These tunnels are stateless

  The endpoints not the encapsulators have visibility into the
end-to-end loss
  latency properties of the path.

  the encapsulator is an intermediate hop, similar to any other router
in the path.

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 16 Jan 2014 17:19:50 +0000

Hi,

> These tunnels are stateless

yep. (But they don't have to be.)

> end-to-end loss latency properties of the path.

Yep. But when you tunnel some L2 in UDP, apps that were limited to L2 domains - where not reacting to congestion may be OK - can now go over the wider Internet, where this is not OK.

I'd be great if those apps would change. But in the meantime, it's the duty of the encapsulator - who enables this traffic to break out of an L2 domain and go over the wider net - to make sure the traffic it emits conforms to our BCPs.

>  the encapsulator is an intermediate hop, similar to any other router

It's not. For the rest of the network, that encapsulator is indistinguishable from any other app that sends UDP traffic.

UDP is a transport-layer protocol, and we have practices how it is to be used on the net. If you want to use it for encapsulation, you bind yourself to these BCPs.

Look at it the other way: if transport area folks would want to send MPLS packets into the network in some problematic way, I'm sure the routing and ops folks would not be amused.

Lars

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 16 Jan 2014 17:35:25 +0000

> yep. (But they don't have to be.)
Ah, they do if they are to scale. State at speed is really hard. The sort
of systems we are talking about do things like pipeline counters
and it is loooooots of packets later before the counter is actually
incremented.
> Look at it the other way: if transport area folks would want to send MPLS packets into the network in some problematic way, I'm sure the routing and ops folks would not be amused.
The root cause of the problem here is that UDP, has bifurcated into
a general purpose encapsulation.

Stewart

---

From: Ross Callon <[email-masked]>
Date: Thu, 16 Jan 2014 22:32:19 +0000

> yep. (But they don't have to be.)

The tunnels strictly speaking do not have to be stateless. However, if you want routers to actually implement them, and you want to scale in both forwarding speed and number of tunnels, then yes they do have to be stateless. 

Ross
(speaking only as an individual contributor) 

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Eggert, Lars
Sent: Thursday, January 16, 2014 12:20 PM
To: EXT - [email-masked]
Cc: [email-masked]; IETF discussion list
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Hi,

Lars

---

From: <[email-masked]>
Date: Fri, 17 Jan 2014 01:19:41 +0000

Surely you mean minimised state?

The tunnels have to know where the other tunnel endpoint is - state. This is distinct from having a congestion-aware state machine that e.g. TCP includes...

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: mpls [[email-masked]] On Behalf Of Ross Callon [[email-masked]]
Sent: 16 January 2014 22:32
To: Eggert, Lars
Cc: EXT - [email-masked]; [email-masked]; IETF discussion list
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Ross

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Eggert, Lars
Sent: Thursday, January 16, 2014 12:20 PM
To: EXT - [email-masked]
Cc: [email-masked]; IETF discussion list
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Hi,

Lars

---

From: <[email-masked]>
Date: Fri, 17 Jan 2014 03:03:05 +0000

There's an opportunity to define a simple generic IPv6 encapsulating mechanism a la UDP, but
with a payload checksum of varying coverage (header only, partial payload, full payload)
and with the (stronger than UDP) checksum placed at the end of the packet.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: ietf [[email-masked]] On Behalf Of Stewart Bryant [[email-masked]]
Sent: 16 January 2014 17:35
To: Eggert, Lars; Joel Jaeggli
Cc: [email-masked]; IETF discussion list
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Stewart

---

From: joel jaeggli <[email-masked]>
Date: Thu, 16 Jan 2014 23:38:12 -0800

> yep. (But they don't have to be.)

They do in-fact have to be stateless with respect to the contents. They
will no doubt be created in a same distributed forwarding  engines that
currently do IP-MPLS encapsulation and L2-MPLS encapsulation. likewise
their decapsulation will occur in such devices.

> Look at it the other way: if transport area folks would want to send
> MPLS packets into the network in some problematic way, I'm sure the
> routing and ops folks would not be amused.

Today congestion occurs on MPLS networks. intermediate hops that are
congested have no choice but to drop packets.

---

From: joel jaeggli <[email-masked]>
Date: Fri, 17 Jan 2014 12:42:58 -0800

On 1/16/14, 9:35 AM, Stewart Bryant wrote:

>> Look at it the other way: if transport area folks would want to 
>> send MPLS packets into the network in some problematic way, I'm 
>> sure the routing and ops folks would not be amused.
> The root cause of the problem here is that UDP, has bifurcated into a
> general purpose encapsulation.

One observation with respect to general purpose use of UDP as an mpls
encapsulation on the internet as a whole that seems to have been missed
in this discussion is the rather dire prospects for MTU and potential for
fragmentation. Effectively, assuring, that you do not need to fragment
requires as certain amount of coordination.

1500 mtu hops pretty effectively (imho) preclude the use of this
encapsulation on the internet as a whole (if encapsulated traffic is not
aware of a correspondingly lower MTU) this is especially true for L2vpn
where you don't have recourse to PMTUD). As the document notes (section 4)
preventing fragmentation is rather important.

---

From: <[email-masked]>
Date: Fri, 17 Jan 2014 23:00:33 +0000

Yes, it would be unreasonable for the router vendors themselves to design
something with a trailing checksum and pseudo-header check that would meet
their forwarding needs while ensuring reliability.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Curtis Villamizar [[email-masked]]
Sent: 17 January 2014 18:05
To: Wood L  Dr (Electronic Eng)
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

In message <[email-masked]>
[email-masked] writes:

> There's an opportunity to define a simple generic IPv6 encapsulating
> mechanism a la UDP, but with a payload checksum of varying coverage
> (header only, partial payload, full payload) and with the (stronger
> than UDP) checksum placed at the end of the packet.
>
> Lloyd Wood

Go for it!

When a critical mass of routers hash on this new protocol number and
port space we'll let you know and the MPLS WG can obsolete MPLS over
UDP and replace it with MPLS over this new protocol.

Thanks for offering to write this.

Curtis

---

From: <[email-masked]>
Date: Mon, 20 Jan 2014 23:11:05 +0000

> Router vendors focus on real needs identified by customers.  If this
> were to become something customers were bugging them about, then
> they'll do it.

And pollution of ports for other traffic, like congestion, is something
experienced by people who are not customers - the rest of the net.

Which is why this draft does not address the possibility of missent traffic,
and does not address congestion. Not a tunnel customer problem,
therefore not a vendor problem.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Curtis Villamizar [[email-masked]]
Sent: 20 January 2014 22:47
To: Wood L  Dr (Electronic Eng)
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

We are continuing to stray off topic, but...

Router vendors focus on real needs identified by customers.  If this
were to become something customers were bugging them about, then
they'll do it.  OTOH, most hardware in the field today can only track
one 32 bit FCS, given that only one L2 header would need this.  The
ability to run on existing hardware is important.

> UDP and replace it with MPLS over this new protocol.

Providers and router vendors are also aware that most routers look
only for protocol numbers 6 and 17 for load balancing on port
numbers.  So if the motivation of MPLS over UDP is an interim solution
to get ECMP, then not using UDP is not an option.

Curtis

---

From: "Eggert, Lars" <[email-masked]>
Date: Tue, 21 Jan 2014 13:22:50 +0000

Hi,

On 2014-1-17, at 18:19, Curtis Villamizar <[email-masked]> wrote:
> You have made your assertions about your desire to uphold the purity
> of any new UDP applications and adhere to the BCP you wrote.
> 
> You appear to be very nearly alone in this argument and certainly no
> one that works with MPLS is siding with you.

the reason we wrote the RFC when I was TSV AD was that we were seeing a whole bunch of questionable uses of UDP over the eyars and we were having the same arguments over and over. That's why we decided to write down the practices we expect users of UDP to follow. This is yet another such questionable use.

(Also, I don't appreciate you turning this into a personal argument.)

> In the end we can put anything we want in the RFC *but* IETF has never
> truly had the final word on what vendors and operators do in provider
> networks.

Aka the "take my toys and go home" argument. Heard it many times.

> In this case, regardless of what changes are made to the draft,
> implementations will offer at least the option for non-RFC behavior by
> using zero checksums and not using any congestion control.  And
> providers will make use of it, perhaps exclusively.

And there's nothing wrong with that - the BCP even says that one SHOULD NOT use congestion control for some deployment cases.

But for others, one SHOULD. For those, a mechanism needs to be available, i.e., it needs to be specified and implemented.

> The document might as well reflect reality, despite reality not
> conforming to your notions of architectural purity.

I'm sorry, but we have certain architectural principles in the Internet that we have IETF consensus on. At least since RFC2914, that includes the need to have congestion control in place.

There are always special deployment scenarios where these principles do not apply, and we typically explain in applicability statements when out specifications can only be safely used under certain conditions. I don't see any such statement in draft-ietf-mpls-in-udp, which to me means it's targeted at general Internet-wide use.

> The best course of action is to put a SHOULD in regarding checksums
> and put a SHOULD in regarding congestion avoidance.  Even the BCP does
> not go any further than to say a tunneling protocol SHOULD use
> congestion control and there were reasons that the word MUST was not
> acceptable in the BCP.

The SHOULD for congestion control needs to actually describe a mechanism that can be used when needed. It can't be a blanket "you SHOULD use something but we don't tell you what it is"-statement.

> If we are still arguing over two instances of SHOULD vs MUST we have
> wasted a lot of bandwidth on those two words.

It's not SHOULD vs. MUST. It's two SHOULDs, but in both cases it needs to be specified what is to be done. In the case of checksums, that's obvious (calculate it and check it); in the case of congestion control, some actual mechanism needs to be described (e.g., a circuit breaker).

> IMHO The only remaining question is whether the document can go forward
> with the definition of congestion control for MPLS over UDP left out
> of scope and for another document if a need arises.

In my opinion, it cannot.

> If this is not acceptable to you (I doubt it is) please indicate what
> you would like to see in the document and since this is IETF last call
> where consensus matters and no one individual has veto power, we'll
> have to see if there is consensus behind your proposed changes.

I would like the document to specify at the very least a circuit breaker mechanism, that stops the tunneled traffic if severe packet loss is detected along the path.

And this isn't about an "individual veto". This is about a document that is at the moment in violation of IETF consensus at least as far back as RFC2914.

Lars

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 17 Jan 2014 12:19:52 -0500

Lars,

We seem to be in an endless loop here.

Curtis

In message <[email-masked]>
"Eggert, Lars" writes:
 
> Yep. But when you tunnel some L2 in UDP, apps that were limited to L2 =
> domains - where not reacting to congestion may be OK - can now go over =
> the wider Internet, where this is not OK.
>  
> I'd be great if those apps would change. But in the meantime, it's the =
> duty of the encapsulator - who enables this traffic to break out of an =
> L2 domain and go over the wider net - to make sure the traffic it emits =
> conforms to our BCPs.
>  
> It's not. For the rest of the network, that encapsulator is =
> indistinguishable from any other app that sends UDP traffic.
>  
> UDP is a transport-layer protocol, and we have practices how it is to be =
> used on the net. If you want to use it for encapsulation, you bind =
> yourself to these BCPs.
>  
> Look at it the other way: if transport area folks would want to send =
> MPLS packets into the network in some problematic way, I'm sure the =

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 17 Jan 2014 13:02:37 -0500

Lloyd,

If we really want to get silly we can claim that a UDP control block
or socket state on a UDP listen is state and therefore UDP is not a
stateless protocol.

But we don't want to get silly.  Do we?  Or are we long past that
point?

Curtis

PS - type "man setsocopt" on a BSD system to see a list of state, most
of which applies to any socket including UDP sockets.

In message <[email-masked]>
[email-masked] writes:
 
> The tunnels have to know where the other tunnel endpoint is -
> state. This is distinct from having a congestion-aware state machine
> that e.g. TCP includes...
>  
> Lloyd Wood

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 17 Jan 2014 13:05:46 -0500

Go for it!

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 17 Jan 2014 13:16:11 -0500

> > routing and ops folks would not be amused.

They would never notice since MPLS packets are only passed around
internally and in some cases among peer providers.  If you did try to
do that and succeeded in getting even one packet through there are
some security holes to worry about.  Point is that you can't get MPLS
packets into provider networks unless something is very seriously
wrong and the already persistant attackers would have a picnic.

> congested have no choice but to drop packets.

And nothing bad happens because providers that allow any congestion to
occur by provisioning so that it can happen (usually occasionally or
very rarely) carry two kinds of traffic in that network.  One is IP in
MPLS or IP native.  The other is high margin VPN services and emulated
legacy L2 services that are given a different TC value.  What the
providers want to happen on light congestion happens.  Slight drop
gets IP (mostly TCP still to this date) to reduce load.  The high
paying VPN and emulated legacy L2 services are completely unaffected
(as the provider wants given they are paying a lot for that).

So I'm agreeing with you (Joel).

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 20 Jan 2014 17:47:47 -0500

Curtis

---

From: Alia Atlas <[email-masked]>
Date: Tue, 21 Jan 2014 15:20:01 -0500

Apologies for top-posting, but what struck me as very useful that Lars said
is:

"There are always special deployment scenarios where these principles do
not apply, and we typically explain in applicability statements when out
specifications can only be safely used under certain conditions. I don't
see any such statement in draft-ietf-mpls-in-udp, which to me means it's
targeted at general Internet-wide use."

Why don't we add an applicability statement?  That seems easier than trying
to define the appropriate congestion control mechanism which is highly
likely to interact badly with the congestion control applied to the
encapsulated traffic?

Alia

Alia

On Tue, Jan 21, 2014 at 8:22 AM, Eggert, Lars <[email-masked]> wrote:

> the reason we wrote the RFC when I was TSV AD was that we were seeing a
> whole bunch of questionable uses of UDP over the eyars and we were having
> the same arguments over and over. That's why we decided to write down the
> practices we expect users of UDP to follow. This is yet another such
> questionable use.
>
> And there's nothing wrong with that - the BCP even says that one SHOULD
> NOT use congestion control for some deployment cases.
>
> But for others, one SHOULD. For those, a mechanism needs to be available,
> i.e., it needs to be specified and implemented.
>
> I'm sorry, but we have certain architectural principles in the Internet
> that we have IETF consensus on. At least since RFC2914, that includes the
> need to have congestion control in place.
>
> There are always special deployment scenarios where these principles do
> not apply, and we typically explain in applicability statements when out
> specifications can only be safely used under certain conditions. I don't
> see any such statement in draft-ietf-mpls-in-udp, which to me means it's
> targeted at general Internet-wide use.
>
> The SHOULD for congestion control needs to actually describe a mechanism
> that can be used when needed. It can't be a blanket "you SHOULD use
> something but we don't tell you what it is"-statement.
>
> It's not SHOULD vs. MUST. It's two SHOULDs, but in both cases it needs to
> be specified what is to be done. In the case of checksums, that's obvious
> (calculate it and check it); in the case of congestion control, some actual
> mechanism needs to be described (e.g., a circuit breaker).
>
> I would like the document to specify at the very least a circuit breaker
> mechanism, that stops the tunneled traffic if severe packet loss is
> detected along the path.
>
> And this isn't about an "individual veto". This is about a document that
> is at the moment in violation of IETF consensus at least as far back as
> RFC2914.
>
> Lars
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 21 Jan 2014 17:04:17 -0800

+1

Edward Crabbe is right - the encapsulated protocols ought to take care 
of congestion control when using UDP.

When they do NOT - e.g., when using MPLS to transfer traffic that either 
isn't IP or is IP but isn't congestion controlled, then some other 
mechanism needs to be employed.

It's not enough to assume "someone else, somewhere else" will handle this.

I agree with Lars - this needs to be part of the encapsulation protocol, 
because it's not enforced by MPLS, and not provided by UDP.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 21 Jan 2014 17:07:57 -0800

> behavior seems backwards.

By that argument, apps using TCP shouldn't expect the transport to 
control congestion. They ought to control it at the app layer.

Tunneled MPLS, when encapsulated inside UDP, *is* the "application". UDP 
expects the app to deal with congestion, so it's entirely reasonable for 
UDP to expect the tunneling system to do this.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 21 Jan 2014 17:11:32 -0800

> The tunnels strictly speaking do not have to be stateless. However, if you want routers to actually implement them, and you want to scale in both forwarding speed and number of tunnels, then yes th…

There's clearly a problem though:

	- tunnels must be stateless to be efficiently implemented

	- transport layer tunnels must have congestion control

Saying that the only way we can make tunnels cheap is to make them break 
the Internet isn't a good solution.

Maybe it's time to expect something that's inherently costly to end up 
being expensive?

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 21 Jan 2014 17:12:36 -0800

+1

I don't even care if the safety breaker is a little late, e.g., a few
hundreds of packets (as might be needed for efficient implementation).

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 21 Jan 2014 17:14:45 -0800

+1 on all points.

I suspect if more people in transport had the time to track this 
protracted discussion, you'd get the same from them too.

Joe

---

From: Ross Callon <[email-masked]>
Date: Wed, 22 Jan 2014 02:40:00 +0000

If the upper layers (the thing that runs over the tunnel) involves applications over TCP over IP, or if it is otherwise responding to congestion in the same way that we expect anything running over IP to respond to congestion, then we don't want the tunnel to also independently try to respond to congestion (two independent cooks cooking the same meal does not necessarily lead to success). 

If the upper layer does not respond to congestion, then perhaps it shouldn't be running over the open Internet (with or without a tunnel), unless the *total* bandwidth that could be used is inherently quite low. On the other hand, it might want to run within a data center or internally to a service provider network with appropriate provisioning. 

Ross

-----Original Message-----
From: Joe Touch [mailto:[email-masked]] 
Sent: Tuesday, January 21, 2014 8:12 PM
To: Ross Callon; Eggert, Lars
Cc: [email-masked]; IETF discussion list
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Joe

---

From: Scott Brim <[email-masked]>
Date: Tue, 21 Jan 2014 22:50:40 -0500

On Tue, Jan 21, 2014 at 8:07 PM, Joe Touch <[email-masked]> wrote:
> By that argument, apps using TCP shouldn't expect the transport to control
> congestion. They ought to control it at the app layer.
>
> expects the app to deal with congestion, so it's entirely reasonable for UDP
> to expect the tunneling system to do this.

Joe, I believe you are confusing a protocol with an architectural
function. It's a UDP encapsulation, but that encapsulation has nothing
to do with transport, and what runs over it is not an "application".
It may be a client layer (with the encapsulation a service layer), but
that's a relative relationship, not an absolution one about stack
position.  This instance of UDP is way below transport, is just in
fact a bit of lubrication for the packet, and considered
_functionally_ has nothing to do with congestion control. The only
reason for using UDP encapsulation is to get through middleboxes. If
something else worked better for that, they would use it.

---

From: Scott Brim <[email-masked]>
Date: Tue, 21 Jan 2014 23:03:21 -0500

> If the upper layer does not respond to congestion, then perhaps it shouldn't be running over the open Internet (with or without a tunnel), unless the *total* bandwidth that could be used is inheren…

To paraphrase: if this problem exists in the new encapsulation, then
it exists already.

Lars is right, this does allow traffic that was formerly run over
provisioned paths in well-managed networks to possibly be part of
general Internet traffic. It would be good if there were a way to be
_sure_ there was e2e congestion control. But there is no signaling
between this low-layer UDP encapsulation and anything above it that
might already be reacting to congestion. There is no reasonably easy
way for it to know what it is carrying. Yes there is a way to do
congestion control at the bottom layer, but doing so could destroy
performance if one (or more) layer(s) is already doing it up above. We
have experience with that.

I can't remember who said it, but an applicability statement might
satisfy everyone, especially since it's been said (Curtis?) that this
just isn't going to be used in situations where congestion will be a
problem.

Alternatively, a paragraph laying out the problem and saying if this
is used in a way that could impact ordinary traffic, a mechanism must
be defined.

Scott

---

From: Spencer Dawkins <[email-masked]>
Date: Wed, 22 Jan 2014 08:48:06 -0600

For what it's worth ...

> just isn't going to be used in situations where congestion will be a

Alia Atlas made this suggestion.

Speaking as an AD who has been following this conversation with some 
interest ... that's gotta help.

> is used in a way that could impact ordinary traffic, a mechanism must

I would read such a paragraph with some interest ...

Spencer

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jan 2014 07:18:36 -0800

On 1/21/2014 6:40 PM, Ross Callon wrote:
> If the upper layers (the thing that runs over the tunnel) involves
> applications over TCP over IP, or if it is otherwise responding to
> congestion in the same way that we expect anything running over IP to
> respond to congestion, then we don't want the tunnel to also
> independently try to respond to congestion (two independent cooks
> cooking the same meal does not necessarily lead to success).

That's true, but is an issue only if the tunnel congestion control is 
dynamic over the same or shorter timescales than the 
congestion-controlled traffic being tunneled.

> If the upper layer does not respond to congestion, then perhaps it
> shouldn't be running over the open Internet (with or without a tunnel),
> unless the *total* bandwidth that could be used is inherently quite low.

Sure, but who would enforce that? If you're using MPLS, the only entity 
in a position to know (whether the traffic is congestion controlled or 
not) is the tunnel encapsulator.

> On the other hand, it might want to run within a data center or
> internally to a service provider network with appropriate provisioning.

All bets are always off for private networks; they need not comply with 
any Internet standards or expectations, even when they leverage Internet 
header formats and protocols.

But that's not a strict constraint of the proposed approach here.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jan 2014 07:33:29 -0800

On 1/21/2014 7:50 PM, Scott Brim wrote:
> function.

http://www.isi.edu/rna ought to clear that misconception up.

> It's a UDP encapsulation, but that encapsulation has nothing

That would be correct if the 8-byte UDP header were interpreted anywhere 
*except* the current Internet as the demuxing layer above IP. But that 
semantics is exactly what this document seeks - to use that UDP 
information for load balancing, e.g.

Consider this from the Internet's viewpoint; UDP traffic is traversing 
it, sourced by the tunnel ingress, and travels through the Internet with 
the congestion control expectations outlined in RFC5405.

To the Internet, this UDP header is a transport layer, and the tunnel 
ingress is the application.

Yes, if you were just using these bytes somewhere else, interpreted by 
some other mechanism (or not), they'd be just bytes of encapsulation. 
But in the Internet, the roles are determined by RFC768, RFC1122, and 
RFC5405.

> position.

In an arbitrary environment, without knowledge of any of the other 
protocol layers, yes (that's the principle behind RNA, above).

When instantiated inside the Internet, when you take a physical signal, 
interpret it as a link layer packet, examine the next 20-40 bytes as the 
IP header, and the next 8 as UDP, then that UDP header *is* the 
transport layer for that network (the Internet), and the application 
layer *is* whatever generated the contents of that packet (the 
encapsulator).

 > This instance of UDP is way below transport, is just in

That's true from the perspective of the MPLS packet and its origin 
network, but not from the perspective of the network the UDP 
encapsulated result traverses.

> _functionally_ has nothing to do with congestion control.

Oh, sure - UDP has nothing to do with congestion control. But that's 
exactly why the transport area generated RFC5405's section 3.1.3 - 
requiring that the application (the party that generates the UDP 
packets) includes congestion control (directly, or by transitive closure 
of the source of the source of the source... that generated the original 
highest layer of packet, which eventually became encapsulated in UDP).

 > The only

According to the doc, this is also for load distribution. That aside, 
however, it'd be nice if there were a layer that just "got through 
middleboxes" and had no other properties.

UDP isn't that layer. Nor is TCP.

TCP comes with its own baggage (including reactive congestion control) 
of a heavyweight mechanism that isn't desired for most high-speed tunnels.

UDP comes with RFC5405. That requires some sort of congestion reaction. 
No need to be on the same timescale as TCP - it could be much slower, or 
could simply be (as Lars suggested) a throttle-limit (circuit breaker).

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jan 2014 07:37:16 -0800

> have experience with that.

Lars isn't suggesting congestion control on the same timescale as TCP, 
which is where the problem would be.

Long-timescale control OR something as simple as a throttle-cap would be 
sufficient.

> just isn't going to be used in situations where congestion will be a

If we accepted that premise, we wouldn't need any congestion control. 
Nobody openly claims to want to generate congestion.

Congestion control is required to protect the rest of us from cases 
where that claim turns out to be false (typically because the protocol 
designers/use case authors turned out to be wrong).

> is used in a way that could impact ordinary traffic, a mechanism must

The problem is very clearly already stated in RFC5405. I can't see how 
this mechanism would ever be used if not for ordinary traffic.

So yes, a mechanism must be defined IMO too.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jan 2014 07:46:32 -0800

> of any new UDP applications and adhere to the BCP you wrote.

The BCP that the transport area as a whole recommends, in conjunction 
with the IETF as a whole.

> You appear to be very nearly alone in this argument...

The IETF is a very large DDOS attack on people's time. It's very easy 
for faulty ideas to pop up and not be challenged - easy because there 
are a lot of people (companies) with vested (financial) interest in 
promoting their particular approaches, and only a few people who care to 
spend the time playing "whack a mole" to defend the Internet against 
these approaches (by either pulling them back into compliance with 
agreed principles, such as BCPs, or shooting down ideas altogether).

So if your room is full of your own choir singing so loud for your own 
sake that you can't hear the objections of the few who are providing 
feedback from outside the room, that shouldn't be surprising. But that 
neither means the choir is in tune nor that there aren't many outside 
the room (or who don't attend meetings) who aren't trying to tell you 
otherwise.

FWIW.

:-)

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jan 2014 07:51:13 -0800

> The non-congestion controlled application is what needs fixing.

In all three cases, RFC5405 expects that traffic inside UDP is 
congestion controlled. That can happen when the source application does 
so, but when that isn't known, it needs to happen at whatever layer puts 
the packet inside the UDP header that the Internet ends up seeing.

> underneath the non-congestion controlled application

Yes, but it's not wrong to require some sort of congestion control at 
any layer that generates a UDP packet inside the Internet or any other 
Internet-protocol-based network that has the same RFC5405 expectations.

Joe

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 21 Jan 2014 15:58:38 -0500

Lars,

The IETF consensus in RFC5405 was to put a SHOULD in regarding use of
congestion control in tunneling protocols.

RFC2119 states:

  3. SHOULD  This word, or the adjective "RECOMMENDED", mean that there
     may exist valid reasons in particular circumstances to ignore a
     particular item, but the full implications must be understood and
     carefully weighed before choosing a different course.

We have discussed the reasons why congestion control is in general a
good thing.  We have also discussed why there may be valid reasons to
not include congestion control in MPLS over UDP.

The question is not whether there is already IETF consensus on
congestion control in UDP in general.  There is consensus and that
consensus was to use the word "SHOULD".

The question we now face is whether MPLS in UDP needs to up the prior
consensus to a MUST or can keep it as SHOULD.  I see one or maybe two
people vigorously arguing to upgrade this to a MUST and otherwise
consensus to keep this as SHOULD.  Further I see no objection except
the same one or maybe two people to making the congestion control the
topic of a later work if a need for it arises.  Consensus does not
require unanimous agreeement.

If we are trying to gauge consensus, maybe both you and I should sit
back and let other people weigh in.

Curtis

In message <[email-masked]>
"Eggert, Lars" writes:
> the reason we wrote the RFC when I was TSV AD was that we were seeing a > whole bunch of questionable uses of UDP over the eyars and we were > having the same arguments over and over. That's why we decided to write > down the practices we expect users of UDP to follow. This is yet another > such questionable use.
>  

Nothing personal intended.  

The important point is the sentence "You appear to be very nearly
alone in this argument and certainly no one that works with MPLS is
siding with you." was intended to point out that there is no consensus
behind your argument regardless of how vigorously you make that
argument.

> Aka the "take my toys and go home" argument. Heard it many times.

It has been successful many times in the past.  It turns into the "its
deployed so get over it" argument after a few years.

> And there's nothing wrong with that - the BCP even says that one SHOULD > NOT use congestion control for some deployment cases.
>  
> But for others, one SHOULD. For those, a mechanism needs to be > available, i.e., it needs to be specified and implemented.
>  
> I'm sorry, but we have certain architectural principles in the Internet > that we have IETF consensus on. At least since RFC2914, that includes > the need to have congestion control in place.
>  
> There are always special deployment scenarios where these principles do > not apply, and we typically explain in applicability statements when out > specifications can only be safely used under certain conditions. I don't > see any such statement in draft-ietf-mpls-in-udp, which to me means it's > targeted at general Internet-wide use.
>  
> The SHOULD for congestion control needs to actually describe a mechanism > that can be used when needed. It can't be a blanket "you SHOULD use > something but we don't tell you what it is"-statement.
>  
> It's not SHOULD vs. MUST. It's two SHOULDs, but in both cases it needs > to be specified what is to be done. In the case of checksums, that's > obvious (calculate it and check it); in the case of congestion control, > some actual mechanism needs to be described (e.g., a circuit breaker).
>  
> > IMHO The only remaining question is whether the document can go > forward
> I would like the document to specify at the very least a circuit breaker > mechanism, that stops the tunneled traffic if severe packet loss is > detected along the path.
>  
> And this isn't about an "individual veto". This is about a document that > is at the moment in violation of IETF consensus at least as far back as > RFC2914.
>  
> Lars

---

From: Martin Stiemerling <[email-masked]>
Date: Wed, 22 Jan 2014 16:56:53 +0100

Hi Curtis,

I want to make an important observation below:

> one that works with MPLS is siding with you.

Lars is not alone but took the time and energy to comment on the draft.

Lars' arguments are just right and need to be addressed.

Regards,

   Martin

Transport Area Director

---

From: Joe Touch <[email-masked]>
Date: Thu, 23 Jan 2014 13:22:24 -0800

Curtis,

If you're going with this argument (SHOULD vs MUST), then you need to 
explain exactly *why* you're not doing what's recommended. "SHOULD" 
isn't carte blanche to claim an exception.

And the need to avoid layered congestion control is a good justification 
for not layering TCP-friendly on top of TCP-friendly, but not for 
avoiding circuit breaker-style congestion controls/limits.

Joe

On 1/21/2014 12:58 PM, Curtis Villamizar wrote:
>> the reason we wrote the RFC when I was TSV AD was that we were seeing a =
>> whole bunch of questionable uses of UDP over the eyars and we were =
>> having the same arguments over and over. That's why we decided to write =
>> down the practices we expect users of UDP to follow. This is yet another =
>> such questionable use.
>>
>> And there's nothing wrong with that - the BCP even says that one SHOULD =
>> But for others, one SHOULD. For those, a mechanism needs to be =
>> available, i.e., it needs to be specified and implemented.
>>
>> I'm sorry, but we have certain architectural principles in the Internet =
>> that we have IETF consensus on. At least since RFC2914, that includes =
>> the need to have congestion control in place.
>>
>> There are always special deployment scenarios where these principles do =
>> not apply, and we typically explain in applicability statements when out =
>> specifications can only be safely used under certain conditions. I don't =
>> see any such statement in draft-ietf-mpls-in-udp, which to me means it's =
>> The SHOULD for congestion control needs to actually describe a mechanism =
>> that can be used when needed. It can't be a blanket "you SHOULD use =
>> It's not SHOULD vs. MUST. It's two SHOULDs, but in both cases it needs =
>> to be specified what is to be done. In the case of checksums, that's =
>> obvious (calculate it and check it); in the case of congestion control, =
>> some actual mechanism needs to be described (e.g., a circuit breaker).
>>
>>> IMHO The only remaining question is whether the document can go =
>> forward
>> I would like the document to specify at the very least a circuit breaker =
>> mechanism, that stops the tunneled traffic if severe packet loss is =
>> detected along the path.
>>
>> And this isn't about an "individual veto". This is about a document that =
>> is at the moment in violation of IETF consensus at least as far back as =
>> RFC2914.
>>
>> Lars

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 23 Jan 2014 22:20:54 -0500

> avoiding circuit breaker-style congestion controls/limits.

Joe,

I agree with you that the word SHOULD is not an automatic pass and
quoted from RFC 2119 because I think that on this thread the criteria
"valid reasons in particular circumstances" and "implications
... carefully weighed" have both been met.

Perhaps you joined the discussion a little late.

Please allow me to recap.  There are two issues under discussion, UDP
checksums and congestion control.

All parties seem to be OK with limiting the use of MPLS over UDP to
service providers only within their own infrastructure only, except
among cooperating service providers with explicit agreement
(consenting adults).

Within that context, some service providers find that parts of their
infrastructure either doesn't support MPLS at all or doesn't support a
usable variation of multipath for MPLS but does support IP ECMP (as I
understand it, this is primarily in the fringes, access and less so
edge - but someone who knows for sure please correct me if not).

This eliminates the "expands the reach of MPLS argument".

First UDP checksums:

  The UDP checksum is at the beginning of the payload.  Please see
  http://www.ietf.org/mail-archive/web/mpls/current/msg11279.html
  This makes filling in a new UDP checksum infeasible on most high end
  hardware.

  Within all of that context, all of the links have at least 32 bit
  FCS (either Ethernet framing or SONET or OTN) and some has much
  better such as the FEC found in OTN.  This makes undetected
  corruption by any of the links extremely unlikely.

  Corruption internal to the router the only potential impact.  There
  is good reason to think that this is extremely rare.

  Infeasibility of doing UDP checksums in some instances (not just
  performance concerns) is sufficient to make use of UDP checksums a
  SHOULD.

On to congestion control:

  Some MPLS payloads such as IP over MPLS have congestion control.
  While it can't be assured that all IP traffic makes use of
  congestion control, the problem is in the UDP services carried over
  MPLS regardless of whether MPLS over UDP is used.

  The only other type of traffic carried by MPLS in any volume (not
  counting ISIS control traffic carried as CLNP over MPLS, for
  example) is PW.

    PW in turn supports Ethernet, FR, ATM, TDM, and maybe a few
    others.  The vast majority of payload in Ethernet, FR, ATM, is IP
    so were are back to the prior bullet.  It PW packlet carrying
    Ethernet, FR, or ATM are dropped the congestion response is
    virtually identical to IP because the payload is IP.

    The other PW case is TDM.  TDM serves two purposes in SP networks.
    A small number of legacy customers circuits, generally T1 or T3
    use it.  These are 1.5 and 45 Mb/s services on 1 Gb/s and 10 Gb/s
    links.  Hardly a source of congestion, plus they pay a lot more
    and therefore are priority traffic.  Another is 3G mobile
    backhaul.  Again this is T1 or T3, but maybe as much as small
    multiples of T3.  Still very low capacity.

  TCP-like congestion control in a tunnel is very bad for any TCP
  carried within the tunnel.  The vast majority on MPLS traffic is IP
  and therefore is TCP.

  Regardless of the weak argument in favor of it given the operational
  reality, the authors have conceded to state that "MPLS over UDP
  SHOULD have congestion control" in the document.

  Because in the scenario it is intended to be deploy congestion
  control is very unlikely to prove to be needed, defining the exact
  form of congestion control is deferred to a later document which
  will be written only if needed.

That said, I have in the past argued that the "circuit breaker" form
of congestion control would be one of the worst things we could
possibly do.  It would be terrible for TCP and most anything else.
See http://www.ietf.org/mail-archive/web/mpls/current/msg11262.html
The right place to put any circuit breaker would be TDM over PW, but
given the capacities involved, that is not really needed either.

In the past I've said:

  The technical sticky point is knowing that the congestion is
  occuring.  Perhaps a LM like packet on another UDP port could be
  exchanged with the same caveats about inaccuracy if implemented in
  software and inaccuracy if reordering occurs, and a recommendation
  can be made to do something to introduce drop if congestion is
  indicated by this method.  How to introduce drop should be an
  implementation detail, a form of AQM or leaky bucket being possible
  choices, but entirely a local matter.

MPLS LM OAM would be a good choice for detection of congestion (as
evidenced by loss).  A leaky bucket (aka shaper) likely with AQM seems
like a reasonable choice.  This could be put into another document if
a need ever truly arises for congestion control in MPLS and it could
be applied to MPLS over anything (though it could prove impractical
for MPLS over avian carrier).

Curtis

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Jan 2014 11:15:12 -0800

On 1/23/2014 7:20 PM, Curtis Villamizar wrote:
...
>    This makes filling in a new UDP checksum infeasible on most high end

That argument would make sense if most hardware wasn't store-and-forward 
on a per-packet basis.

>    is good reason to think that this is extremely rare.

I don't buy the argument that per-hop checksums replace the need for 
end-to-end checksums; that's part of why more modern tunneling systems 
(e.g., SEAL) include their own head-to-tail checksum. So while I agree 
that UDP checksums may not be strong enough here, that might be an 
argument for a better tunneling mechanism than raw MPLS in IP, rather 
than ditching UDP checksums.

>    performance concerns) is sufficient to make use of UDP checksums a

Your argument hinges on feasibility, but that doesn't hold water (esp. 
given I've designed hardware that does Internet checksums - including 
some recent all-optical designs).

>    MPLS regardless of whether MPLS over UDP is used.

That is the key problem. As you point out below, other traffic is 
somewhat 'noise' on the overall equation.

>      multiples of T3.  Still very low capacity.

You can't know whether the PW cases you expect will change over time, or 
are accurate as per above. I would not be surprised if someone started 
(or is already) using PW to connect datacenters to exchange bulk data 
using non-CC protocols.

>    carried within the tunnel.  The vast majority on MPLS traffic is IP

Yes, and that's why nobody is asking for that.

>    SHOULD have congestion control" in the document.

OK (that matches RFC 5405).

>    control is very unlikely to prove to be needed,

That's pure speculation, and more importantly undermines the SHOULD 
you've just conceded.

> defining the exact

That document is needed now, exactly because of the SHOULD. You can't 
make a SHOULD recommendation without backing it up with the detail 
needed to implement it.

Other wise, it's a MAY, or (more to the point), a (nonexistent) MIGHT 
SOMEDAY.

> possibly do.  It would be terrible for TCP and most anything else.

Maybe, but if so, then you need the head-end to employ that breaker only 
on non-CC traffic.

> for MPLS over avian carrier).

See above. MIGHT SOMEDAY isn't a reasonable solution to a problem for 
which the need for a solution is already conceded.

Joe

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 27 Jan 2014 16:40:50 +0000

On 24/01/2014 19:15, Joe Touch wrote:
> That argument would make sense if most hardware wasn't 
> store-and-forward on a per-packet basis.
They may be store and forward, but most of the high end designs
use multiple grades of memory putting the packet in "slow memory"
and providing a snapshot of the header in "fast memory" to the
forwarder. Thus although the whole packet is in the system, it
it is not accessible to the engine that would need to calculate the
c/s.

Stewart

---

From: Joe Touch <[email-masked]>
Date: Mon, 27 Jan 2014 08:48:35 -0800

Those same mechanisms have provided hardware checksum support for a very long time. 

Joe

> On Jan 27, 2014, at 8:40 AM, Stewart Bryant <[email-masked]> wrote:
> 
>> That argument would make sense if most hardware wasn't store-and-forward on a per-packet basis.

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 27 Jan 2014 17:16:18 +0000

Sorry, are you talking the same h/w that does IP c/s?

That h/w can only see the IP header.

S

---

From: Joe Touch <[email-masked]>
Date: Mon, 27 Jan 2014 10:00:51 -0800

> Sorry, are you talking the same h/w that does IP c/s?

It's been trivial to implement this for TCP checksums for a long time. 
18 years ago I showed how to do it for what was then $40 at 1Gbps (RFC 
1936).

They're done in hardware in many network interface cards on end systems, 
and it's been necessary for routers that use TCP tunnels as well.

Splitting headers from packet body for high-throughput routing has been 
known and done for a long time too (e.g., S. Walton, A. Hutton, and J. 
Touch, “High-Speed Data Paths in Host-Based Routers,” IEEE Computer, 
Nov. 1998, pp. 46-52.); during body transfer, the contents can easily be 
summed, and that sum can be combined with the header sum easily too.

Joe

---

From: joel jaeggli <[email-masked]>
Date: Mon, 27 Jan 2014 10:48:43 -0800

> Those same mechanisms have provided hardware checksum support for a very long time. 

The new header and the payload are actually in different parts of the
forwarding complex until they hit the output queue, you can't checksum
data you don't have.

---

From: Joe Touch <[email-masked]>
Date: Mon, 27 Jan 2014 10:53:09 -0800

> forwarding complex until they hit the output queue, you can't checksum

You can (and some do) the checksum component parts when things go into 
memory; the partial sums can be added as the parts are combined in the 
output queue.

I appreciate that we're all taking about what might be done, but the 
reality is that there are many 'transparent TCP proxies' that have to do 
this, so there's clearly a solution, and it clearly runs fast enough.

Joe

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 27 Jan 2014 14:19:04 -0500

Yes Joe, routers could ahve been built to do those calcualtions at that 
performance scale.
There are however two major problems:

1) That is not how routers are built.
2) The target performance scale is rather higher.

So could someone build an ASIC to do what you want?  Probably.  Is there 
any reason in the world to expect operators to pay the significant extra 
cost for such?  Not that I can see.
And even if we could and they would, that is not the world into which we 
are deploying these tunnels.

Yours,
Joel

---

From: Joe Touch <[email-masked]>
Date: Mon, 27 Jan 2014 11:24:34 -0800

On 1/27/2014 11:19 AM, Joel M. Halpern wrote:
> So could someone build an ASIC to do what you want?

Has. It's already part of nearly every DMA ASIC in a network interface 
already.

 > Probably.  Is there
> cost for such?Not that I can see.

We're talking about a ring of full adders, the specs for which are given 
in an RFC that's 18 years old, and that is already implemented in nearly 
every host interface, including 10Gps NICs.

And we're talking about "routers", many variants of which operate at 
very high speeds and transparently proxy TCP already. So this is a 
solved problem.

> are deploying these tunnels.

We're back to "that's not what they do now", at least in some devices.

Well, they don't use MPLS in UDP (since no spec exists), so clearly if 
they're limited to doing what they already do, this is an exercise in 
futility.

Joe

---

From: <[email-masked]>
Date: Mon, 27 Jan 2014 23:17:24 +0000

...and UDP Lite just covering the L3 and L4 headers, but not the full payload a la UDP,
 is still possible when the full payload is not visible.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: ietf [[email-masked]] On Behalf Of Joe Touch [[email-masked]]
Sent: 27 January 2014 16:48
To: [email-masked]
Cc: [email-masked]; IETF discussion list; [email-masked]
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Joe

---

From: <[email-masked]>
Date: Mon, 27 Jan 2014 23:18:56 +0000

"ASICs can't do IPv6 in hardware! IPv6 can only be done with slow
software forwarding! We're stuck with v4!"

I think we've been here before.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: mpls [[email-masked]] On Behalf Of Joe Touch [[email-masked]]
Sent: 27 January 2014 19:24
To: Joel M. Halpern; joel jaeggli; [email-masked]
Cc: [email-masked]; IETF discussion list
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Joe

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 27 Jan 2014 20:00:45 -0500

In message <[email-masked]>
Joe Touch writes:
 
> reality is that there are many 'transparent TCP proxies' that have to
> do this, so there's clearly a solution, and it clearly runs fast
> enough.
>  
> Joe

Joe,

Chips that did 4 x 10 Gb/s are old stuff now but in their day they
pushed silicon limites.  Chips now do N x 100 Gb/s.  Stewart is
describing how these chips (both the prior generation and current) are
architected and you are going back to some idea that there is a common
memory where this all resides and its just a matter of looking at it.
This is not software on a general purpose processor.

If a queue forms, the headers are in SRAM and the body of the packet
is off chip in DRAM.  There isn't enough memory bandwidth to pull the
packet back until its time to send it out.  At that point the header
and body are joined but the header processing has been completed long
ago.

If there is not much of a queue the packet may go into SRAM cache for
the buffer DRAM and go right out of that SRAM.

If there is no queue at all cut-through can happen.  The header gets
transmitted before the entire packet arrives at the input of the
chip.  (And if FCS fails a runt with bad FCS goes out).

At the very least this can't work if cut-through is used to reduce
latency.  When the two are joined, about the only processing is in the
MAC, its mostly loading a shift register and serializing but it also
does the FCS and sticks that *at the end*.  The MAC is quite
inflexible as it is mostly silicon gates designed to do some minimal
processing for a layer-2 such as Ethernet or GFP/OTN.

Think for a moment what N x 100 Gb/s means.  For small packets 150
Mpps per interface with Ethernet overhead (large overhead).  That is
20 clock cycles per packet for a 3 GHz clock rate.  Divide that by N.
This has to be done in specialized hardware and if you also think
about the memory bandwidth needed, you need very wide and very fast
DRAM and get one write when the packet arrives and one read before it
leaves and even then have to play tricks in the SRAM cache like
concatonating short packets going out the same queue.

BTW - "transparent TCP proxies" don't need to look at the payload for
the purpose of updating a checksum.  Whenever a header modification is
made you only need to know the old checksum, the old information and
the new information replacing it.  That is why IP checksum
modification can be done quickly.  Typically only TTL changes.  These
are also running at least a decimal order of magnitude or two slower.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 27 Jan 2014 20:29:16 -0500

> Has. It's already part of nearly every DMA ASIC in a network interface

There is no DMA ASIC in these designs.  There is no shared memory to
DMA from.  A router is not a specialized PC.

At any one point in time the best DMA ASICs run at a fraction of the
speed of forwarding ASICs of that same era.  Some forwarding ASICs
through parallelism or pipelining (or both) can forward a packet in
under ten clock ticks of the ASIC.  [The best forwarding chips today
are technically not ASICs but custom silicon though they are still
often called forwarding ASICs.]

BTW - the PCIe FCS is at the end of the transmission not the front so
the DMA ASIC doesn't have to read memory twice, the first time to
compute a checksum to put at the front.

> We're talking about a ring of full adders, the specs for which are
> given in an RFC that's 18 years old, and that is already implemented
> in nearly every host interface, including 10Gps NICs.
>  

See prior email.  They don't need to look at the payload to modify IP
or TCP headers and then update the checksum.

> they're limited to doing what they already do, this is an exercise in

You seem to be missing the point that MPLS over UDP is not considered
a good solution going forward on which to base the design of new
hardware, but rather an interim solution to accomodate old hardware
that doesn't load split MPLS traffic.

In any case, two passes, one to compute a checksum and put it on the
front, would increase latency.  If anything UDP-Heavy with an FCS at
the end would be used, even though two FCS is considered bad form.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 27 Jan 2014 20:56:20 -0500

In message <[email-masked]>
[email-masked] writes:
 
> ...and UDP Lite just covering the L3 and L4 headers, but not the full
>  payload a la UDP, is still possible when the full payload is not
>  visible.
>  
> Lloyd Wood

... but doesn't solve the problem of ECMP using old routers so
UDP-Lite is a non-starter.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 27 Jan 2014 21:23:42 -0500

> http://about.me/lloydwood

At what point does this discussion no longer have any relevance at all
to draft-ietf-mpls-in-udp?

The draft is going to say SHOULD wrt UDP checksums.  If you have an
issue with the wording of the draft, please comment on the draft.

Curtis

---

From: Edward Crabbe <[email-masked]>
Date: Mon, 27 Jan 2014 16:25:51 -0800

I assume we're talking about the chksum issue here.  If this is the case,
please go fix IPv6 by adding a hop by hop checksum extension header and
getting it implemented by vendors.  ^_^

If by 'this' we mean stateful congestion control of multiple terabits of
tunnel traffic, then it is *most certainly* not a solved problem, and
definitely not something I'm interested in paying for.

cheers,
   -ed

On Mon, Jan 27, 2014 at 11:24 AM, Joe Touch <[email-masked]> wrote:

> And we're talking about "routers", many variants of which operate at very
> high speeds and transparently proxy TCP already. So this is a solved
> problem.
>
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Jan 2014 08:40:38 -0800

On Jan 27, 2014, at 4:25 PM, Edward Crabbe <[email-masked]> wrote:

> I assume we're talking about the chksum issue here.  If this is the case, please go fix IPv6 by adding a hop by hop checksum extension header and getting it implemented by vendors.  ^_^

Please don't use UDP as a network layer.

See how easy that works?

> If by 'this' we mean stateful congestion control of multiple terabits of tunnel traffic, then it is *most certainly* not a solved problem, and definitely not something I'm interested in paying for.  

Oh, nobody wants to pay for anything, including (and especially) following standards. Fortunately, the IETF isn't an extension of an industrial company's marketing department (or isn't yet, I should say).

Joe

> Has. It's already part of nearly every DMA ASIC in a network interface already.
> 
> 
> > Probably.  Is there
> We're talking about a ring of full adders, the specs for which are given in an RFC that's 18 years old, and that is already implemented in nearly every host interface, including 10Gps NICs.
> 
> And we're talking about "routers", many variants of which operate at very high speeds and transparently proxy TCP already. So this is a solved problem.
> 
> 
> Well, they don't use MPLS in UDP (since no spec exists), so clearly if they're limited to doing what they already do, this is an exercise in futility.
> 
> Joe
> 
> 
> 
> Yours,
> Joel
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Jan 2014 08:49:59 -0800

On Jan 27, 2014, at 5:00 PM, Curtis Villamizar <[email-masked]> wrote:

> pushed silicon limites.  Chips now do N x 100 Gb/s.

Yes, and it's 18 years later. 

> Stewart is

I never said software; I said that the checksum can be computed as the data moves (which is how we did it in 1996).

> and body are joined but the header processing has been completed long

At some point the two come back together, at which point the partial sums of the header and body can be combined and inserted in the header.

...
> latency.

Latency is a complex function of the bandwidth, propagation, coding, and processing delays, and there are very rare cases where cut-through reductions would impact E2E latency. 

In those cases, there are other good reasons not to use UDP encapsulation.

> When the two are joined, about the only processing is in the
> This has to be done in specialized hardware

The hardware required is trivial, and easy to include in the same transfer mechanism used to move data (as it already has been for many years in network interfaces).

> the purpose of updating a checksum.

That depends on whether the segments boundaries are preserved.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Jan 2014 08:57:00 -0800

> DMA from.  A router is not a specialized PC.

Agreed; the point was that it isn't an ASIC, but a very small function that can be included in any data transfer mechanism.

> compute a checksum to put at the front.

Right, and until you modify UDP to have a trailer checksum, or use a different encapsulation that has one (e.g., via an IPv6 option), then you need to deal with the fact that the UDP checksum would require a store-and-forward delay.

IMO, if you don't like that, then use another encapsulation.

> that doesn't load split MPLS traffic.

We live with interim solutions for decades.

I'd be OK with something that says that the UDP checksum SHOULD be used, but that legacy deployments MAY ignore it.

That solves what happens in the short term, but doesn't saddle us all with a poor solution if/when it persists.

> front, would increase latency.

Again, latency is a complex issue, and store-and-forward delays inside routers is irrelevant for all but specialty deployments (stock trading esp.), and in those cases the use of additional encapsulation should be avoided anyway.

Joe

>  If anything UDP-Heavy with an FCS at

---

From: Ross Callon <[email-masked]>
Date: Thu, 30 Jan 2014 18:27:09 +0000

+1 to what Joel said. It is clear that Joe's view of "fast enough" for a host interface is not the same as my customer's view of "fast enough" for a core router (nor should it be the same -- routers have to be faster than host interfaces). 

More important is the issue of what can we get vendors to implement, which is probably better worded as what can we get network operators to push vendors to implement. If you are talking about fundamentally changing the internal data path for packets within a router or changing the data path into and out of specific chips to carry significantly more information, then we are talking about at least hundreds of millions and probably billions of dollars of equipment that needs to be replaced. This needs a very compelling justification before it is actually going to happen. 

Ross

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Monday, January 27, 2014 2:19 PM
To: Joe Touch; EXT - [email-masked]; [email-masked]
Cc: [email-masked]; IETF discussion list
Subject: Re: [mpls] Last Call: <draft-ietf-mpls-in-udp-04.txt> (Encapsulating MPLS in UDP) to Proposed Standard

Yours,
Joel

---

From: Joe Touch <[email-masked]>
Date: Thu, 30 Jan 2014 11:28:55 -0800

I think we agree that if the router needs faster, then the router gets 
faster.

Where we disagree is whether the "bull in the china shop" gets to ignore 
the rules of the road to get what it wants.

I.e., I won't disagree on what router vendors want or what they're 
willing to support (though I will disagree on what hardware can do, 
based on experience).

I will disagree that routers get to run roughshod over the rest of us to 
do it.

Simple solution -- don't use UDP.

Joe

---

From: Scott Brim <[email-masked]>
Date: Thu, 30 Jan 2014 15:20:47 -0500

There are two topics that should be separate: checksums and congestion
control.

> I think we agree that if the router needs faster, then the router gets

That's in response to Ross, about checksums.

> the rules of the road to get what it wants.

congestion

> I.e., I won't disagree on what router vendors want or what they're willing
> to support (though I will disagree on what hardware can do, based on
> experience).
>

checksums

congestion.

Joe, do I understand correctly that you've conceded about checksums and now
you want to talk about congestion again?

swb

---

From: Joe Touch <[email-masked]>
Date: Thu, 30 Jan 2014 12:29:21 -0800

> There are two topics that should be separate: checksums and congestion

Agreed.

> On Thu, Jan 30, 2014 at 2:28 PM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     I think we agree that if the router needs faster, then the router
>     gets faster.
>

Also about the complaint that routers run to fast to run any sort of 
congestion algorithm, including circuit breakers (which is clearly false).

>     Where we disagree is whether the "bull in the china shop" gets to
>     ignore the rules of the road to get what it wants.
>
> congestion

Yes, in the sense that incorrect checksums should be checked at the 
egress only anyway, and thus ignoring them would only impact the network 
after the egress.

No, in the sense that anything that peeks inside a UDP header has the 
right to want to check whether that data is uncorrupted - that includes 
things like NATs.

>     willing to support (though I will disagree on what hardware can do,

Both, as per above.

>     I will disagree that routers get to run roughshod over the rest of
>     us to do it.
>

Both. If you want the benefits of UDP, pay the penalty. If you don't 
want the penalty, don't use UDP.

I don't buy the claims that there's a need here to eat cake and have it too.

> Joe, do I understand correctly that you've conceded about checksums and
> now you want to talk about congestion again?

I'm talking about the overall argument, which has circled back to "we 
need to do what vendors are willing to do", rather than "vendors need to 
play nice in the Internet that they're making money off of".

Joe