Subject: [tsvwg] gre-in-udp draft (was: RE: Milestones changed for tsvwg WG)

From: "Black, David" <[email-masked]>
Date: Wed, 8 Jan 2014 18:20:52 -0500

Lloyd,

<tsvwg WG co-chair hat OFF>

This is only at the draft adoption stage - the tsvwg WG gets to take a hard
look at the zero checksum conditions in the gre-in-udp-encap draft and
make sure that they're right (the WG could even remove the support for
zero checksums) - that's among the good reasons for adopting the draft
in tsvwg as opposed to elsewhere.   

Speaking of elsewhere, draft-ietf-mpls-in-udp is in IETF Last Call and has
language allowing zero UDP checksums - I might suggest that you (and anyone
else who cares about this topic) go read that draft and consider whether
making Last Call comments would be appropriate.

Thanks,
--David
</tsvwg WG co-chair hat OFF>

> -----Original Message-----
> From: tsvwg [mailto:[email-masked]] On Behalf Of [email-masked]
> Sent: Wednesday, January 08, 2014 3:58 PM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [tsvwg] Milestones changed for tsvwg WG
> 
> Zero UDP checksums are being selected for convenience, without an appreciation
> of the overall effects and  costs on other traffic. I see this is occurring in
> both tsvwg
> and lisp, and it's probably happening elsewhere:
> 
> From the recently adopted by tsvwg draft:
> http://tools.ietf.org/html/draft-yong-tsvwg-gre-in-udp-encap-02
>    To simplify packet processing at the tunnel egress, packets destined
>    to this assigned UDP destination port [TBD] SHOULD have their UDP
>    checksum and Sequence flags set to zero because the egress tunnel
>    only needs to identify this protocol.  Although IPv6 [RFC2460]
>    restricts the processing a packet with the UDP checksum of zero,
>    [RFC6935] and [RFC6936] relax this constraint to allow the zero UDP
>    checksum.
> 
> from http://tools.ietf.org/html/draft-ietf-lisp-introduction-03
>    The UDP checksum is zero because the inner packet usually already has
>    a end-end checksum, and the outer checksum adds no value.  [Saltzer]
> 
> (That's a misreading of Saltzer - the UDP checksum is also protecting against
> misdelivery and pseudoheader corruption, and 'usually' is not a good defence.
> For shame, Noel.)
> 
> After all, the best examples of end2end systems failure are with zero
> checksums
> at the endhosts.
> 
> The TCP and UDP checksums catch significant numbers of errors, e.g. Stone's
> work:
> http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigcomm2000-9-1.pdf
> and some of those errors will appear in the headers, where they will send
> the data to other ports and destinations.
> 
> The potential for polluting other ports and applications because
> there is no pseudoheader demultiplexing sanity check is there.
> 
> IPv6 leaving this check implemented  on a per-transport basis has opened the
> door,
> and rewriting that section of RFC2460 in RFC6935 has taken the door off its
> hinges.
> These RFCs break the minimal multiplexing pseudoheader sanity check that
> RFC2460
> offered; IPv6 is a mess in so many ways, but making it worse?
> 
> I think publishing RFC6935 and 6936 and letting in zero checksums again to
> IPv6
> was a mistake, frankly. (alas, I wasn't paying much attention at the time -
> moving
> countries etc.)
> 
> A strong recommendation that UDP-Lite covering headers offers minimal
> computation
> overhead on tunnelled packets while protecting against polluting other ports
> is one
> solution, while acknowledging deployment limitations. Section 2.4 of RFC696 is
> mostly there.
> But zero UDP checksums should always come with copious warnings on the effects
> not on the
> carried traffic, which can have its own payload checks, but on traffic sharing
> the network
> with traffic delivered with zero UDP checksums. Drafts relying on zero
> checksums should be discouraged, not adopted by workgroups.
> 
> It's a tragedy of the commons, but the I'm-alright-Jack engineers who want
> zero udp checksums for their traffic, and do protect against the effects of
> zero
> checksums on their own payloads, won't care about effects on other traffic.
> Hey, maybe this should be treated as an attack, which falls under perpass?
> That
> should get it attention...
> 
> tsvwg needs to give (or get) some careful input on the implications here.
> 
> Lloyd Wood
> http://about.me/lloydwood
> ________________________________________
> From: [email-masked] [[email-masked]]
> Sent: 08 January 2014 12:55
> To: Wood L  Dr (Electronic Eng)
> Cc: [email-masked]
> Subject: Re: [tsvwg] Milestones changed for tsvwg WG
> 
> Lloyd,
> 
> Here is a little context... which could help. The IETF agreed to update
> the UDP checksum behaviour for IPv6 in RFC 6935, but only subject to the
> applicability specified in RFC 6936.
> 
> One of the reasons why a simple encapsulation like this needs to be done
> in tsvwg is to minimise the end-to-end implications on other traffic.
> Sure, using a zero checksum has such implications, and my own first
> concern is that the new GRE-in-UDP work follows the applicability
> statement in RFC 6936. To me, it seems the authors are heading this way -
> but maybe more help is needed. It would be no bad thing to highlight the
> implications of using zero checksums on other traffic.
> 
> Gorry
> 
> > Am I the only one who finds putting zero checksums in proposed standards
> > to be a worrying
> > trend?
> >
> > Lloyd Wood
> > http://about.me/lloydwood
> > ________________________________________
> > From: tsvwg [[email-masked]] On Behalf Of IETF Secretariat
> > [[email-masked]]
> > Sent: 07 January 2014 20:20
> > To: [email-masked]
> > Subject: [tsvwg] Milestones changed for tsvwg WG
> >
> > Added milestone "Submit 'Specification of GRE in UDP encapsulation' to
> > IESG as a PS RFC", due December 2014.
> >
> > URL: http://datatracker.ietf.org/wg/tsvwg/charter/
> >
> 
>

---

From: <[email-masked]>
Date: Thu, 9 Jan 2014 02:59:28 +0000

Thanks, David.

Apart from the pollute-other-ports aspect that I've discussed in
other mails, MPLS itself is not self-checking (GRE is for header and
payload -  if its optional checksum is used.) Without checking,  corrupt
the MPLS label, send it to another tunnel. Without checking, corrupt the
UDP port, send the packet elsewhere and pollute another port and stream.

I see L. Yong and X. Xu, who are coauthors of 
draft-yong-tsvwg-gre-in-udp-encap,
are also authors of
draft-ietf-mpls-in-udp
which again plays fast and loose with UDP checksums.

This is clearly a widespread problem, and the reliability of the internet is
being undermined.

We can class UDP zero checksum use as an attacks on other streams and
services, if that helps. Reliability is an unfashionable topic, but an attack?
That must be mitigated immediately.

Because they specify zero UDP checksums, 
I oppose publication of draft-ietf-mpls-in-udp in its current form
I oppose tsvwg adoption of draft-yong-tsvwg-gre-in-udp-encap in its current form.
I oppose the IETF lisp documents.

regards
Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Black, David [[email-masked]]
Sent: 08 January 2014 23:20
To: Wood L  Dr (Electronic Eng); [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: gre-in-udp draft (was: RE: [tsvwg] Milestones changed for tsvwg WG)

Lloyd,

Thanks,
--David

> moving countries etc.)

---

From: <[email-masked]>
Date: Thu, 9 Jan 2014 08:06:57 +0000

Randy,

okay, let  tsvwg adopt draft-yong-tsvwg-gre-in-udp-encap, and let's get consensus on  it. And then the authors can adopt that consensus for mpls-in-udp, which overlaps in authorship...

thanks,

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Randy Bush [[email-masked]]
Sent: 09 January 2014 07:51
To: Wood L  Dr (Electronic Eng)
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE: [tsvwg] Milestones changed for tsvwg WG)

> I oppose the IETF lisp documents.

lloyd,

i think i understand your position.  but i disagree with preventing wg
adoption of draft-yong-tsvwg-gre-in-udp-encap, mainly because i strongly
see wg adoption as how we get to discuss and work on a document, not as
approval of the document.  as david said, i think we need to discuss it
so we can decide if it should be fixed.  to do so, we have to adopt it.

randy

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 9 Jan 2014 10:21:44 -0000

Lloyd and Randy,

With respect to draft-ietf-mpls-in-udp, this is why we have IETF last calls, so
thanks for the comments.

We did take the precaution of sending this I-D for an early TSV Directorate
review because of the concern about a number of factors and the overlap with
tsvwg work, but the review came back "clean". Of course, such a review is just
one person, so this conversation is good.

Wrt zero checksum, where do you stand on nested checksums? There is some claim
that they represent a waste of processing. I am not convinced by that when each
layer is using dedicated hardware (that can presumably process checksums at line
speed), but I am interested in the consequences for cheap hardware and for
software implementations (as have been claimed to be some of the motivations for
this work).

Other TSV-related issues that surely pop up are:
- allocation of ports for foo-in-UDP
- congestion control

Please note that there are a number of I-Ds that you missed in your broad sweep
of "I am opposed". You should probably look at the NVGRE and VXLAN work (which I
think is lurking around the NVO3 working group) because that is also looking at
UDP encaps of a tunnelling protocol.

Thanks,
Adrian

Health warnings:
I am responsible AD for draft-ietf-mpls-in-udp
I am a co-author of the gre-in-udp draft.

> From: mpls [mailto:[email-masked]] On Behalf Of [email-masked]
> Sent: 09 January 2014 08:07
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [mpls] draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE:
> [tsvwg] Milestones changed for tsvwg WG)
> 
> Randy,
> 
> okay, let  tsvwg adopt draft-yong-tsvwg-gre-in-udp-encap, and let's get
> consensus on  it. And then the authors can adopt that consensus for
mpls-in-udp,
> which overlaps in authorship...
> 
> thanks,
> 
> Lloyd Wood
> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> _______________________________________________
> mpls mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mpls

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 09 Jan 2014 10:59:08 +0000

On 09/01/2014 02:59, [email-masked] wrote:
> MPLS itself is not self-checking (GRE is for header and
> the MPLS label, send it to another tunnel.
Lloyd

I would be interested in what operators are observing in their
network, because I have never heard a concern expressed that this
was happening.

- Stewart

---

From: Randy Bush <[email-masked]>
Date: Thu, 09 Jan 2014 16:51:33 +0900

lloyd,

randy

---

From: Mark Tinka <[email-masked]>
Date: Thu, 9 Jan 2014 13:03:52 +0200

On Thursday, January 09, 2014 12:59:08 PM Stewart Bryant 
wrote:

> I would be interested in what operators are observing in
> their network, because I have never heard a concern
> expressed that this was happening.

You mean for native MPLS, or for encapsulated MPLS?

Mark.

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 09 Jan 2014 12:08:43 +0000

> You mean for native MPLS, or for encapsulated MPLS?
Hi Mark,

Either or both.

I am interested in how often in practice MPLS packets get
misdelivered due to label corruption.

- Stewart

---

From: Loa Andersson <[email-masked]>
Date: Fri, 10 Jan 2014 00:16:59 +0800

Changed the subject line !

On 2014-01-09 20:36, Mark Tinka wrote:
> On Thursday, January 09, 2014 02:08:43 PM Stewart Bryant
> wrote:
>
>> Either or both.
>
> I can only speak to native MPLS, as I've never run tunneled
> MPLS.
>
> Well, first of all, routers would need to report corrupted
> MPLS frames so operators can glean this data. This isn't
> something I've come across, but it would be good to find
> some kind of way to count this across interfaces, if the
> routers can detect and report them.

This would be possible to do with MPLS-TP OAM, wouldn't it?

/Loa
>
> The known issue about mis-delivery of MPLS frames is poorly-
> sized MTU interfaces. I have no empirical data as to how
> this can corrupt successive MPLS frames that may fit into
> the transit MTU. But in this case, as with any Layer 2
> traffic, not enough MTU = dropped frame.
>
> Mark.
>

-- 

Loa Andersson                        email: [email-masked]
Senior MPLS Expert                          [email-masked]
Huawei Technologies (consultant)     phone: +46 739 81 21 64

---

From: Gregory Mirsky <[email-masked]>
Date: Thu, 9 Jan 2014 17:30:27 +0000

Hi Loa, et al.,
I think that you're referring to Connectivity Verification in MPLS-TP (RFC 6428). It would only detect mis-connection but would not count leaked in frames.

Such problem may be more apparent in Segment Routing (SPRING WG). Perhaps CV should be a requirement in SR OAM.

	Regards,
		Greg

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Loa Andersson
Sent: Thursday, January 09, 2014 8:17 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: [mpls] misdelivered mpls packets - Was: Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft

On 2014-01-09 20:36, Mark Tinka wrote:
> I can only speak to native MPLS, as I've never run tunneled MPLS.
>
> Well, first of all, routers would need to report corrupted MPLS frames 
> so operators can glean this data. This isn't something I've come 
> across, but it would be good to find some kind of way to count this 
> across interfaces, if the routers can detect and report them.

/Loa
>
> The known issue about mis-delivery of MPLS frames is poorly- sized MTU 
> interfaces. I have no empirical data as to how this can corrupt 
> successive MPLS frames that may fit into the transit MTU. But in this 
> case, as with any Layer 2 traffic, not enough MTU = dropped frame.
>
> Mark.
>

--

---

From: Mark Tinka <[email-masked]>
Date: Thu, 9 Jan 2014 14:36:56 +0200

Mark.

---

From: "Andrew G. Malis" <[email-masked]>
Date: Thu, 9 Jan 2014 15:12:16 -0500

I suspect the only way that we can count "misdelivered" MPLS packets
is by looking for missing packets at the receiver, such as by using
the sequence number in the PW control word, TCP, or UDP header,
depending on the MPLS LSP payload.

Thinking about the consequences of an MPLS label somehow becoming
corrupted in transit, nothing will happen until the corrupted label
reaches the top of the stack. At that time, either the packet will be
dropped because the corrupted label isn't in the local LIB, or it is
in the database and the packet will be most likely forwarded out of
the wrong interface, to be dropped at the next label pop because the
label now at the top of the stack is most likely not in that router's
LIB.

Eventually, the packet will most likely be dropped in the network, or
there is a small but nonzero probability that it will be delivered to
an incorrect non-MPLS external interface after the last label has been
popped. If the payload was a globally routable IP packet (not in an IP
VPN) and PHP is in use, then the IP forwarding lookup in the LER may
get the packet to the right place (which will most probably be not on
that particular LER).

Even if PHP isn't in use, if the external interface happens to be to
another router in the same IP address space, then again IP forwarding
may get the packet to the right place.

Of course, it could be a PW control word that gets corrupted, in which
case the packets would reach the PW termination point, but would fail
the sequence number check if the sequence number is in use and that's
the part of the CW that was corrupted.

I don't think anything really bad happens if an entropy label is corrupted.

An interesting thought experiment.

Cheers,
Andy

---

From: <[email-masked]>
Date: Fri, 10 Jan 2014 04:19:24 +0000

The origin of this discussion is zero UDP checksums and
http://www.ietf.org/mail-archive/web/ietf/current/msg85354.html

I suggest reading Jonathan Stone's papers and PhD thesis for a good
understanding of where errors can come from.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: mpls [[email-masked]] On Behalf Of Curtis Villamizar [[email-masked]]
Sent: 10 January 2014 02:17
To: David Allan I
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] misdelivered mpls packets - Was: Re:        draft-ietf-mpls-in-udp was RE: gre-in-udp draft

Since we are all top posting ...

For counting dropped packets LM OAM is what you need.  CV doesn't do
that.  Direct LM will give best results on a per LSP basis but
requires forwarding chip support to do so.

I also don't know the origin of this discussion.

>> misdelivered due to label corruption.

Not very often if ever at least for major vendor products.  I don't
know if second tier or third tier players have bugs.

(Very often circa 2000 or earlier particularly with one specific
vendor and with one provider who made things much worse by mucking
with the ILM through management plane, but that is ancient history).

Bit rot in TCAM or SRAM is very rare.  Bit rot in circa 1980s DRAM was
a problem.  Bit rot in modern non-ECC DRAM is rare.  Bit rot in modern
ECC DRAM is very rare.  Therefore to have a bad ILM you need bugs.

Curtis

In message <[email-masked]>
David Allan I writes:

Hi  (trimmed received list)

I'm not sure what the starting point of this dialog is, but I can at least say Greg is right on the CV front, it only detects and does not measure misdirection.

The only other possibility to count anything is a "no ILM" condition where the label value is unrecognized by the receiving LSR, which IMO could not be made authoritative if it is label values in either the frame or the NHLFE and not exclusively next hops that is corrupted.

Cheers
D

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Gregory Mirsky
Sent: Thursday, January 09, 2014 9:30 AM
To: Loa Andersson; [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] misdelivered mpls packets - Was: Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft

Hi Loa, et al.,

        Regards,
                Greg

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Loa Andersson
Sent: Thursday, January 09, 2014 8:17 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: [mpls] misdelivered mpls packets - Was: Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft

/Loa

--

---

From: <[email-masked]>
Date: Sun, 12 Jan 2014 02:59:41 +0000

On nested checksums, the question is how they are nested; it's a matter
of scope. With a bunch of checksums checking only a payload and any
inner checksums like Russian Matryoshka dolls, the end-to-end argument
tells us that for reliable receipt of the payload, only the innermost checksum
matters.

But here, we are not solely checking the payload, but information on how to
deliver and identify that payload - and while an outer Ethernet CRC is across
the last link, the UDP checksum, though weak, provides a check on the IP
addresses and UDP ports (via the  pseudoheader check) and MPLS stack
from UDP/IP source to UDP/IP destination (and the payload, which is the bit
everyone focuses on as the performance hit as redundant and a processing
cost when the payload has its own check, and the bit that UDP-Lite can leave out).

Nothing else checks that scope. The scope is wider, and affects the network
as a whole. Errors in these unchecked fields lead to misdirection and lead to
misdelivery. Or pollution of other ports.

The MPLS assumption is that it's protected and checked by a strong link CRC like
Ethernet, and checked/regenerated by stack processing between hops; here,
in a path context, with zero UDP checksums MPLS has no checking at all.

"consequences for cheap hardware and for software implementations"

I'm sorry, when was MPLS cheap?

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Adrian Farrel [[email-masked]]
Sent: 09 January 2014 10:21
To: Wood L  Dr (Electronic Eng); [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: RE: [mpls] draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE: [tsvwg] Milestones changed for tsvwg WG)

Lloyd and Randy,

Thanks,
Adrian

Health warnings:

mpls-in-udp,

---

From: <[email-masked]>
Date: Sun, 12 Jan 2014 21:22:58 +0000

Curtis

I suggest reading Stone's work, particularly
''When The CRC and TCP Checksum Disagree'
for discussion of corruption.

Particularly its conclusions: 'In the internet, that means
we are sending large volumes of incorrect data without
anyone noticing'.

The Layer-2 check is per link, not end-to-end. That matters.

The MPLS assumption is that it crosses a link with a frame
checksum. Putting MPLS over UDP breaks that assumption.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Curtis Villamizar [[email-masked]]
Sent: 12 January 2014 18:09
To: Wood L  Dr (Electronic Eng)
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE: [tsvwg] Milestones changed for tsvwg WG)

> in a path context, with zero UDP checksums MPLS has no checking at all.

That UDP would be running over IP over Ethernet or POS or GFP or ...

There is no layer-2 currently in use that does not have a robust FCS,
generally a 32 FCS and therefore the MPLS assumption of checking at a
lower layer is still valid.

Curtis

---

From: <[email-masked]>
Date: Sun, 12 Jan 2014 21:32:28 +0000

> Right, which is probably why routers today can count badly
> checksum'ed Ethernet frames, but don't have the equivalent
> for MPLS.

If Ethernet frames keep failing the check, you know you
have a local problem that needs fixing. That's why it's
instrumented.

Do any routers count TCP/UDP checksum failures, much less
expose the count via SNMP?

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Mark Tinka [[email-masked]]
Sent: 12 January 2014 12:26
To: [email-masked]
Cc: Wood L  Dr (Electronic Eng); [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE: [tsvwg] Milestones changed for tsvwg WG)

On Sunday, January 12, 2014 04:59:41 AM [email-masked]
wrote:

> The MPLS assumption is that it's protected and checked by
> a strong link CRC like Ethernet, and checked/regenerated
> by stack processing between hops; here, in a path
> context, with zero UDP checksums MPLS has no checking at
> all.

Right, which is probably why routers today can count badly
checksum'ed Ethernet frames, but don't have the equivalent
for MPLS.

> I'm sorry, when was MPLS cheap?

Current-generation ASIC's have no problem forwarding MPLS
frames at wire rate. One could go so far as to say that MPLS
has allowed vendors to make cheaper line cards also because
IP FIB's and traffic queues can be scaled down dramatically
(not that I'd every buy such line cards, but...).

Mark.

---

From: <[email-masked]>
Date: Mon, 13 Jan 2014 00:45:52 +0000

Really, you'd want to expose the pseudoheader check at endhosts; a well-instrumented Linux box could tell you a lot
about checksum failures.

But in this case, a router would be decapping UDP/MPLS tunnels as an endpoint, so could report on checksum failures -
if the checksum wasn't zero.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Dino Farinacci [[email-masked]]
Sent: 12 January 2014 21:37
To: Wood L  Dr (Electronic Eng)
Cc: <[email-masked]>; <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [lisp] [mpls] draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE: [tsvwg] Milestones changed for tsvwg WG)

> expose the count via SNMP?

Typically they do but only for packets destined to them. Much like hosts would check the header checksum.

Dino

---

From: [email-masked]
Date: Mon, 13 Jan 2014 20:31:43 -0000

>
> One of the reasons that IPv6 (rfc2460) dropped the header checksum
> that was in IPv4 is that with link layers all doing FCS it served no
> purpose.
>  For the same reason TCP and UDP checksums server no purpose.
>
That's not what I understood, it is needed for endpoint delivery
verification. But in this case, TCP & UDP checksums incorporate the IP
address info in the pseudo header, and this was thought sufficient.

> The link layer checksum fails and the packet is dropped.  Since this
> is usually not at the last hop (often a local Ethernet) but instead in
> a WAN link, the packet never arrives at the destination for anything
> to count IP header or TCP or UDP checksum errors.
>
Again, that presumes that everything above the link works flawlessly,
which isn't always the case.

> Also all of the following handle both data integrity and congestion
> avoidance the same way:
>
>   UDP over IP over Ethernet
>   UDP over IP over MPLS over Ethernet
>   UDP over IP over MPLS over UDP over IP over Ethernet
>   s/Ethernet/{POS,GFP,etc}/ for all of the above
>   s/^UDP over IP/PW/ for all of the above
>
> In all of the above cases data integrity is handled by the link layer.
> In all of the above cases congestion avoidance is handled by the
> application (or not at all).  Whether MPLS is carried directly over a
> link layer or over UDP/IP over a link layer makes no difference.
>
RFC 6936 section 3.1 says more about what happens when packets happen to
be mis-delivered to the wrong host or socket.

Gorry

> In message
> <[email-masked]>
>> Really, you'd want to expose the pseudoheader check at endhosts; a
>> well-instrumented Linux box could tell you a lot
>> about checksum failures.
>>
>> But in this case, a router would be decapping UDP/MPLS tunnels as an
>> endpoint, so could report on checksum failures -
>> Cc: <[email-masked]>; <[email-masked]>; [email-masked];
>> [email-masked]; [email-masked]; [email-masked]; [email-masked];
>> [email-masked]; [email-masked]
>> Subject: Re: [lisp] [mpls] draft-ietf-mpls-in-udp was RE: gre-in-udp
>> draft (was: RE: [tsvwg] Milestones changed for tsvwg WG)
>>

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 14 Jan 2014 14:17:46 +0000

Lloyd

I have just read the Stone paper and I have some significant
concerns about its validity with modern h/w. Certainly it
is hard to credit the notion that the  error rate
is in the range 1:1000 to 1:32000 as reported by the authors.

The paper was written in 2000 with hardware that would have
have been designed in the mid 1990s. In that era, h/w was
far more marginal, with performance traded against signal
integrity, and indeed a lot less signal integrity measurement
and simulation took place at both board and chip level. This
was also the era where metastability was just beginning to
become widely understood, and its lack of understanding
would be a possible source of DMA errors.

I therefore do not think we should place much reliance
on this paper, but should instead look at the rather more
modern statistics.

Such statistics ought to be readily available by looking at
the tcp/udp c/s error stats in hosts and routers. As a tiny
and perhaps erroneous sample I looked at three Macs
in the office here and the tcp c/s error stats were
313/29144518, 0/3000000, 0/5000000. Only one of those
three systems got within a factor of 3 of the lowest error
rate reported by Stone.

Bottom line, it seems that we could use more recent data
and then an understanding of how important these low
background error rates are in the tunneling application that
we are considering here.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 14 Jan 2014 18:26:48 +0000

I agree the paper is now obsolete.

Stewart

On 14/01/2014 17:06, Curtis Villamizar wrote:
> Lloyd,
>
> Maybe you should reread the paper too before citing it as evidence.
> Check the date on it.  Check the cited causes of errors.
>
> Packet traces from 1998 and 1999 are prehaps not so relevante today,
> particularly wrt error rates due to hosts, router memories, and link
> error rates.  Look at the cited caused of errors and realize that many
> things have changed.
>
> The largest single error in the paper (paerhaps as high as 99.9% of
> the errors) was the ACK-of-FIN bug in Windows NT.  They may have fixed
> that by now.  Other large causes were host hardware and host software
> bugs (sent bad data in the first place).
>
> One fifth of campus errors were caused by two hosts.  This is cited as
> a bug in Mac OS on Powermac 8100, fixed at the time of publication.
>
> A lot of host software errors are discussed, where the checksums are
> sent bad and therefore will pass any router link FCS.
>
> Sending host hardware errors were thought to be in large part caused
> by no data integrity check on host DMA transfers.  I think progress
> has been made on that front.  You can check PCIe.  It has a 32bit CRC
> per lane in the link layer and retransmits on error.
>
> Router memory errors was thought to play a large role in the non-host
> part of the error rate in the paper.  Routers use ECC now.  Going that
> far back they didn't always have parity RAM (and sometimes ran parity
> disabled to avoid parity error reloads).
>
> VJHC software errors?  Anyone still use VJHC?  Those errors should be
> gone.
>
> IP over HDLC over T1 missed a few errors at the link layer in those
> days.  That could be true and occurances dependent on the providers.
> In the paper that was thought to be a very small contributor.
>
> Both HDLC and PPP had an option for 16 bit or 32 bit FCS.  Often 16
> bit was used.  Some HDLC equipment could be and was configured to
> count errors and send the packets on their way on the assumption that
> it was better to count errors and deliver bad packets rather than
> deliver no packet.  Perhaps also to hide packet loss.  Today all of
> the link layers in use have 32 bit FCS and count and toss errored
> packets.  In most equipment all of the memories have ECC and all of
> the buses ECC or FCS if serialized.
>
> It is now 14 years since that paper was published in Signcomm and 16
> years since some of the observations.  Things have changed.  Nice bit
> of nostangia but that paper may no longer be relevant.
>
> Curtis
>
> reference -
> In message <[email-masked]>
> .

-- 
For corporate legal information go to:

http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: Huub van Helvoort <[email-masked]>
Date: Thu, 09 Jan 2014 18:58:48 +0100

Hello Greg,

You wrote:

> I think that you're referring to Connectivity Verification in
 > MPLS-TP (RFC 6428). It would only detect mis-connection
 > but would not count leaked in frames.

Indeed.
However, there is the Loss Measurement tool which can be used
to count dropped and mis-delivered packets.

Note that the CV tool in G.8113.1 includes packet loss counting.

> Such problem may be more apparent in Segment Routing (SPRING WG).
 > Perhaps CV should be a requirement in SR OAM.

Indeed.

Regards, Huub.

-- 
*****************************************************************
               请记住，你是独一无二的，就像其他每一个人一样

---

From: Mark Tinka <[email-masked]>
Date: Thu, 9 Jan 2014 20:48:15 +0200

On Thursday, January 09, 2014 06:16:59 PM Loa Andersson 

> This would be possible to do with MPLS-TP OAM, wouldn't

Sorry, no experience. 

I don't personally like MPLS-TP :-), but happy to hear of 
any experiences from those that do.

Mark.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 09 Jan 2014 21:17:57 -0500

Curtis

Cheers
D

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Gregory Mirsky
Sent: Thursday, January 09, 2014 9:30 AM
To: Loa Andersson; [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] misdelivered mpls packets - Was: Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft

Hi Loa, et al.,

	Regards,
		Greg

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Loa Andersson
Sent: Thursday, January 09, 2014 8:17 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: [mpls] misdelivered mpls packets - Was: Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft

/Loa

--

---

From: Randy Bush <[email-masked]>
Date: Sat, 11 Jan 2014 06:32:22 +0900

what may scare me most about this discussion is the lack of p in mpls
vns.  you just sent my very sensitive unencrypted packet to some very
bad place.

---

From: Mark Tinka <[email-masked]>
Date: Sun, 12 Jan 2014 14:26:23 +0200

Mark.

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 12 Jan 2014 13:09:01 -0500

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 12 Jan 2014 13:37:16 -0500

> (not that I'd every buy such line cards, but...).

Perhaps if you actully worked for a company that made line cards you
would not make the above statement.

Many of the big router vendors use the same TCAM hardware for MPLS
lookups as they do for IP lookups.  Doing the lookup is not the rate
limiting factor even in hardware that has an ILM SRAM table and some
form of radix based lookup.  All of the hardwre I've seen in the last
15 years or so forwards MPLS and IP at the same rate.  ... Except IPv6
before they decided to waste the lower half of the address space with
40 quintilion host in a bridged subnet and you really did have to look
at the whole 128 bits.  Back when forwarding was done in software
(circa 1995) your statement would have been true if MPLS preceeded
forwarding ASICs which it did not.

Also the statement "Current-generation ASIC's have no problem
forwarding MPLS frames at wire rate" is not true for most (or all)
hardware with 40 byte payloads (even with plus 4 with TCP SACK plus 4
if MPLS) and 100 Gb/s interfaces.  It is true for "average packet
size" traffic and on most hardware only true if bursts of 40 byte
packets are very limited in duration.

Curtis

---

From: Mark Tinka <[email-masked]>
Date: Sun, 12 Jan 2014 21:04:40 +0200

On Sunday, January 12, 2014 08:37:16 PM Curtis Villamizar 
wrote:

> Perhaps if you actully worked for a company that made
> line cards you would not make the above statement.

I work for an operator that pays real money to deploy and 
use those line cards, and I make it my business to know what 
I'm paying for.

> Many of the big router vendors use the same TCAM hardware
> for MPLS lookups as they do for IP lookups.  Doing the
> lookup is not the rate limiting factor even in hardware
> that has an ILM SRAM table and some form of radix based
> lookup.  All of the hardwre I've seen in the last 15
> years or so forwards MPLS and IP at the same rate.

Which was exactly my point - unless you somehow missed that.

> ...
> Except IPv6 before they decided to waste the lower half
> of the address space with 40 quintilion host in a
> bridged subnet and you really did have to look at the
> whole 128 bits.

It is no secret that forwarding of IPv6 packets on some 
vendor equipment is about half the rate of IPv4 or MPLS. But 
then again, because MPLS control planes are IPv4-driven 
today, I'm hoping I didn't have to be explicit about not 
including IPv6 in that group, on this list.

> Back when forwarding was done in
> software (circa 1995) your statement would have been
> true if MPLS preceeded forwarding ASICs which it did
> not.

You might not know that line cards from C like the LSP 
(Label Switch Processor) and much of the PFE from J's PTX 
are forwarding engines that have been made cheaper by 
reducing the IP FIB on the assumption that all traffic will 
be carried in MPLS. This is, obviously, an assumption I do 
not support because:

	- I run IPv6 natively, and at the moment, there are
	  no production-grade IPv6 control planes for MPLS.

	- It assumes providers will run 6PE (in which case,
	  IPv6 traffic enjoys MPLS and IPv6 wire rate
	  forwarding speeds).

I do not buy such line cards because for anyone running 
native IPv6, you could potentially run out of FIB slots to 
host IPv6 routes.

That is why I say MPLS has allowed vendors to put out 
cheaper line cards compared to the costs of those which have 
large IPv4/IPv6 scale. Because MPLS forwarding is so 
mainstream, there is no additional cost associated with 
forwarding rates similar to IPv4. So much of the cost of 
line cards is in QoS queues and routing FIB slots (and MPLS-
biased line cards are stripped of that, hence making them 
cheaper).

> Also the statement "Current-generation ASIC's have no
> problem forwarding MPLS frames at wire rate" is not true
> for most (or all) hardware with 40 byte payloads (even
> with plus 4 with TCP SACK plus 4 if MPLS) and 100 Gb/s
> interfaces.  It is true for "average packet size"
> traffic and on most hardware only true if bursts of 40
> byte packets are very limited in duration.

C'mon, Curtis. Everybody knows this already. Moreover, real 
world IP traffic is not all 40 bytes.

If operators have doubts about what forwarding engines can 
do below 128 bytes, that is what PoC labs are for before you 
buy. And even then, several operators accept the 
restrictions up to a certain point, because the lab and real 
life vary significantly.

Mark.

---

From: Dino Farinacci <[email-masked]>
Date: Sun, 12 Jan 2014 13:37:04 -0800

Dino

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 13 Jan 2014 14:11:12 -0500

One of the reasons that IPv6 (rfc2460) dropped the header checksum
that was in IPv4 is that with link layers all doing FCS it served no
purpose.  For the same reason TCP and UDP checksums server no purpose.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 10:50:45 -0500

In message <[email-masked]>
Mark Tinka writes:
 
> On Sunday, January 12, 2014 08:37:16 PM Curtis Villamizar=20
> wrote:
>  
> I work for an operator that pays real money to deploy and=20
> use those line cards, and I make it my business to know what=20
> I'm paying for.
>  

What you said was "Current-generation ASIC's have no problem
forwarding MPLS frames at wire rate".

Since you didn't mention IP packets I assumed you were making the 20
year old argument that MPLS forwarding was faster than IP.

> It is no secret that forwarding of IPv6 packets on some=20
> vendor equipment is about half the rate of IPv4 or MPLS. But=20
> then again, because MPLS control planes are IPv4-driven=20
> today, I'm hoping I didn't have to be explicit about not=20

None of the equipment I have worked on (a while back) or the chips we
evaluated recently had such a problem for IPv6 using only the top 64
bits for forwarding.  An argument was made in the early 2000s (by me
and others) that if we allocated global routes only from the top 64
bits all equipment at that time could forward at essentially the same
rate as for IPv4.  The reaction from the IPv6 community was to
completely throw away the bottom half of the address and make it the
host part.

There are chips for which looking at a 32 bit prefix or a 64 bit
prefix is done within the same pipeline and therefore takes the same
amount of time.  Other chips which parallelize packet handling budget
enough microengines to get the job done (only a part of which is the
MPLS ILM or IP destination lookup).

> You might not know that line cards from C like the LSP=20
> (Label Switch Processor) and much of the PFE from J's PTX=20
> are forwarding engines that have been made cheaper by=20
> reducing the IP FIB on the assumption that all traffic will=20
> be carried in MPLS. This is, obviously, an assumption I do=20
> not support because:

Those line cards (and designs I was recently involved in) assume what
has historically been called a BGP-free core.  Only the IGP routes are
carried in the core.  BGP is mapped onto the IGP routes.  MPLS carries
traffic across the core.  The core has no BGP routes and therefore
needs a lot fewer entries and the ILM can be put into SRAM rather than
use a TCAM and only a small set of IP routes needs to be supported.

That is done to eliminate a need for large external DRAM or TCAM which
itself and the SERDES needed to go off chip produces heat and requires
power, board space, and cooling and therefore reduces density.
Putting more simple filters in the core also helps increase density.
It is more a power to operate and floor space per Gb/s issue.

For the most part IP/MPLS transport equipment seems to be going this
way, though some people are stuck on MPLS-TP and trying to build a L2
overlay.

If you go back to before 2000 at least one provider built a BGP-free
core over Frame Relay, but FR itself soon died and the overlay didn't
scale well.  Some providers wanted to do this over MPLS but the
RSVP-TE restoration times were too long and the fallback to IP routing
was still needed to make up for that.

> I do not buy such line cards because for anyone running=20
> native IPv6, you could potentially run out of FIB slots to=20
> host IPv6 routes.

Unless you run a BGP-free core, whether IPv4 or IPv6.  Then you have
plenty of FIB space.

> That is why I say MPLS has allowed vendors to put out=20
> cheaper line cards compared to the costs of those which have=20
> large IPv4/IPv6 scale. Because MPLS forwarding is so=20
> mainstream, there is no additional cost associated with=20
> forwarding rates similar to IPv4. So much of the cost of=20
> biased line cards are stripped of that, hence making them=20
> cheaper).

You put forwarding at full rate and cheaper line cards in the same
paragraph with no mention of smaller FIB size due to eliminating the
BGP routes.

> C'mon, Curtis. Everybody knows this already. Moreover, real=20
> If operators have doubts about what forwarding engines can=20
> do below 128 bytes, that is what PoC labs are for before you=20
> buy. And even then, several operators accept the=20
> restrictions up to a certain point, because the lab and real=20
> life vary significantly.
>  
> Mark.

OK ... so this is all interesting but I've lost the connection between
this discussion and draft-ietf-mpls-in-udp.  Hence the OT in the
subject.  Would you please remind me what that connection was.

Curtis

---

From: Mark Tinka <[email-masked]>
Date: Tue, 14 Jan 2014 18:09:36 +0200

On Tuesday, January 14, 2014 05:50:45 PM Curtis Villamizar 
wrote:

> None of the equipment I have worked on (a while back) or
> the chips we evaluated recently had such a problem for
> IPv6 using only the top 64 bits for forwarding.  An
> argument was made in the early 2000s (by me and others)
> that if we allocated global routes only from the top 64
> bits all equipment at that time could forward at
> essentially the same rate as for IPv4.  The reaction
> from the IPv6 community was to completely throw away the
> bottom half of the address and make it the host part.

Agree.

> There are chips for which looking at a 32 bit prefix or a
> 64 bit prefix is done within the same pipeline and
> therefore takes the same amount of time.  Other chips
> which parallelize packet handling budget enough
> microengines to get the job done (only a part of which
> is the MPLS ILM or IP destination lookup).

Agree.

> Those line cards (and designs I was recently involved in)
> assume what has historically been called a BGP-free
> core.  Only the IGP routes are carried in the core.  BGP
> is mapped onto the IGP routes.  MPLS carries traffic
> across the core.  The core has no BGP routes and
> therefore needs a lot fewer entries and the ILM can be
> put into SRAM rather than use a TCAM and only a small
> set of IP routes needs to be supported.

Yes, but a "proper" BGP-free core is only true for IPv4.

If you are running a native IPv6 backbone, you can't support 
a BGP-free core for IPv6 (I'll just call it a BGPv6-free 
core, for typing brevity).

The only way you can have a BGPv6-free core is to do 6PE, 
and like I mentioned before, I don't like tunneled IPv6 (but 
lots of other providers do it, good for them).

> That is done to eliminate a need for large external DRAM
> or TCAM which itself and the SERDES needed to go off
> chip produces heat and requires power, board space, and
> cooling and therefore reduces density. Putting more
> simple filters in the core also helps increase density.

All good and well, but where do my native BGPv6 routes go?

> For the most part IP/MPLS transport equipment seems to be
> going this way, though some people are stuck on MPLS-TP
> and trying to build a L2 overlay.

I think MPLS-TP is a way for the incumbents to migrate to 
Ethernet and still keep their ITU point-to-point-everything 
way of life that they know and love, but let's not digress 
:-)...

C have had the LSP forwarding engine for about three years 
now. I've never once considered buying it (and I always 
support new tech.), and most of my friends that I know 
closely haven't either. I have no doubt that some operators 
have bought it, but I have no empirical data to know how 
quickly it's flying off the shelves.

J's PTX and C's new NCS will continue this trend, but 
without implementing IPv6 control planes for MPLS (and the 
requisite data plane support), those of us that run native 
IPv6 will never have a BGPv6-free core.

So I can only afford line cards and forwarding engines that 
don't skimp on FIB, so I can sleep at night knowing my IPv6 
network won't fall over.

> If you go back to before 2000 at least one provider built
> a BGP-free core over Frame Relay, but FR itself soon
> died and the overlay didn't scale well.  Some providers
> wanted to do this over MPLS but the RSVP-TE restoration
> times were too long and the fallback to IP routing was
> still needed to make up for that.

Was before my time, but I can appreciate that :-).

> Unless you run a BGP-free core, whether IPv4 or IPv6. 
> Then you have plenty of FIB space.

Again, without 6PE, no-can-do for BGPv6.

> You put forwarding at full rate and cheaper line cards in
> the same paragraph with no mention of smaller FIB size
> due to eliminating the BGP routes.

Because without FIB's, where will the native BGPv6 routes 
go?

> OK ... so this is all interesting but I've lost the
> connection between this discussion and
> draft-ietf-mpls-in-udp.  Hence the OT in the subject. 
> Would you please remind me what that connection was.

I agree, this is quickly getting off topic, but the source 
was:

*****

Mark.

*****

Cheers,

Mark.

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 12:06:15 -0500

Lloyd,

Curtis

reference -

---

From: David Allan I <[email-masked]>
Date: Thu, 9 Jan 2014 20:26:27 +0000

Cheers
D

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Gregory Mirsky
Sent: Thursday, January 09, 2014 9:30 AM
To: Loa Andersson; [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] misdelivered mpls packets - Was: Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft

Hi Loa, et al.,

	Regards,
		Greg

-----Original Message-----
From: mpls [mailto:[email-masked]] On Behalf Of Loa Andersson
Sent: Thursday, January 09, 2014 8:17 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: [mpls] misdelivered mpls packets - Was: Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft

/Loa

--

---

From: <[email-masked]>
Date: Tue, 14 Jan 2014 21:57:20 +0000

I don't think sayng 'oh, that error source is no longer a problem' disproves
Stone's overall point about undetected errors, though the
examples he uses from the technology of the day are necessarily
dated. Dismissing the overall  point because the examples use obsolete
technology is throwing the baby out with the bathwater; a host-to-host
error check catches things that intermediate checks cannot.

Measuring error rates across end-to-end  Internet traffic is something that has
not received much attention , as error detection is not
instrumented well - hence citing Stone's published work,  in the absence
of awareness of anything newer (and as high profile/immediately recognisable
as sigcomm) in the area.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Stewart Bryant [[email-masked]]
Sent: 14 January 2014 18:26
To: [email-masked]; Wood L  Dr (Electronic Eng)
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [mpls] OT (was Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE: [tsvwg] Milestones changed for tsvwg WG))

Stewart

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 14 Jan 2014 17:07:32 -0500

> of awareness of anything newer (and as high profile/immediately recognisable

+1 ... the message in the paper is applicable to layered systems
and internetworks in general.  Changes in the link technology
since then don't invalidate it, especially since we know that
the technology not only changes rapidly, but also is always
growing in diverse directions, such that there things almost
universally true today may be turned on their heads tomorrow.

Designs for stacking layers need to follow solid general
principles in order to be robust to changes (above and below).

-- 
Wes Eddy
MTI Systems

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 14 Jan 2014 22:46:15 +0000

> principles in order to be robust to changes (above and below).
Note that it is not only the link layer technology that has moved on,
the signal integrity of the h/w at all stages of the design and
implementation process has moved on.

Can we agree that the statistics in the paper are discredited?

If not, why not?

What is the error rate in modern h/w and network systems?

Is this significant in the application under consideration?

Finally if we are really concerned that we do actually need a
c/s (I am not in tunnel applications) why are we still happy to
use what is frankly a pathetic check in modern terms? Why
for example are we not moving to something like
the  Fletcher 64 bit c/s?

Stewart

---

From: <[email-masked]>
Date: Tue, 14 Jan 2014 23:05:14 +0000

Stewart,

your 'I'm not in tunnel applications' suggests you've misunderstood
the argument here. The point is not to protect the tunnel traffic
(which can quite happily checksum itself), it is to protect everything
else on the network from misdelivery. It's not the tunnel application,
it's every application sharing the internet with the tunnel which
has UDP checksums turned off. See all of  RFC 6936 section 3.1.
Tunnel is fine, sideeffects of misdelivery  do not affect tunnel.

And in IPv4 and IPv6, the pseudo-header checksum built into
TCP and UDP is all we have. IPv6 deliberately copied v4 here.

> What is the error rate in modern h/w and network systems?

No-one measures end-to-end misdelivery. No-one knows.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Stewart Bryant [[email-masked]]
Sent: 14 January 2014 22:46
To: Wesley Eddy; Wood L  Dr (Electronic Eng); [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tsvwg] [mpls] OT (was Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE: Milestones changed for tsvwg WG))

If not, why not?

Stewart

---

From: <[email-masked]>
Date: Wed, 15 Jan 2014 05:20:27 +0000

That's robustness _for the tunnelled traffic_.

Not for anything else sharing the network - that hasn't been instrumented and measured.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Curtis Villamizar [[email-masked]]
Sent: 15 January 2014 03:43
To: Wood L  Dr (Electronic Eng)
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tsvwg] [mpls] OT (was Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE: Milestones changed for tsvwg WG))

Lloyd,

The part about RFC 6936 section 3.1 most relevant might be:

   There is extensive experience with deployments using tunnel
   protocols in well-managed networks (e.g., corporate networks and
   service provider core networks).  This has shown the robustness of
   methods such as Pseudowire Emulation Edge-to-Edge (PWE3) and MPLS
   that do not employ a transport protocol checksum and that have not
   specified mechanisms to protect from corruption of the unprotected
   headers (such as the VPN Identifier in MPLS).  Reasons for the
   robustness may include:

If the rate of undetected modified packets is extremely low in
"well-managed networks", as we beleive is the case, then UDP checksums
won't change the situration much.

So why *not* make them optional if experience has shown they are not
needed in the types of deployments we are talking about.

Curtis

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 15 Jan 2014 11:02:20 +0000

Sent from my iPad

> On 14 Jan 2014, at 23:05,"[email-masked]"  <[email-masked]>  wrote:
>
> Stewart,
>

Lloyd,

With the significant improvements in h/w design methodology
and the addressing of the s/w bugs that Curtis and I have been
explaining to you, it would seem that the fundamental
paper that underpins your argument is without credibility, and
by your own admission below, it seems that there are no
valid statistics for the current Internet to sustain your case.

Unless you can show that the misdelivery rate from this effect
exceeds the current rather high unwanted packet rate that
hosts need to fend off, I cannot see any basis to maintain
your assertion that the c/s is required in tunnel applications
to protect the rest of the net.

With regard to the IPv4, the IP checksum, which, in all router
implementations I have seen, is checked at every hop and
only incrementally changed, there is surely adequate
protection against misdelivery to another endpoint.

With regard to IPv6, I find it difficult to understand that
the threat is of any significance given that absence of demand
for the retrofit of a similar checksum in MPLS to protect
PEs and prevent the incorrect egress of traffic.

> No-one measures end-to-end misdelivery. No-one knows.

The upper bound is the packet loss rate less the congestion
discard rate. Do we have a figure for that?

Stewart

> To: Wesley Eddy; Wood L  Dr (Electronic Eng);[email-masked]
> Cc:[email-masked];[email-masked];[email-masked];[email-masked];[email-masked];[email-masked];[email-masked]
>>> On 1/14/2014 4:57 PM,[email-masked]  wrote:

.

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 15 Jan 2014 11:31:18 +0000

On 15/01/2014 11:08, Randy Bush wrote:
> [ you insist on cc:ing me, so you get to endure my opinions ]
>
>> it seems that there are no valid statistics for the current Internet
>> to sustain your case.
> as we discussed privately, there seem to be no real measurements to
> sustain any case.  this is all conjecturbation.
>
> what i do not understand is why, given the lack of solid evidence that
> we are in a safe space, you and others are not willing to spend a few
> euro cents to have a reasonable level of assurance at this layer.
>
> randy
Randy,

It is not a few cents, it is likely the re-engineering of a lot
of silicon.

The reason that UDP is of interest is that the on path silicon
knows how to process it, for example it knows how to to ECMP it.

The reason that the UDP c/s is a problem for a tunneler is that
it needs to have access to the whole pkt to calculate the
c/s, but as you know the silicon optimised that access away
a long time ago.

The alternative would be UDP-lite, but the ability of on path
silicon to process that as competently and as completely as it
processes UDP is by no means clear.

- Stewart

---

From: <[email-masked]>
Date: Wed, 15 Jan 2014 11:57:18 +0000

you've got the perfect application to encourage UDP lite adoption and deployment here.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: Stewart Bryant [[email-masked]]
Sent: 15 January 2014 11:31
To: Randy Bush
Cc: Wood L  Dr (Electronic Eng); [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tsvwg] [mpls] OT (was Re: draft-ietf-mpls-in-udp was RE: gre-in-udp draft (was: RE: Milestones changed for tsvwg WG))

Randy,

- Stewart

---

From: [email-masked]
Date: Wed, 15 Jan 2014 20:18:11 -0000

You may care to reference this to Section 2.2 of RFC 6936, which provides
some background to where UDP-Lite may help, and some of the potential
pitfalls.

Gorry

>
> Or perhaps UDP heavy with a FCS at the end and no checksum at all.
>
> You do make a good point that perhaps UDP lite should be mentioned in
> MPLS over UDP as an option.
>
> Curtis
>
>
> In message
> <[email-masked]>
>> Cc: Wood L  Dr (Electronic Eng); [email-masked];
>> [email-masked]; [email-masked]; [email-masked];
>> [email-masked]; [email-masked]; [email-masked]; [email-masked]
>> Subject: Re: [tsvwg] [mpls] OT (was Re: draft-ietf-mpls-in-udp was RE:
>> gre-in-udp draft (was: RE: Milestones changed for tsvwg WG))
>>

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 22:29:33 -0500

Lloyd,

At this point in time those who would know what error rates are for
current technologies are likely to be people who work for transport
vendors, router vendors, and operators and those are the type of
people you have been arguing with.  You don't really think that no one
has thought about improving error rates in 15 years, right?

That said, if an academic can do a direct measurement *today* the
results might add something to the discussion.  A paper that old adds
little for this particualar topic because too much has changed.

I anyone thought today's rates of undetectable errors were a problem
they would be adding something more robust such as a 64 bit CRC to
important link layers.  [Some bandwidth would be chewed up.  TCP ACK
packets would get bigger.  Mpps rates for "small packets" would be
easier to hit.  What's not to like. :-) ]

Curtis

In message <[email-masked]>
[email-masked] writes:

Lloyd Wood

Stewart

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 14 Jan 2014 22:43:52 -0500

Lloyd,

Curtis

---

From: Randy Bush <[email-masked]>
Date: Wed, 15 Jan 2014 17:08:09 +0600

randy

---

From: Alexander Vainshtein <[email-masked]>
Date: Wed, 15 Jan 2014 11:54:17 +0000

Stewart, and all,
I fully agree that UDP checksums is not a real-life issue with the protocol in question. They could probably help to check corrupted packets if corruption happens when a packet passes thru a router (i.e. when the ingress data link FCS has already been terminated and the egress data link FCS has not been generated yet). But this is hopefully rare - and since MPLS does not care about it, why should the MPLS encapsulator care?

I also do not think that congestion control is a serious issue for this protocol, not in the least because the primary purpose of this protocol is ECMP.

But I would like to understand whether this protocol can really result in reasonable distribution of traffic. "Reasonable" means that (a) there is sufficient entropy and (b) that the order in specific micro-flows is preserved. The draft skips this issue (unless you consider a recommendation to use a fixed  randomly selected source port value if the tunnel does not need ECMP a valid answer) .

Any ideas as to how reasonable distribution of traffic  can be achieved with this protocol? 

Regards,
       Sasha 
Email: [email-masked]
Mobile: 054-9266302

> From: mpls [mailto:[email-masked]] On Behalf Of Stewart Bryant
> Sent: Wednesday, January 15, 2014 1:31 PM
> To: Randy Bush
> Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [mpls] [tsvwg] OT (was Re: draft-ietf-mpls-in-udp was RE: gre-in-
> udp draft (was: RE: Milestones changed for tsvwg WG))
> 
> It is not a few cents, it is likely the re-engineering of a lot of silicon.
> 
> The reason that UDP is of interest is that the on path silicon knows how to
> process it, for example it knows how to to ECMP it.
> 
> The reason that the UDP c/s is a problem for a tunneler is that it needs to
> have access to the whole pkt to calculate the c/s, but as you know the silicon
> optimised that access away a long time ago.
> 
> The alternative would be UDP-lite, but the ability of on path silicon to process
> that as competently and as completely as it processes UDP is by no means
> clear.
> 
> - Stewart
> 
>

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 15 Jan 2014 11:57:54 -0500

> http://about.me/lloydwood

The reason that MPLS and PWE3 are successfully deployed is that this
traffic is carried over "well-managed networks" as RFC 6936 puts it.

If a UDP and IP encapsulation is added, then nothing changes.

If the UDP encapsulation occurs on a lower end router, it is likely
that the whole packet is available on which to perform a checksum.  On
a really low end router the checksum is likely to be done in software.

If it occurs on a high end router then the part of the hardware that
can today modify checksums only, doesn't even have access to the
packet to do a checksum and put it into the front of the packet.
There are two reasons for this.

  1.  A lot of high end routers split off the top 128-256 bytes and
      send it to a decision engine which can call for header
      modifications.  The rest of the packet takes another path and is
      later concatonated before sending out.  This works fine for
      checksum modifications but does not work for creating a new
      checksum.

  2.  A lot of high end hardware, particularly hardware intended for
      high end enterprise and data centers, uses a technique called
      "cut-through".  The first 128-256 bytes go to a decision engine
      and get processed before the packet has been entirely received.
      The decision and header modifications are done and if there is
      no standing output queue, the header starts going out before all
      of the packet has been received.  This is done to reduce
      latency.  In these implementations if the incoming FCS is bad,
      an outgoing runt packet occurs.

In the second case the UDP header is sent before the bytes over which
the UDP header is computed are received.  That is a consequence of
putting the UDP checksum before the data.  L2 encapsulation put the
FCS after the data for this reason.

So what you are asking, a UDP checksum on a fresh new encapsulation is
impossible for some hardware.  [ps - thanks to an offline discussion
with Joel Halpern for bringing this up.]

Perhaps we need a new UDP with a robust FCS at the back of the
packet, but without that making a UDP checksum manditory for MPLS over
UDP is guarenteed to be ignored in some deployments and with no
adverse consequences.  Is that what we want?

Perhaps there can be discussion added to the draft to indicate why a
UDP checksum is desirable and why in some circumstances it may be
impossible to generate or difficult to check.  This along with a
SHOULD use a UDP checksum might be the best course of action.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 15 Jan 2014 15:02:07 -0500

> [ you insist on cc:ing me, so you get to endure my opinions ]

Not a problem (this time).  :-)

Randy,

See http://www.ietf.org/mail-archive/web/mpls/current/msg11279.html
for reasons why routers would want to avoid having to fill in a
checksum.  It would have been more feasible for an FCS at the end of
the packet for these cases but the UDP checksum is in the front.

This entire discussion is over putting in a SHOULD rather than a MUST
in two places, UDP checksum and congestion control, plus deferring
defining the congestion control for MPLS over UDP until deployments
show a need for it.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 15 Jan 2014 15:05:31 -0500

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 15 Jan 2014 15:34:34 -0500

In message <[email-masked]>
Alexander Vainshtein writes:
> 
> Stewart, and all,
>  
> I fully agree that UDP checksums is not a real-life issue with the
> protocol in question. They could probably help to check corrupted
> packets if corruption happens when a packet passes thru a router
> (i.e. when the ingress data link FCS has already been terminated and
> the egress data link FCS has not been generated yet). But this is
> hopefully rare - and since MPLS does not care about it, why should the
> MPLS encapsulator care?

Why the MPLS over UDP encapsulation cares is in
http://www.ietf.org/mail-archive/web/mpls/current/msg11279.html

> I also do not think that congestion control is a serious issue for
> this protocol, not in the least because the primary purpose of this
> protocol is ECMP.

The chips I had worked with all did a lookup and retrieved a next-hop
index.  That could point to a single hop or and entry into the
multipath hardware gunk (tables for some not so good implementations).

My guess is Stewart is concerned about hardware than can lookup an IP
adddress and do multipath but not do an MPLS ILM lookup and then do
multipath.  [I find that hard to believe since it would be tough to do
MPLS over a LAG.  Most chips that had not anticipated this could do
this with firmware changes because there was enough flexibility in the
hardware intended for LAG to get the job done.  Some might have
limitations with ECMP where component links of the ECMP were LAG.
OTOH the further you go back in time the more severe chip limitations
you will run into.]

IMHO the draft should remove the ECMP motivation from the introduction
and then there would be no need to debate this.  The draft need only
define the protocol.

> But I would like to understand whether this protocol can really result
> in reasonable distribution of traffic. "Reasonable" means that (a)
> there is sufficient entropy and (b) that the order in specific
> micro-flows is preserved. The draft skips this issue (unless you
> consider a recommendation to use a fixed randomly selected source port
> value if the tunnel does not need ECMP a valid answer) .
>  

ECMP has been around for a long time and providers carefully tune IGP
metrics to get a better but often not very good load balance with
ECMP.

With other multipath methods you can get non-equal load balance and
remove the equal cost limitiation but that might need more recent
hardware than Steward and others are faced with.

ECMP is a blunt tool.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 15 Jan 2014 15:48:47 -0500

> some background to where UDP-Lite may help, and some of the potential

The right tool for the job but ...

Expect some pushback because older hardware looks at TCP and UDP in
the protocol field and then checks port numbers for ECMP load
balance.  That older hardware will not do this for UDP-Lite.

With MPLS over UDP, only the dest port number matters and the src port
number can be used like the MPLS Entropy Label.  That is about the
only thing that would not work in some networks with UDP-Lite.

IMHO it would be advantageous to provide the option to use UDP-Lite
rather than UDP with no checksum at all.  The section in RFC 6936
could be cited.  The limitation I mentioned could also be cited.

Curtis

---

From: Saku Ytti <[email-masked]>
Date: Fri, 17 Jan 2014 18:33:07 +0200

> purpose.  For the same reason TCP and UDP checksums server no purpose.

One datapoint from today. Peering router has 4xLACP to core and in one of
these LACP members it is sometimes mangling packets during send and FCS is
being calculated for this mangled data.
All egress PE boxes start logging errors (maybe 1 error per 30min), because
they check IP checksum, which is now incorrect.

Router is latest generation service provider router from major vendor running
recent software and affected router is logging no errors.  Can't imagine when,
if ever, would this issue have been found without IP checksums.

-- 
  ++ytti

---

From: Saku Ytti <[email-masked]>
Date: Sat, 18 Jan 2014 00:32:43 +0200

On (2014-01-17 15:28 -0500), Curtis Villamizar wrote:

> This sounds like a router acting as a host for the given packet.  The
> forwarding data paths in routers are usually protected.  If the router

This was transit packets ingressing as IP packet, egressing as MPLS packet.
Consequently the peer router needed to touch TTL and checksum. Then generate
new L2 frame.
MPLS core obviously need not to worry about packet being broken, as frame is
fine.

> was forwarding and mangled packets, that speak well for the router
> vendor since the major vendors claim to protect their data paths from

Are such claims realistic or practical? I think it sounds like snakeoil when
someone claims such absolutes.

> this sort of thing.  Less so for the path from the route processor.

This was forwarding plane.

> You've made a good anecdotal case for keeping TCP and UDP checksums.

This would actually require IP checksum specifically, otherwise we would have
probably never found it. If each egress PE suffers from it maybe once 30min,
each customer suffers from it less than once a month, even if everyone would
catch it and open trouble ticket, we couldn't correlate the cases to any
single peering router, without having historic routing data for each egress
PE.

Disclaimer, I'm not for or against checksum/FCS in any particular layer, I've
not thought about the matter deeply. It was just extremely timely thing in
real-life network I wanted to share.

-- 
  ++ytti

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 17 Jan 2014 15:28:19 -0500

In message <[email-masked]>
Saku Ytti writes:
 
> recent software and affected router is logging no errors.  Can't imagine
> when, if ever, would this issue have been found without IP checksums.
>  
> -- 
>   ++ytti

If this was originated at the router it would also be caught if AH was
enabled.  Were these unauthenticated control plane packets?

Curtis