Subject: RE: Stewart Bryant's Discuss on draft-ietf-6man-udpzero-06: (with DISCUSS and COMMENT)

From: "Adrian Farrel" <[email-masked]>
Date: Tue, 9 Oct 2012 13:46:38 +0100

Are we in the weeds yet?

One thing about MPLS is that it is not substantially run as a native layer 2
encapsulation. The result is that it is usually MPLSoE or similar (even when the
LSP is being used to carry Ethernet). The data link thus usually provides some
form of robustness such as CRC that protects the packet on the wire. So we are
left with random hits on packets inside router buffers, or subverted/malicious
routers.

Hence MPLS seems to be considered safe at the moment.

It is unclear what will happen when/if MPLS is used as a native layer 2. Maybe,
however, MPLSoLambda will use its own FCS perhaps provided by GFP.

Adrian

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Stewart Bryant
> Sent: 09 October 2012 13:34
> To: [email-masked]
> Cc: [email-masked]; [email-masked];
> [email-masked]; The IESG; [email-masked]
> Subject: Re: Stewart Bryant's Discuss on draft-ietf-6man-udpzero-06: (with
> DISCUSS and COMMENT)
> 
> On 08/10/2012 09:07, [email-masked] wrote:
> > Thanks for your comments, see below.
> >
> > Gorry
> >
> >> Stewart Bryant has entered the following ballot position for
> >> draft-ietf-6man-udpzero-06: Discuss
> >>
> >> When responding, please keep the subject line intact and reply to all
> >> email addresses included in the To and CC lines. (Feel free to cut this
> >> introductory paragraph, however.)
> >>
> >>
> >> Please refer to http://www.ietf.org/iesg/statement/discuss-criteria.html
> >> for more information about IESG DISCUSS and COMMENT positions.
> >>
> >>
> >> ----------------------------------------------------------------------
> >> DISCUSS:
> >> ----------------------------------------------------------------------
> >>
> >> This discuss applies to both this draft and
> >> draft-ietf-6man-udpchecksums-04
> >>
> >> Fundamentally I support the liberalization of the position regarding IPv6
> >> UDP checksums  when UDP is used for tunnels and hope to get to a yes
> >> position on both drafts. However, there is some text  that needs
> >> discussion, and then we need to discuss the consequences of that
> >> discussion. The rational for the slightly conservative position taken in
> >> the case of tunnels seems to based on the following text:
> >>
> >> "IP packets may be corrupted as they traverse an Internet path.
> >> Evidence has been presented [Sigcomm2000] to show that this was once
> >> an issue with IPv4 routers, and occasional corruption could result
> >> from bad internal router processing in routers or hosts.  These
> >> errors are not detected by the strong frame checksums employed at the
> >> link-layer [RFC3819].  There is no current evidence that such cases
> >> are rare in the modern Internet, nor that they may not be applicable
> >> to IPv6. It therefore seems prudent not to relax this constraint.
> >> The emergence of low-end IPv6 routers and the proposed use of NAT
> >> with IPv6 further motivate the need to protect from this type of
> >> error."
> >>
> >> However we do have a body of evidence that is not discussed in the
> >> document. Firstly we have a decade of experience with MPLS VPNs where the
> >> VPN Identifier label, which is used to steer the traffic to a particular
> >> customer network (i.e. is functionally equivalent to an address), is not
> >> protected by a checksum. I am not aware of any concerns expressed by
> >> operators in this regard, and I am not aware of any work in the MPLS WG
> >> to introduce checksum like protection.
> >>
> > 1) I'm not sure whether you are saying MPLS is robust to these effects
> > *OR* that MPLS stacks have monitored for this case, e.g. do log such an
> > unexpected VPN Identifier label, and somehow report this (and hence there
> > is evidence that MPLS routers do not see such errors)
> MPLS is not robust against this. If it gets a packet with a valid DL CRC
> and it finds the VPNid in the label table, the packet gets delivered
> without further ado. However if it gets delivered to the wrong customer
> then that is a security violation, and that does not seem to be an
> issue expressed by the operator or user community, and thus I conclude
> that the occurrence is not significant.
> 
> It's implementation specific, but I am fairly sure that unknown VPNid will
> be counted in most implementations.
> 
> >> We also have a decade of experience with pseudowires. With one exception
> >> that we will discuss in a minute, none of the PW designs have any form of
> >> data integrity protection other than the CRC imposed by the network
> >> datalink at each hop. In the specific case of the Ethernet PW, there are
> >> two modes: one in which the CRC is stripped at ingress to the PW and
> >> recalculated at egress, and another where it is preserved end to end.
> >> There is very little, if any, deployment of the end to end CRC
> >> preservation mode. Given that there must be some low level of corruption
> >> of the frames, I would conclude that the protocols that are likely to be
> >> tunneled are sufficiently hardened that the occasional corruption is of
> >> no practical consequence.
> >>
> > 2) On acceptability of residual corruption: I think I missed something -
> > If you are saying that using Ethernet "bridging" is safe at L2, then
> > that's not this discussion. I am interested in any experience that would
> > help in understanding the impact of no checksum for IPv6, e.g. experience
> > in how often the IPv4 checksum and IPv4/UDP checksum fail.
> To be clear I am only considering the tunneling case not the general case,
> and note that with the most frequently used tunneling protocol (MPLS)
> it is unprotected and this does not seem to be causing any issue.
> >
> > 3) On practical rate of residual corruption:  Is there experience of using
> > MPLS/PWE across low-end and legacy routers?  I am not aware of this, but I
> > would be really interested in knowing more about the reliability across an
> > end-to-end path i.e. PWE and MPLS through home gateways, firewalls,
> > load-balancers, NAT and NAPT. What experience has the community seen?
> People are looking at low end with a view to pushing MPLS out to
> closer to the edge of the network, but the concern is with label
> security rather than label corruption. However there is not much
> experience with low end boxes.
> >
> >> Thus I would conclude that the level of corruption is sufficiently rare
> >> and of sufficiently minor consequence that at least in the case of
> >> service provider networks implementing UDP tunneling, it is safe to omit
> >> the UDP checksum without analysis of the application.
> >>
> > 4) I disagree.
> I would assume so. I suspect because we are looking at different
> aspect of the problem.
> >
> >> I would propose that the text should include some acknowledgement of this
> >> recent body of evidence and that the recommendation be aligned in
> >> consequence to unconditionally allow C/S free tunneling, at least within
> >> a well managed domain without additional consideration.
> >>
> > 5) It seems OK to note the new evidence for use within a well-managed
> > domain, can you supply references to the usage and experience? I have a
> > question: would the domain include host stacks and end-to-end usage or
> > just carrier usage?
> The experience we have is with carrier usage, and we only have (strong)
> negative evidence. People rarely write reports saying that a system works
> well (as expected).
> 
> >
> >> Section 5.1 seems to be duplicated in the two drafts. It would be better
> >> if the text were just in one RFC and the other pointed to it.
> >>
> >>
> >> ----------------------------------------------------------------------
> >> COMMENT:
> >> ----------------------------------------------------------------------
> >>
> >> You also note that an application may rely on the checksum to protect it
> >> against packets that may damage it. I find this very weak argument, since
> >> such an application would be vulnerable to packets deliberately malformed
> >> by an attacker, or malformed by a software bug in a peer.  The correct
> >> approach is surely to harden the application, and thus the checksum
> >> argument is not persuasive.
> >>
> > 6) I'm surprised/misunderstood - many deployed uses of the UDP transport
> > have relied on the UDP cksum (more latterly this not been the advice of
> > the IETF, but hard to enforce). Is this suggesting the IETF actually
> > declare these (existing deployed IPv4) apps as unsuitable for use with
> > IPv6 over the general Internet. That's NOT something I would personally
> > like to do, have I understood correctly?
> No, but your argument is non-the-less weak. If the only thing holding
> the app from melting is the UDP c/s, then it really needs some
> security hardening.
> 
> >
> >> You note the need to signal the agreement not to use c/s, I think that
> >> you need to include the configuration of this property as an alternative,
> >> and note that configuration may be implicit in the definition of the UDP
> >> payload.
> >>
> > 7) The intent was not to change the semantic of the cksum for apps that
> > expected the original behaviour. If you have an app that is willing to use
> > no cksum - such as the tunnels - this may be fine, then I see two
> > possibilities: It works (e.g. within a carrier network) or it may fail
> > (e.g. via a NAPT/firewall that recalculates a cksum). I think it would be
> > "nice" for the "app" to know it has UDP connectivity, but no UDP zero
> > cksum connectivity - i.e. not just break when a route changes or a user
> > moves to a new network. What do you suggest apps should do?
> >
> By definition an app does not know that it is being tunneled.
> If the app wants to set the c/s in the UDP header that it requests from
> the host stack, that is something that I actively support. However
> the path over the network once it leaves that stack (including
> tunneling in the host itself) is not something that it should know
> or care about.
> 
> - Stewart

---

From: Brian Haberman <[email-masked]>
Date: Tue, 09 Oct 2012 08:56:16 -0400

> Are we in the weeds yet?

I believe we are.

> left with random hits on packets inside router buffers, or subverted/malicious

Agreed.  As long as there is a robust L-2, the chance of in-transit 
errors is low.  If MPLS is predominantly carried over Ethernet, I 
suspect the Ethernet CRC is providing sufficient/significant protection.

> however, MPLSoLambda will use its own FCS perhaps provided by GFP.

Agree as well.

> Adrian

Regards,
Brian

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 09 Oct 2012 14:45:03 +0100

>> however, MPLSoLambda will use its own FCS perhaps provided by GFP.

I agree with the above, but it is even more applicable to the UDP 
tunneling case, since
the stack will normally be datalink(CRC)/IPv6/UDP/Payload, and 
IPv6oLambda is not
AFAICS on anyone's radar.

Stewart

S

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 10 Oct 2012 10:30:44 +0200

> On 10/9/12 8:46 AM, Adrian Farrel wrote:

Likely, but the weeds here are fascinating and possibly relevant.

> suspect the Ethernet CRC is providing sufficient/significant protection.

I think you are missing one point of the following reference:
[Sigcomm2000]
              Jonathan Stone and Craig Partridge , , "When the CRC and
              TCP Checksum Disagree", 2000.

That is that this paper found a significant amount of erroneous packets
where the UDP or TCP checksum discarded the packet, not the L2 CRCs. The
whole point of the paper is trying to analyze what type and why their is
errors that only the transport checksums detects.

When the topic of UDP checksums was discussed in the WG it was
questioned if this now 12 year old paper was still relevant. People in
the WG went looking at their network interface stats and found that
several different machines has UDP checksum discard rates that was
higher than one every 1000 packets. So yes there is still a lot of
packets being discard in the destination host due to transport
checksums. If "Why" has changed we don't know because someone needs to
repeat the study Stone and Partridge did.

I would not be surprised if there is differences between IP routing,
especially at the edge and MPLS label switching in regards to error
characteristics.

I think my main point around this whole issue is that you have to be
careful to understand the environment and what mitigating factors that
may exist so that you don't see these issues.

I am fine with updating the text to better reflect reality. However, I
don't understand this MPLS and PW area sufficiently to determine what
error factors and saving graces that happens to be built into the system.

The whole point of the argumentation and checklist is. Checksums are
there for a reason. IF you have analyzed it the implications of turning
of the checksum and what it means to use a particular protocol header
field that is now unprotected then yes you can go ahead and use it.

But don't forget how other users of the network might suffer from your
traffic.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Brian Haberman <[email-masked]>
Date: Wed, 10 Oct 2012 13:25:48 -0400

> Likely, but the weeds here are fascinating and possibly relevant.

Given what you say below, I am not sure how relevant it is (regardless 
of the fascination factor).

> errors that only the transport checksums detects.

If your comment about missing the point of the above paper is directed 
to me, it is quite the opposite.  Given that I was involved in those 
6MAN WG discussions about this draft, I have a good understanding of the 
questions raised by the paper.

> especially at the edge and MPLS label switching in regards to error

Completely agreed.

> may exist so that you don't see these issues.

Yes!  And that is why the draft is currently restricting the UDP 
checksum value of 0 to tunneling protocols.  I don't see any 
justification, at *this* time, to loosen the rules in general.

> But don't forget how other users of the network might suffer from your

Right.  This leads me to believe that the draft should remain focused on 
allowing tunneling protocols to use checksum=0 when the encapsulated 
payload is protected by a checksum.

Regards,
Brian

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 10 Oct 2012 18:37:49 +0100

> Right. This leads me to believe that the draft should remain focused 
> on allowing tunneling protocols to use checksum=0 when the 
> encapsulated payload is protected by a checksum. 
I would phrase this as "the draft should allow tunneling protocols to 
use checksum=0 and assume that the encapsulated payload is protected by 
a checksum."  Since that is what is going to happen in practice.

Stewart