Subject: Re: RE: [802.1] 802.1ad Customer FCS Retention

From: Eric Rosen <[email-masked]>
Date: Mon, 24 Nov 2003 13:31:35 -0500

Scott> As some background, what started  MCI's quest for FCS retention was a
Scott> lab infrastructure  Ethernet switch  that was corrupting  packets and
Scott> calculating a  new FCS on packets  where no changes  should have been
Scott> occurring.   This was  one  of  the top  Ethernet  switch vendors  as
Scott> provided by Gartner in 2002.   This took days to troubleshoot, and as
Scott> service providers, scared the Hell out of us.

My understanding  is that the customers  noticed the problem in  the form of
TCP checksum errors, and complained to MCI. 

I'm not  sure how  having the customers  notice the  problem in the  form of
ethernet checksum errors rather than  TCP checksum errors would have made it
any easier for you to diagnose the problem.  

Scott> The TRANS layer often refers to transport over SONET, but there is no
Scott> reason  why it  shouldn't also  support the  model of  transport over
Scott> Ethernet,  in which  case  the requirements  are  different from  ETH
Scott> requirements. 

Why is that?   In PWE3, we are accustomed to hearing  that SPs can't provide
an IP service  unless IP becomes more  like ATM, but this is  the first I've
heard that  SP's can't provide  an ethernet service unless  ethernet becomes
more like SONET.

Scott> The  idea  of  VLAN  stacking,  VLAN  swapping,  marking  as  discard
Scott> eligible, and  messing with packets in  general seems to  open up the
Scott> possibility of such things happening even more. 

All these things have been  happening for many years.  In routers, rewriting
of  the data  link  layer header  has  to happen.   I am  not  aware of  any
empirical  data suggesting that  a node  which rewrites  the header  is more
likely to corrupt  the data than a  node which does not (or  is not supposed
to) rewrite the header.

---

From: "Mick Seaman" <[email-masked]>
Date: Mon, 24 Nov 2003 12:05:29 -0800

There seems to be an undercurrent in this discussion that somehow an FCS-32
is good protection against arbitrary packet mangling. Sorry, not true.

The FCS/CRC-32 is designed to protect against a limited number of changes in
bits, and was designed against a noise error model that rated changes in
more than a certain number of bits improbable. You can't reliably use the
CRC to defend against a wider range of errors such as omission or
replacement of whole bytes or words (which has been know to sporadically
occur in DMA controllers designed close to their limits), or substitution of
part of the frame for another frame (which might occur if frames were
interleaved on a synchronous clock and the clock slipped a bit position once
in a while for example). Nor does the FCS protect well against software
errors where a relay system does something other than it is supposed to do
when forwarding a frame.

The protection offered by a CRC-32 against such events is little better than
the chance of randomly guessing the right number in 2**32. To put that in
perspective if all frames on a fully utilised 10 Gb/s link were errored you
would get a frame with a good FCS about once every 20 minutes. Of course if
all the frames were errored the link would probably come down, but spread
around a network or many the errored 10Gb/s aggregate is more plausible. Now
if you factor out all the errors that are going to get caught with the exact
same CRC algorithm used on a link by link basis then the extra CRC is pretty
much a waste of 4 bytes. At least use a different integrity check if you are
serious! (and not one that the existing hardware is not going to
accidentally emulate by accident when there is a software bug!).

This discussion seems to be largely driven by philosophy, if it is going to
go anywhere at all there better be an engineering component.

Separately you are not going to get bit level transparency for a bridged
network. In particular we have already established the layering model will
not be broken to allow media specific frames (PAUSE, LACP) etc. to transit
though bridges.

Mick

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Scott Kotrla
Sent: Monday, November 24, 2003 9:40 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [PWE3] RE: [802.1] 802.1ad Customer FCS Retention

As a service provider, I want layers.  I want to be as a transparent as
possible.  Throwing away the customer's FCS and introducing the
possibility of undetected errors is not transparent.  VLAN stacking
isn't the best solution here for layer encapsulation, but it can work.
MAC-in-MAC is architecturally a better fit because the whole customer
frame goes into the new payload field, but wasting 12 bytes for dummy
DA/SA that won't be used for TRANS (MEF layer) applications doesn't make
a lot of sense.  ETH layer functionality is the application of existing
802 technologies to service providers, so arguments about what exactly
does or does not belong has value.

TRANS layer functionality for physical Ethernet transport is new
functionality that 802 was not designed for.  But for service providers,
this provides new requirements.  The TRANS layer often refers to
transport over SONET, but there is no reason why it shouldn't also
support the model of transport over Ethernet, in which case the
requirements are different from ETH requirements.  If physical Ethernet
is being used for point-to-point TRANS functionality, new features like
a Path FCS are required.  Otherwise, Ethernet over Fiber is not a viable
alternative to Ethernet over SONET.

In this context, it may be clearer how Stacked VLANs and PW are
comparable.  As a provider, I am looking for a way to tunnel customer
ports or customer VLANs from point-to-point from end-to-end
transparently across the access network.  I am open to other
alternatives.  Ethernet TRANS should maintain customer FCS end-to-end as
a separate layer from the service Ethernet Link FCS.

As some background, what started MCI's quest for FCS retention was a lab
infrastructure Ethernet switch that was corrupting packets and
calculating a new FCS on packets where no changes should have been
occurring.  This was one of the top Ethernet switch vendors as provided
by Gartner in 2002.  This took days to troubleshoot, and as service
providers, scared the Hell out of us.  The idea of VLAN stacking, VLAN
swapping, marking as discard eligible, and messing with packets in
general seems to open up the possibility of such things happening even
more.

Thanks,

Scott

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Friday, November 21, 2003 11:16 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [PWE3] RE: [802.1] 802.1ad Customer FCS Retention

Hi Norm....We are on the same page I am sure, just using different
words....please see in-line.

regards, Neil

> Norman Finn wrote 21 November 2003 12:03
>
>
> Neil,
>
> I think Pat Thaler's reply helps to straighten out this
> confusion, but let me
> try, also.
>
> There is a *lot* of confusion about the words "end-to-end",
> and about "transport layers".
NH=> Agreed.  G.805 and G.809 func arch really helps sort this out...as
I hope was clear this week from Maarten's excellent analysis of the
ethernet OAM issues at SG13.

> There are lots of ends in this picture, and there are layers
> and there are layers.
NH=> Agreed.....just wish some more people grokked this stuff as it
should help mitigate arch violates.
>
> An 802.1Q VLAN bridge, and by extension, a P802.1ad provider
> bridge, does NOT supply a
> "transport layer" in the same sense as IP-over-Ethernet or
> Ethernet-over-IP.  This is
> because, at the data forwarding layers, VLAN tags are best
> though of as extensions to
> the address space, not as additional protocol layers.
NH=> Understood and agree.

>  If you
> want to call it a
> "transport layer", feel free!
NH=> No, that would be completely wrong....VLANs do *not* form a layer
network in any way, shape or form.

>  But don't assume that VLANs
> are equivalent to other
> "transport layers".  Data integrity, specifically, is
> different in bridges.
>
> Routers provide "best effort" not only in frame delivery, but
> in data integrity.  There
> is no upper limit specified for the probability of delivery
> of a packet with undetected
> errors.  IP's response, if you demand a particular level of
> data integrity, is that
> your application must supply its own checksum.
>
> Bridges, however, are not routers!
>
> Bridges *do* guarantee a level of service.  They manage to do
> this by guaranteeing the
> quality of each bridge, and by *demanding* the quality of
> each link.  Non-VLAN bridges,
> VLAN bridges, and Provider Bridges all make that guarantee to
> their users using this
> method.  The Customers of a Provider Bridge service are not
> an "802.1Q application"
> over an "802.1ad transport layer".  They are, in the 802.1
> model, just endstations
> attached to a bridge.  The Customers' data is just as well
> protected as an enterprise
> bridges' endstations, as long as the model is maintained.
> Adding an additional CRC
> to end-to-end service, where the ends are Customers attached
> to 802.1ad Provider
> Bridges, is therefore pointless.  Changing that model is as
> silly as turning the model
> for routing upside down.
NH=> I agree with all this....so not quite sure what you think I am not
understanding here.
>
> When a Pseudowire is used to connect two bridges,
NH=> Similar to VLANs, a PW is *not* a layer network either.  MPLS does
create layer networks however....but I suspect that is not well
understood by many.  One of my colleagues, who like Maarten is a real
expert in the func modelling detail, is formally describing MPLS right
now...but of course MPLS is a not a singular layer network, it has many
flavours and this work will clearly bring out the different arch
behaviours.

> the
> "end-to-end" is now just those
> two bridges.
NH=> Agreed.  If one drew out the proper func arch model one can see
this immediately.

> Those bridges are using that PW as an Ethernet
> to carry their traffic.
> The bridges demand that every wire be as reliable as an
> Ethernet.  That's the PW CRC
> issue.
NH=> Grokked totally. A PW is a form of adaptation function that lies
between a client (here we are discussing ethernet) and MPLS (or IP if
one is happy with a cnls server behaviour)...and then there are further
layer networks below this, eg the VC4 layer network of SDH, until we hit
the duct.  The fact that a PW is not a layer network and not really
necessary is neither here nor there....other than its just something
else to manage.  When the port on a bridge is not connected to an
'ethernet shared media' (orginal case), but some technology (any) then
one simply has to adapt the ethernet PDU (*including its FCS*, as its
part of the client ethernet CI) to the payload of the server layer.  One
should *not* be playing around with client layer network structures in a
simple client->server adaptation process at all.  So I am totally
against all the 'Martini' PW compression stuff....the BW squeezing is
simply not worth it (compared to other technolgies that will squander
BW). So !
 IMO the ethernet FCS is part of the ethernet's CI and should not be
removed by *any* server layer technology.
>
> -- Norm
>
> [email-masked] wrote:
> > Norm....As per our discussion today face-face.  I don't
> really see what the problem is here.  The client layer
> network should not care about the server layer network (and
> vice versa)....we need this layer independence for lots of
> reasons.  An ethernet traffic unit has a FCS.  This should
> simply be transported transparently in the payload area of
> whatever the server layer is....and the server layer should
> use its own (OAM) 'integrity mechanisms' irrespective of the
> client.  Problems arise when people start doing things like
> client layer compression and thus forming inter-layer dependencies.
> >
> > Further, as we agreed today and as I posted on an earlier
> mail to this thread, the ethernet FCS is a CRC that can be
> linerarly updated for adaptation function changes (eg VLAN ID
> change) on each link-connection (as provided by some server
> layer trail technology) whilst preserving its end-end
> intergrity check of the rest of the ethernet traffic unit
> between its trail termination points.
> >
> > regards, Neil
> >
> >
> >>-----Original Message-----
> >>From: Norman Finn [mailto:[email-masked]]
> >>Sent: 18 November 2003 08:56
> >>To: mark seery
> >>Cc: [email-masked]; [email-masked]
> >>Subject: Re: [PWE3] RE: [802.1] 802.1ad Customer FCS Retention
> >>
> >>
> >>Mark,
> >>
> >>You say below,
> >>
> >> > but a PW will be running over a physical media itself
> >>already, possibly
> >> > even SONET.............
> >>
> >>Exactly!  Here, the trail gets a bit murky.  If I'm in total
> >>control, and the PW
> >>is running over known Ethernet links, then I can get away
> >>without the CRC in the
> >>PW.  I have no problem whatsoever with the current definition
> >>of PWs without CRC.
> >>
> >>But, if I don't have tight control over the physical media
> >>over which the PW runs,
> >>then I really need a CRC on the PW for the bridge to be able
> >>to trust the "wire"
> >>and thus provide the same level of data integrity that
> >>bridges offer over physical
> >>Ethernets.  So, I claim that bridges need a CRC option on PWs.
> >>
> >>-- Norm
> >>
> >>mark seery wrote:
> >>
> >>>Norman Finn wrote:
> >>>
> >>>
> >>>><snip>
> >>>>In other words, routers pass about $0.93 of the data
> >>>
> >>integrity buck to
> >>
> >>>>bridges,
> >>>>and bridges pass about $0.75 of that to the data links.
> >>>
> >>The buck has
> >>
> >>>>to stop
> >>>>somewhere, and the data link is where it stops.  The PW
> >>>
> >>has to take
> >>
> >>>>the same
> >>>>hit as the physical media.
> >>>>
> >>>>-- Norm
> >>>
> >>>
> >>>but a PW will be running over a physical media itself
> >>
> >>already, possibly
> >>
> >>>even SONET.............
> >>>
> >>>mark
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>_______________________________________________
> >>pwe3 mailing list
> >>[email-masked]
> >>https://www1.ietf.org/mailman/listinfo/pwe3
> >>
> >
> >
>
>
>
> _______________________________________________
> pwe3 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/pwe3
>