Subject: RE: RE: [802.1] 802.1ad Customer FCS Retention

From: "Mick Seaman" <[email-masked]>
Date: Tue, 25 Nov 2003 11:53:37 -0800

Scott,

If switches in deployment were corrupting packets at the rate you observed
in your lab the millions of switch ports that have been deployed simply
would not be there.

Mick
-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Scott Kotrla
Sent: Tuesday, November 25, 2003 8:36 AM
To: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [PWE3] RE: [802.1] 802.1ad Customer FCS Retention

So are you suggesting because a CRC-32 will miss 1 in 4 billion frames
it is not useful to know?  I don't need to catch every frame to know
there is a problem.

The standard Ethernet CRC protects the link and not the internal switch,
and it is very difficult to put requirements on the internal working of
a switch.  As stated in a previous message:

"The model of operation is simply a basis for describing the
functionality of the MAC Bridge. It is in no way intended to constrain
real implementations of a MAC Bridge; these may adopt any internal model
of operation compatible with the externally visible behavior that this
standard specifies. Conformance of equipment to this standard is purely
in respect of observable protocol."

I am not worried about link errors.  I am worried about corruption
within switches.  If switches have unknown failure modes that add
unknown corruption, what besides a Path FCS will be able to determine
that there is a problem.  Once again, detection is at least 10 times as
important as diagnosis.  If we can come up with a better way to detect
when (not if) switches are corrupting packets, I would be very
interested in understanding the benefits of this approach.

The service provider requirement can be succinctly stated as:

The network shall be able to detect when (not if) switches are
corrupting packets.  Suggesting that switches will never unknowingly
corrupt packets is wishing the problem away and is not an acceptable
answer.  Never is a very long time.  Besides, out of the 100 or so
switches I have worked with, I have already seen it happen.  And I am
not all that lucky.

Thanks,

Scott

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Mick Seaman
Sent: Monday, November 24, 2003 2:05 PM
To: [email-masked]; [email-masked]
Subject: RE: [PWE3] RE: [802.1] 802.1ad Customer FCS Retention

There seems to be an undercurrent in this discussion that somehow an
FCS-32
is good protection against arbitrary packet mangling. Sorry, not true.

The FCS/CRC-32 is designed to protect against a limited number of
changes in
bits, and was designed against a noise error model that rated changes in
more than a certain number of bits improbable. You can't reliably use
the
CRC to defend against a wider range of errors such as omission or
replacement of whole bytes or words (which has been know to sporadically
occur in DMA controllers designed close to their limits), or
substitution of
part of the frame for another frame (which might occur if frames were
interleaved on a synchronous clock and the clock slipped a bit position
once
in a while for example). Nor does the FCS protect well against software
errors where a relay system does something other than it is supposed to
do
when forwarding a frame.

The protection offered by a CRC-32 against such events is little better
than
the chance of randomly guessing the right number in 2**32. To put that
in
perspective if all frames on a fully utilised 10 Gb/s link were errored
you
would get a frame with a good FCS about once every 20 minutes. Of course
if
all the frames were errored the link would probably come down, but
spread
around a network or many the errored 10Gb/s aggregate is more plausible.
Now
if you factor out all the errors that are going to get caught with the
exact
same CRC algorithm used on a link by link basis then the extra CRC is
pretty
much a waste of 4 bytes. At least use a different integrity check if you
are
serious! (and not one that the existing hardware is not going to
accidentally emulate by accident when there is a software bug!).

This discussion seems to be largely driven by philosophy, if it is going
to
go anywhere at all there better be an engineering component.

Separately you are not going to get bit level transparency for a bridged
network. In particular we have already established the layering model
will
not be broken to allow media specific frames (PAUSE, LACP) etc. to
transit
though bridges.

Mick

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Scott Kotrla
Sent: Monday, November 24, 2003 9:40 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [PWE3] RE: [802.1] 802.1ad Customer FCS Retention

As a service provider, I want layers.  I want to be as a transparent as
possible.  Throwing away the customer's FCS and introducing the
possibility of undetected errors is not transparent.  VLAN stacking
isn't the best solution here for layer encapsulation, but it can work.
MAC-in-MAC is architecturally a better fit because the whole customer
frame goes into the new payload field, but wasting 12 bytes for dummy
DA/SA that won't be used for TRANS (MEF layer) applications doesn't make
a lot of sense.  ETH layer functionality is the application of existing
802 technologies to service providers, so arguments about what exactly
does or does not belong has value.

TRANS layer functionality for physical Ethernet transport is new
functionality that 802 was not designed for.  But for service providers,
this provides new requirements.  The TRANS layer often refers to
transport over SONET, but there is no reason why it shouldn't also
support the model of transport over Ethernet, in which case the
requirements are different from ETH requirements.  If physical Ethernet
is being used for point-to-point TRANS functionality, new features like
a Path FCS are required.  Otherwise, Ethernet over Fiber is not a viable
alternative to Ethernet over SONET.

In this context, it may be clearer how Stacked VLANs and PW are
comparable.  As a provider, I am looking for a way to tunnel customer
ports or customer VLANs from point-to-point from end-to-end
transparently across the access network.  I am open to other
alternatives.  Ethernet TRANS should maintain customer FCS end-to-end as
a separate layer from the service Ethernet Link FCS.

As some background, what started MCI's quest for FCS retention was a lab
infrastructure Ethernet switch that was corrupting packets and
calculating a new FCS on packets where no changes should have been
occurring.  This was one of the top Ethernet switch vendors as provided
by Gartner in 2002.  This took days to troubleshoot, and as service
providers, scared the Hell out of us.  The idea of VLAN stacking, VLAN
swapping, marking as discard eligible, and messing with packets in
general seems to open up the possibility of such things happening even
more.

Thanks,

Scott

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Friday, November 21, 2003 11:16 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [PWE3] RE: [802.1] 802.1ad Customer FCS Retention

Hi Norm....We are on the same page I am sure, just using different
words....please see in-line.

regards, Neil

> Norman Finn wrote 21 November 2003 12:03
>
>
> Neil,
>
> I think Pat Thaler's reply helps to straighten out this
> confusion, but let me
> try, also.
>
> There is a *lot* of confusion about the words "end-to-end",
> and about "transport layers".
NH=> Agreed.  G.805 and G.809 func arch really helps sort this out...as
I hope was clear this week from Maarten's excellent analysis of the
ethernet OAM issues at SG13.

> There are lots of ends in this picture, and there are layers
> and there are layers.
NH=> Agreed.....just wish some more people grokked this stuff as it
should help mitigate arch violates.
>
> An 802.1Q VLAN bridge, and by extension, a P802.1ad provider
> bridge, does NOT supply a
> "transport layer" in the same sense as IP-over-Ethernet or
> Ethernet-over-IP.  This is
> because, at the data forwarding layers, VLAN tags are best
> though of as extensions to
> the address space, not as additional protocol layers.
NH=> Understood and agree.

>  If you
> want to call it a
> "transport layer", feel free!
NH=> No, that would be completely wrong....VLANs do *not* form a layer
network in any way, shape or form.

>  But don't assume that VLANs
> are equivalent to other
> "transport layers".  Data integrity, specifically, is
> different in bridges.
>
> Routers provide "best effort" not only in frame delivery, but
> in data integrity.  There
> is no upper limit specified for the probability of delivery
> of a packet with undetected
> errors.  IP's response, if you demand a particular level of
> data integrity, is that
> your application must supply its own checksum.
>
> Bridges, however, are not routers!
>
> Bridges *do* guarantee a level of service.  They manage to do
> this by guaranteeing the
> quality of each bridge, and by *demanding* the quality of
> each link.  Non-VLAN bridges,
> VLAN bridges, and Provider Bridges all make that guarantee to
> their users using this
> method.  The Customers of a Provider Bridge service are not
> an "802.1Q application"
> over an "802.1ad transport layer".  They are, in the 802.1
> model, just endstations
> attached to a bridge.  The Customers' data is just as well
> protected as an enterprise
> bridges' endstations, as long as the model is maintained.
> Adding an additional CRC
> to end-to-end service, where the ends are Customers attached
> to 802.1ad Provider
> Bridges, is therefore pointless.  Changing that model is as
> silly as turning the model
> for routing upside down.
NH=> I agree with all this....so not quite sure what you think I am not
understanding here.
>
> When a Pseudowire is used to connect two bridges,
NH=> Similar to VLANs, a PW is *not* a layer network either.  MPLS does
create layer networks however....but I suspect that is not well
understood by many.  One of my colleagues, who like Maarten is a real
expert in the func modelling detail, is formally describing MPLS right
now...but of course MPLS is a not a singular layer network, it has many
flavours and this work will clearly bring out the different arch
behaviours.

> the
> "end-to-end" is now just those
> two bridges.
NH=> Agreed.  If one drew out the proper func arch model one can see
this immediately.

> Those bridges are using that PW as an Ethernet
> to carry their traffic.
> The bridges demand that every wire be as reliable as an
> Ethernet.  That's the PW CRC
> issue.
NH=> Grokked totally. A PW is a form of adaptation function that lies
between a client (here we are discussing ethernet) and MPLS (or IP if
one is happy with a cnls server behaviour)...and then there are further
layer networks below this, eg the VC4 layer network of SDH, until we hit
the duct.  The fact that a PW is not a layer network and not really
necessary is neither here nor there....other than its just something
else to manage.  When the port on a bridge is not connected to an
'ethernet shared media' (orginal case), but some technology (any) then
one simply has to adapt the ethernet PDU (*including its FCS*, as its
part of the client ethernet CI) to the payload of the server layer.  One
should *not* be playing around with client layer network structures in a
simple client->server adaptation process at all.  So I am totally
against all the 'Martini' PW compression stuff....the BW squeezing is
simply not worth it (compared to other technolgies that will squander
BW). So !
 IMO the ethernet FCS is part of the ethernet's CI and should not be
removed by *any* server layer technology.
>
> -- Norm
>
> [email-masked] wrote:
> > Norm....As per our discussion today face-face.  I don't
> really see what the problem is here.  The client layer
> network should not care about the server layer network (and
> vice versa)....we need this layer independence for lots of
> reasons.  An ethernet traffic unit has a FCS.  This should
> simply be transported transparently in the payload area of
> whatever the server layer is....and the server layer should
> use its own (OAM) 'integrity mechanisms' irrespective of the
> client.  Problems arise when people start doing things like
> client layer compression and thus forming inter-layer dependencies.
> >
> > Further, as we agreed today and as I posted on an earlier
> mail to this thread, the ethernet FCS is a CRC that can be
> linerarly updated for adaptation function changes (eg VLAN ID
> change) on each link-connection (as provided by some server
> layer trail technology) whilst preserving its end-end
> intergrity check of the rest of the ethernet traffic unit
> between its trail termination points.
> >
> > regards, Neil
> >
> >
> >>-----Original Message-----
> >>From: Norman Finn [mailto:[email-masked]]
> >>Sent: 18 November 2003 08:56
> >>To: mark seery
> >>Cc: [email-masked]; [email-masked]
> >>Subject: Re: [PWE3] RE: [802.1] 802.1ad Customer FCS Retention
> >>
> >>
> >>Mark,
> >>
> >>You say below,
> >>
> >> > but a PW will be running over a physical media itself
> >>already, possibly
> >> > even SONET.............
> >>
> >>Exactly!  Here, the trail gets a bit murky.  If I'm in total
> >>control, and the PW
> >>is running over known Ethernet links, then I can get away
> >>without the CRC in the
> >>PW.  I have no problem whatsoever with the current definition
> >>of PWs without CRC.
> >>
> >>But, if I don't have tight control over the physical media
> >>over which the PW runs,
> >>then I really need a CRC on the PW for the bridge to be able
> >>to trust the "wire"
> >>and thus provide the same level of data integrity that
> >>bridges offer over physical
> >>Ethernets.  So, I claim that bridges need a CRC option on PWs.
> >>
> >>-- Norm
> >>
> >>mark seery wrote:
> >>
> >>>Norman Finn wrote:
> >>>
> >>>
> >>>><snip>
> >>>>In other words, routers pass about $0.93 of the data
> >>>
> >>integrity buck to
> >>
> >>>>bridges,
> >>>>and bridges pass about $0.75 of that to the data links.
> >>>
> >>The buck has
> >>
> >>>>to stop
> >>>>somewhere, and the data link is where it stops.  The PW
> >>>
> >>has to take
> >>
> >>>>the same
> >>>>hit as the physical media.
> >>>>
> >>>>-- Norm
> >>>
> >>>
> >>>but a PW will be running over a physical media itself
> >>
> >>already, possibly
> >>
> >>>even SONET.............
> >>>
> >>>mark
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>_______________________________________________
> >>pwe3 mailing list
> >>[email-masked]
> >>https://www1.ietf.org/mailman/listinfo/pwe3
> >>
> >
> >
>
>
>
> _______________________________________________
> pwe3 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/pwe3
>

---

From: "Mick Seaman" <[email-masked]>
Date: Tue, 25 Nov 2003 12:01:14 -0800

If you are not concerned about protecting the customer but merely locating
repetitively failed switch then OAM-like operations and test should be
sufficient. It remains the case that placing exactly the same error check
algorithm on the outer CRC as the inner does not extend your fault coverage
for the customer much. Switch faults will still be capable of bringing the
entire network to a halt and sending the wrong packets to the wrong
customers. Anyone who buys a network service from a provider with poor
internals on the grounds that the outer CRC will protect him is being simply
naive. I'll go further than that, if a salesman tries to sell me a service
with an outer CRC I will know that I can't trust the network and simply
should not buy.

Moreover as I have pointed out it is perfectly in the spirit of the 802.1
standards that the switch recalculate both the inner and the outer CRC, and
indeed where the format of the frame delievered to the custoomer changes
through the network it will have to recalculate both.

Mick

-----Original Message-----

Thanks,

Scott

-----Original Message-----

Mick

-----Original Message-----

Thanks,

Scott

-----Original Message-----

regards, Neil

---

From: Scott Kotrla <[email-masked]>
Date: Tue, 25 Nov 2003 14:55:37 -0600

I would not sell the customer a two FCS service.  I would use the second
FCS for fault detection.  Most customers would have no clue this is
going on.  The second FCS protects against undetected internal switch
malfunctions.  How likely this increases fault detection is directly
related to how likely switches are to unknowingly corrupt packets.

I am still not exactly clear on your argument.  Are you saying that the
second FCS is extremely unlikely to catch an undetected internal switch
error, or are you saying that undetected internal switch errors are
extremely unlikely?

I'm expecting the path of least resistance here.  While it is possible
to build boxes that purposefully try to break the rules, I wouldn't
expect this to be the norm, since it would be more difficult, especially
for switches in the middle which aren't expected to act on or be aware
of the internal customer VLAN/FCS.  In a MEF TRANS function, the
customer Ethernet frame (including Customer VLAN tag) will never change.
All changes will occur to the Service Tag, which is why a Service FCS
cannot be relied upon for end-to-end error detection.

If switches can do whatever they want internally, I'd say that trying to
enforce a more complex incremental CRC32 calculation will be less
enforceable than dual FCS.

I have to admit, assuming everything will always work the way it is
supposed to would make my life a lot simpler.  If a customer calls and
says service is not working, we'll just say that our network is not
capable of malfunctioning and leave it at that.  

Thanks,

Scott

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Mick Seaman
Sent: Tuesday, November 25, 2003 2:01 PM
To: 'Scott Kotrla'; [email-masked]; [email-masked]
Subject: RE: [PWE3] RE: [802.1] 802.1ad Customer FCS Retention

If you are not concerned about protecting the customer but merely
locating
repetitively failed switch then OAM-like operations and test should be
sufficient. It remains the case that placing exactly the same error
check
algorithm on the outer CRC as the inner does not extend your fault
coverage
for the customer much. Switch faults will still be capable of bringing
the
entire network to a halt and sending the wrong packets to the wrong
customers. Anyone who buys a network service from a provider with poor
internals on the grounds that the outer CRC will protect him is being
simply
naive. I'll go further than that, if a salesman tries to sell me a
service
with an outer CRC I will know that I can't trust the network and simply
should not buy.

Moreover as I have pointed out it is perfectly in the spirit of the
802.1
standards that the switch recalculate both the inner and the outer CRC,
and
indeed where the format of the frame delievered to the custoomer changes
through the network it will have to recalculate both.

Mick

-----Original Message-----

Thanks,

Scott

-----Original Message-----

Mick

-----Original Message-----

Thanks,

Scott

-----Original Message-----

regards, Neil

---

From: Ali Sajassi <[email-masked]>
Date: Tue, 25 Nov 2003 18:24:20 -0800

I ask the question of service applicability for FCS retention some time ago 
and I didn't get a clear answer so let me ask again. When talking about FCS 
retention, which I of the following four Ethernet services do you have in 
mind ?

1) port-based point-to-point Ethernet service
2) vlan-based point-to-point Ethernet service
3) port-based Ethernet bridge service (multipoint)
4) vlan-based Ethernet bridge service (multipoint)

In vlan-based service, the customer vlan is used as service delimiter and 
is NOT the same at the two ends of the service - e.g., just like frame 
relay or ATM where DLCI or VPI/VCI are not the same at two ends. Requiring 
that the VLANs be the same at the two ends, imposes unnecessary restriction 
in customer provisioning model. Therefore, FCS retention is not viable for 
applications where customer VLAN is used as service delimiter.

Also any bridge service (port-based or vlan-based) doesn't require FCS 
retention by the definition of the bridge service itself.

So, the only applicable service for this FCS retention stuff (and I doubt 
viability of it), is for port-based point-to-point service. However, I 
would like to ask the following question:

How do you catch the errors introduced by the switches for the other 
services ??? Given that the same set of networking gears is to support all 
four Ethernet services, are you suggesting that we only need to solve the 
problem for a single service and not to worry about other service type !!! 
Or are you suggesting that you only offer a single service and thus you 
don't have to worry about other services !!!

As I mentioned previously, a better approach is to solve the problem for 
all the Ethernet services - whether it is through OAM mechanisms or through 
internal switch check-sum or a combination of two or whatever. But the 
point is we should try to address the issue for ALL service types.

-Ali