Subject: pwe3-ethernet-encap - FCS...

From: "Christopher N. \"Nick\" DelRegno" <[email-masked]>
Date: Wed, 16 Apr 2003 23:29:07 -0500

Beware, can of worms opening.

>From the original "Martini" draft to today, the customer FCS data has
been discarded.  Initially, my interest was in the "tagged" mode more
than the "port" or "raw" mode of encapsulation.  I noted that the FCS
was discarded in order to save 4 bytes as well as the fact that many
deployed implementations discard the FCS today.  Until recently, we were
okay with this.

In testing a very widely deployed Ethernet switch, we encountered an
anomaly which, if the FCS is discarded, would be a nightmare to
troubleshoot in a live network.  Briefly, while we weren't seeing any
packet loss across a fairly large Ethernet network and every packet was
being received by the destination host, all higher-layer frames were
being dropped by the host.  After spending an inordinate amount of time
using traditional Ethernet performance metrics to attempt to isolate the
problem, not until we intrusively placed Ethernet test equipment on both
ends did we discover that while the Ethernet FCS of every delivered
packet was good, the customer data was corrupt.  Interestingly it was
only in one direction.

Apparently one of the Ethernet switches in the network was internally
corrupting the data between the discarding of the customer FCS and the
calculation of a new FCS.  Mind you, there were no VLAN tag swaps or
stacks or the like.  This was a simple bridged connection, where the
Ethernet packet in theory shouldn't change, that was consistently
corrupting the data.

Now, had the FCS been maintained throughout the data path, the
customer's network-facing Ethernet switch would have dropped every
corrupted packet due to the FCS error.

In the pwe3-ethernet-encap draft as it stands, there is no provision for
keeping the received FCS.  In tagged mode, this is understandable since
I very well may change the VLAN ID at the far end of the network.
However, in the Ethernet mode, where I am sending ALL of the customer's
frames, without looking at the VLAN ID, I need the ability to maintain
the customer's FCS as well.

I realize that A) this is not how the draft is written, B) that if the
FCS is optional, then pwe3-control-protocol will have to be updated as
well to signal the presence or lack of the FCS (possibly via the
Interface Parameters field) and C) that many popular techniques to
stacked tags don't retain the customer FCS.  However, the cost of the
additional 4 bytes per packet and extra signaling information is worth
it.  When providing Ethernet private line type services, we have to rely
on the Layer 2 information to determine the performance of the network.
Maintaining the customer's FCS, resulting in a "Path FCS", gives the SP
an end-to-end error check.

In the truest sense of the term pseudo-wire, we are using a PSN to
emulate an Ethernet-cable.  It would be unacceptable if an Ethernet
cable somehow corrupted the payload of the MAC frames without the FCS
being able to detect it.  It is just as unacceptable if that Ethernet
cable is composed of packet switched networks instead of twisted pairs.

Admittedly, if such internal errors occur in network devices, having a
"Path FCS" will not tell me which switch is errant.  However, it will
tell me a problem exists, hopefully prior to my customer's contacting
me.  

Regards,

Nick DelRegno

---

From: Akiva Sadovski <[email-masked]>
Date: Fri, 18 Apr 2003 12:57:27 +0200

Hi, Nick,

  As i understand, your point is that in Ethernet-over-PSN PW some malicious
switch/router can corrupt the payload; in that case some layer 2 FCS of the
PW packets will be unable to detect these corrupted packets. Is that so ?
  Note, that exactly the same problem may exist in the FrameRelav-over-PSN
encapsulation, where FCS is stripped by the PE device. But in case of
FR-over-PSN the solution will be complicated by the fact that the FCS must
be carried along with bit stuffing, and bit stuffing presence complicate
DLCI analysis etc...

Thanks,
  Akiva Sadovski

-----Original Message-----
From: Christopher N. "Nick" DelRegno
To: [email-masked]
Sent: 17/04/2003 06:29
Subject: [PWE3] pwe3-ethernet-encap - FCS...

Beware, can of worms opening...

Regards,

Nick DelRegno

---

From: Eric Rosen <[email-masked]>
Date: Fri, 18 Apr 2003 14:48:08 -0400

Nick> Apparently one of the Ethernet  switches in the network was internally
Nick> corrupting the data between the discarding of the customer FCS and the
Nick> calculation of a new FCS. 

Well, this is  a classic; you wouldn't believe how many  IP routers I've run
across that  modify the header and  then recompute the  header checksum from
scratch. 

Generally though a  box that corrupts the data internally  would not make it
past beta  test.  Further, in  routed networks, if  a box is  corrupting the
data,  there is  no  detecting  it until  you  reach the  TCP  layer at  the
destination host. 

Nick> What I am asking is that we  have an option to keep or discard the FCS
Nick> and signal accordingly. 

I  don't think  that's unreasonable,  but would  like to  see more  input on
whether  it is  worthwhile  to  have the  network  protect against  "in-box"
corruption (as  opposed to "on wire"  corruption) in the  special case where
the packets being carried are PWE3 ethernet packets.

---

From: "Christopher N. \"Nick\" DelRegno" <[email-masked]>
Date: Fri, 18 Apr 2003 14:07:49 -0500

Eric:

I thought of the IP analogy as well and agree that such corruption is
likely very rare.

Regarding the problematic switch, it was only one faulty box out of
several good one of the same make, model and software rev.  Clearly it
didn't act this way in QC.  In some newer designs we have seen vendors
implementing a sanity check on the payload content prior to
recalculating the FCS.  This would alleviate the situation, but is not
universal.

I am not asking the network to protect against "in-box" corruption.  I
am asking the network to not arbitrarily take away the protection which
is already built in.

Thanks,
Nick

-----Original Message-----
From: Eric Rosen [mailto:[email-masked]] 
Sent: Friday, April 18, 2003 1:48 PM
To: Christopher N. "Nick" DelRegno
Cc: [email-masked]
Subject: Re: [PWE3] pwe3-ethernet-encap - FCS...

Nick> Apparently one of the Ethernet  switches in the network was
internally
Nick> corrupting the data between the discarding of the customer FCS and
the
Nick> calculation of a new FCS. 

Well, this is  a classic; you wouldn't believe how many  IP routers I've
run
across that  modify the header and  then recompute the  header checksum
from
scratch. 

Generally though a  box that corrupts the data internally  would not
make it
past beta  test.  Further, in  routed networks, if  a box is  corrupting
the
data,  there is  no  detecting  it until  you  reach the  TCP  layer at
the
destination host. 

Nick> What I am asking is that we  have an option to keep or discard the
FCS
Nick> and signal accordingly. 

I  don't think  that's unreasonable,  but would  like to  see more
input on
whether  it is  worthwhile  to  have the  network  protect against
"in-box"
corruption (as  opposed to "on wire"  corruption) in the  special case
where
the packets being carried are PWE3 ethernet packets.

---

From: Peter Willis <[email-masked]>
Date: Tue, 22 Apr 2003 11:00:59 +0100

> I thought of the IP analogy as well and agree that such corruption is

Reminds me that some vendors used ECC memory & some didn't. 

> didn't act this way in QC.

I've seen similar faults as well.

> am asking the network to not arbitrarily take away the protection which

It is also essential to be able to locate the faulty box.

Remember boxes break and when one has a large network even "very rare" faults 
happen. And the "rarer" the fault the harder it is to find. Given the choice 
between a dozen common faults or 1 rare one I'd pick the dozen common faults.

So consider that my input on whether it is worthwhile to have the network 
protect against "in-box" corruption.

Peter.