Subject: FCS retention in PWE3

From: Stewart Bryant <[email-masked]>
Date: Fri, 21 Nov 2003 16:24:10 +0000

Plan of Record
--------------

The plan of record in PWE3 with respect to FCS retention is that we
would change the requirements draft to optionally permit the retention
of FCS (currently is says must remove FCS), leave the base drafts
as they are (they say remove FCS), and test the consensus of the WG
for a new draft that calls up the optional retention of FCS for some
encapsulations.

Consensus
---------

Danny and I have reviewed the discussion on the list and agree that
there is consensus for the new draft, and that we should proceed with
the plan of record.

We hence ask Andy to resubmit (draft-malis-pwe3-fcs-retention-00.txt)
as a WG draft.

Applicability statement
----------------------

Looking at the discussion so far, I think that we need to take great
care in our drafting of the applicability statement in the FCS
retention draft.

To quote section 7 of our requirements draft:

"An emulated service SHOULD be as similar to the native service as
possible, but NOT REQUIRED to be identical. The applicability
statement of a PWE3 service MUST report limitations of the emulated
service."

This means that our WG goal is to provide the necessary and
sufficient degree of emulation, not perfection. In the specific
case of Ethernet FCS retention, I would take this to mean that
we must provide the necessary degree of faithfulness required
to support the assumptions made in IEEE 802.1.

I have seen three cases made for FCS retention in the mails to
the list.

1) IEEE 802.1 assumptions
2) OAM
3) Data integrity

These need to be addressed in FCS retention draft applicability
statement.

IEEE 802.1 assumptions
----------------------

This is best summarized by the notes from Norman Finn:

"PW is just another medium to a bridge.  However, *all* media
known to bridges, at least as far as IEEE 802.1 are concerned,
are 802 media, and all 802 media have 32-bit CRCs.  If you connect
a medium that doesn't have at least a 32-bit CRC to a bridge,
then the resultant bridged LAN will suffer a significant
degradation in its ability to transport frames without error,
because bridges depend on reliable media.  And then the routers
on top of the bridges don't work so well."

and

"If I'm in total control, and the PW is running over known
Ethernet links, then I can get away without the CRC in the PW.
I have no problem whatsoever with the current definition of PWs
without CRC.

"But, if I don't have tight control over the physical media over
which the PW runs, then I really need a CRC on the PW for the
bridge to be able to trust the "wire" and thus provide the same
level of data integrity that bridges offer over physical
Ethernets.  So, I claim that bridges need a CRC option on PWs."

That is to say PWE3 provides a service to 802.1 and needs by one
method or another to provide the necessary and sufficient
degree of wire emulation such that their design assumptions hold.

OAM
---

The original reason that FCS retention raised was because of an
OAM issue. A malfunctioning bridge caused a network error, and
it was thought that FCS retention would be useful in generating
an alarm and diagnosing the fault. PWE3 is also developing an
OAM method based on VCCV. We need to understand those
circumstances where FCS retention is the best OAM technique,
and those where VCCV is the better approach.

Data integrity
--------------

Some folks want FCS retention as a means of assuring the data
integrity of the PW service. The degree of enhancement in data
integrity when using FCS retention, compared to the base drafts
is something that we need to treat carefully, because CRC32
offers a relatively weak degree of data assurance (4 or 5 bit
corruption). The IETF standard method of ensuring data
integrity is IPSec, and IPSec is called up in the security
section of the PWE3 architecture for just this purpose. 802.1
are planning their own method of enhanced data assurance which
we would pass this transparently over the PW. Within the PWE3
context the IETF method of providing a similar degree of
data assurance would be to run IPSec between the PEs. We
therefore need to describe the improvement in data integrity
achieved through FCS retention and compare this to the
improvements achieved through the use of IPSec.

Regards

Stewart

---

From: [email-masked]
Date: Fri, 21 Nov 2003 18:14:08 -0000

Hi Stewart,

With all due respect I really don't see what all the fuss is about here.  If the PW adaptation function did not mess about with the clients there would be no problems at all.  There should be exceedingly good reasons why any server layer technology should modify a client layer network's CI (characteristic information)....I am afraid BW squeezing is not a good reason IMO.  So the default behaviour in all client->server adaptation functions (because that is what a PW is doing) should be *do not change the client layer traffic unit at all*.

This is not the only area where problems will arise.  Like-mode technology service I/W is another area where I can see that problems will arise due to the very fact PWs even exist....as, in principle, they are not serving any 'must-have' architectural technical purpose but (IMO) simply a political one.  However, if folks steadfastly refuse to see that MPLS is not the same as IP and both create layer networks (one belonging to the co-ps mode and one the cnls mode) then I am afraid we are never going to get this sorted properly.

regards, Neil

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Stewart Bryant
> Sent: 21 November 2003 16:24
> To: [email-masked]
> Cc: [email-masked]; Mick Seaman; [email-masked]; Norman Finn
> (nfinn)
> Subject: [PWE3] FCS retention in PWE3
> 
> 
> 
> Plan of Record
> --------------
> 
> _______________________________________________
> pwe3 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/pwe3
>

---

From: Eric Rosen <[email-masked]>
Date: Mon, 24 Nov 2003 14:26:58 -0500

Mark> --|a-CE-b|--GbE--|PE|--PW--|PE|--GbE--|c-CE-d|--

Mark> wouldn't ports  "b" and "c" be  able to detect FCS  errors before they
Mark> entered the customers network, and at least provide an indication that
Mark> the  problem  is in  the  operators  network  and not  the  customers,
Mark> therefore  narrowing  down  the  problem of  diagnosing  TCP  checksum
Mark> errors?  

I don't  think so.   Presumably the frame  is ethernet-checksummed  from its
source to  the ingress PE, and  then is ethernet-checksummed  again from the
egress  PE   to  its   destination.   The  only   place  where  it   is  not
ethernet-checksummed is in the SP network.  So if the frame actually arrives
at the destination host with a  bad TCP checksum, chances seem high that the
problem was in the operator's network.

---

From: mark seery <[email-masked]>
Date: Mon, 24 Nov 2003 15:58:47 -0500 (EST)

Hi Eric,

---- Eric Rosen <[email-masked]> wrote:
> <snip> In  my interpretation of  Mark's example,  there's an  FCS carried  from the
> source host  to the operator's  network (native ethernet service),  and then
> another  FCS carried  from the  operator's network  to the  destination host
> (native ethernet service), so corruption  in the customer's network would be
> caught by the FCS. 

My diagram could be interpreted in light of the optional proposal on the table (Mallis et el.) in which case the first part of the above is correct; alternatively it could be interpreted in the context of the general conversation in which case the FCS is retained, copied to the PW, copied to the egress. The diagram could have been clearer. The point is, how does the SP determine (including historically and with trending) where errors are occurring - seems you are headed down the OAM route - which brings  up the issue of an e2e OAM mechanism.

Errors could also be occurring within the customer equipment on the egress side of the CE, a check at CE would be useful in that case. If as Mick states a CRC-32 is not useful, then that's a whole nother ball game.

-mark

---

From: Keith Knightson <[email-masked]>
Date: Mon, 24 Nov 2003 17:59:35 -0500

Guys,

Let's set aside specious architectural arguments for  a moment.

Let's say that the objective is to make the CE1 - CE2 path emulate a 
piece of wire, even though in reality it contains a CE1-PE1-PE2-CE2 
intervening section?

Then and only then would there be a requirement to at least consider 
the problem.

If this was the objective, it seems to me that there is no practical 
solution. Because.....

First we need to capture the FCS at the ingress of PE1 (assuming it 
checks ok, if not discard).  I am not sure that FCS can be kept.   In 
my experience, when a frame is being received the FCS also goes 
through the hardware polynomial division process to get a zero result 
if correct.  There's a piece of math thats shows that Frame plus the 
received FCS all divided by the ploynomial goes to zero. The result 
is binary, good or bad.  This avoids the need for a two step process. 
So there is never a residual FCS to capture. Thus, we may fall at the 
very first hurdle.  Can someone check this point?

Anyway let's suppose the original FCS can be captured and can be 
associated in some way with the frame, by pre-pending, appending, 
whatever.  BTW, a recalculation from the payload will not fit the 
bill, since it could be corrupted in memory of PE1 prior to 
recalculation, which presumably would have to be a software process, 
since there are no outgoing ethernet hardware ports on the PW side of 
the PE1.

Ok, so now payload plus the original FCS is then PW'ed, in IP, MPLS, whatever.

At the ingress to PE2, the payload + plus its FCS would be dumped in 
memory. This could be checked by software, it seems to me it would 
have to be, unless PE2 can somehow transmit the frame to get a new 
FCS derived from payload (minus the original FCS) to itself in order 
that received FCS is still equal to the one stored in memory. This 
would allow PE2 to discard a frame corrupted over the PW. this is 
again conditional on being able to capture FCS from receiving 
hardware.

But now we still have to ensure that PE2 itself does corrupt the 
payload before transmitting to CE2.

We have an outgoing ethernet port on PE2 to CE2, with its autonomous 
sending hardware with its autonomous "on the fly" FCS addition.  As 
far as I know (which admittedly may not be very far) when stuff is 
queued for transmitting that's the end of it. It just goes all by 
itself. The FCS is being calculated as it goes out on the wire and is 
automatically slammed on the end. How on earth is this "on the fly" 
FCS going to be compared to the original FCS still lying about in 
memory somewhere?  I do not believe the FCS is retained after 
transmission, and even if it could it's too late. The frame has 
already gone with the new FCS!

Why is the hardware designed like this? Well, because it has been 
architected that way, it conforms to the IEEE architecture.  So in 
the end, the IEEE original architecture does count, and it is not 
surprising considering the definition of its layering.

It seems to me that the objective is in fact unobtainable given the 
hardware at our disposal.

BTW, when people say "this is an architectural violation' or the 
"architecture requires it", what they usually mean is one or both of 
two things:

a)	my idea of what an architecture is, is better than yours, and
b)	I can't find any other proper justification for what I want 
to you to do (or not do)

I think those who want this service have a duty to explain how this 
is all going to work. I may have missed something vital in my rant, 
and am willing to eat whatever crow is necessary, but I just don't 
see how to do it.

Regards

Keith

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 25 Nov 2003 10:48:47 +0000

Nick DelRegno wrote:

> The problem is that noone will ever see the TCP checksum reported!  What if
> it is UDP and checksum is disabled?
> 
> Please educate me as to how, as a SP, I can detect TCP checksum errors in my
> network, IP, Ethernet or otherwise.
> 
> How often, when a client station is experiencing Internet connectivity
> issues, does Support say, "Please tell me what your TCP Checksum counter
> says," or better yet, Tech Support calls the end user to tell them "Our
> network management system indicates an unusually high number of TCP Checksum
> errors being reported by several end stations in our network, yours being
> one." ??  In my exerperience, never.
> 

But on the other hand you have not had a decent OAM protocol running in
background in the past.

Stewart

---

From: [email-masked]
Date: Tue, 25 Nov 2003 12:41:57 -0000

Stwart,

In general I agree with you.  But what you are actually saying here is that we want independent OAM in each layer network (simply has to be independent....one reason being that we want the *same* OAM solution in the server irrespective of its client).  But please note this:
-	a PW is not a layer network, ie we do not network/switch PW entities;
-	an LSP is a layer network, ie we do network/switch LSP entities;
-	why can't we use Y.1711 OAM on the top level p2p LSP?
-	Y.1711 is sparse/simple/complete, ie all defects are defined with entry-exit criteria and consequent actions.....it even defines availability and does so in order it can be used a single-ended way, ie behaviour of both directions can be observed from 1 end.
-	MPLS and ATM OAM layer I/W is already defined in Y.1712.....and again its really simple.

None of this is done (and in some cases is not appropriate) for the VCCV approach.    

So what is wrong with this solution?

regards, Neil
> Sent: 25 November 2003 10:47
> To: Scott Kotrla
> Cc: [email-masked]; 'mark seery'; [email-masked]
> Subject: Re: [PWE3] FCS retention in PWE3
> 
> 
> 
> A question that I have raised before, and called up in my
> applicability statement notes is whether VCCV (or similar)
> allows us to diagnose the problem more accurately.
> 
> VCCV run between the PEs will allow the SP to audit the PW's
> path over the PSN. The only element not covered is the
> circuit between the CE and the PE, and the different
> internal data path between the VCCV generator/receiver
> and the PE-PSN interface. If the customer also runs an
> OAM protocol, they can audit from CE to CE and validate
> the whole SP PW path.
> 
> Unlike the FCS approach the OAM could be signed with a
> decent crypto and would not be fooled by short error bursts.
> 
> If FCS is the right tool for the path audit we should use it,
> but I am not convinced that it will do better than a
> purpose designed OAM protocol.
> 
> Stewart
> 
> 
> Scott Kotrla wrote:
> 
> > Why couldn't the corruption be occurring in the customer's 
> network?  How
> > are the provider and customer supposed to figure out whose 
> fault it is
> > with certainty?  Because all of the links are checksummed, 
> the error had
> > to have occurred inside a device. But there is NO way to 
> isolate whether
> > it was a provider device or a customer device.  There are 
> likely more
> > customer devices in the path than carrier devices, so it 
> would seem more
> > likely to be a customer problem.  
> > 
> > Is it not clear that this is a huge nightmare for a service 
> provider who
> > deploys this in a large scale?  I just don't get how people 
> cannot see
> > how this could be useful in some SP networks.  The blanket statement
> > that it would NEVER be useful and shouldn't be an OPTION is absurd.
> > 
> > From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Eric
> > Rosen
> > Sent: Monday, November 24, 2003 1:27 PM
> > To: mark seery
> > Cc: [email-masked]
> > Subject: Re: [PWE3] FCS retention in PWE3
> > 
> > Mark> wouldn't ports  "b" and "c" be  able to detect FCS  
> errors before
> > they
> > Mark> entered the customers network, and at least provide 
> an indication
> > that
> > I don't  think so.   Presumably the frame  is 
> ethernet-checksummed  from
> > its
> > source to  the ingress PE, and  then is 
> ethernet-checksummed  again from
> > the
> > egress  PE   to  its   destination.   The  only   place  
> where  it   is
> > not
> > at the destination host with a  bad TCP checksum, chances 
> seem high that
> > the

---

From: "David Allan" <[email-masked]>
Date: Tue, 25 Nov 2003 10:08:14 -0500

Stewart:

I think there is an aspect of your assertions that I am missing. CE to CE
OAM tools are not in control of the provider and IMO the provider asking the
customer to diagnose the service is a non-starter. PE to PE OAM tools have
incomplete coverage in the PE and as the whole ECMP discussion highlights,
is not payload and therefore may not be subject to exactly the same transfer
functions. PW OAM tools are typically single hop at the PW level so IMO are
rather limited in utility.

Key delta here is detection vs. diagnosis. FCS offers detection, and if done
right can also provide a useful degree of fault isolation (via ingress and
egress checking). It is not clear to me that an OAM protocol helps the
detection aspects in any useful timeframe if the nature of the problem is
inherently masked. I also am not grokking the reference to the use of crypto
esp. in the context of diagnosis once we've assumed a problem exists....

What we are really discussing is the economics of service integrity, a.k.a.
is the problem sufficently rare that only diagnosis is required, and
therefore it can be punted into reactive OAM tools and we can save that
1/8th sq. inch of silicon in the forwarding plane. Personally I think there
is a lot of value in a service that works, and self identifies when it is
broken. Ruthless optimization of the forwarding plane on the assumption that
it just works really complicates life when it doesn't work....

cheers
Dave

> From: Stewart Bryant [mailto:[email-masked]] 
> Sent: Tuesday, November 25, 2003 5:47 AM
> To: Scott Kotrla
> circuit between the CE and the PE, and the different internal 
> data path between the VCCV generator/receiver and the PE-PSN 
> interface. If the customer also runs an OAM protocol, they 
> can audit from CE to CE and validate the whole SP PW path.
> 
> but I am not convinced that it will do better than a purpose 
> designed OAM protocol.
> 
> Stewart
> 
> 
> Scott Kotrla wrote:
> 
> network?  
> > How are the provider and customer supposed to figure out 
> whose fault 
> > it is with certainty?  Because all of the links are 
> checksummed, the 
> > error had to have occurred inside a device. But there is NO way to 
> > isolate whether it was a provider device or a customer 
> device.  There 
> > are likely more customer devices in the path than carrier 
> devices, so 
> > it would seem more likely to be a customer problem.
> > 
> provider 
> > who deploys this in a large scale?  I just don't get how 
> people cannot 
> > see how this could be useful in some SP networks.  The blanket 
> > statement that it would NEVER be useful and shouldn't be an 
> OPTION is 
> > absurd.
> > 
> > From: [email-masked] [mailto:[email-masked]] On Behalf Of 
> > Eric Rosen
> > Mark> wouldn't ports  "b" and "c" be  able to detect FCS  errors 
> > Mark> before
> > they
> > Mark> entered the customers network, and at least provide an 
> > Mark> indication
> > that
> > ethernet-checksummed is in the SP network.  So if the frame 
> actually 
> > arrives at the destination host with a  bad TCP checksum, 
> chances seem 
> > high that the

---

From: Scott Kotrla <[email-masked]>
Date: Tue, 25 Nov 2003 10:35:43 -0600

I have no problems requiring that optional FCS retention implementations
do an egress FCS check.

I would much rather have a dropped frame than a corrupt frame that looks
valid.  Customers that aren't receiving frames will know that they have
a problem in the provider network.  Customers that are receiving a bunch
of frames that appear to be valid will not know that they are having a
problem with their service.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Eric
Rosen
Sent: Tuesday, November 25, 2003 9:34 AM
To: David Allan
Cc: [email-masked]'; Scott Kotrla; 'mark seery'; [email-masked]
Subject: Re: [PWE3] FCS retention in PWE3

> FCS offers detection

I don't really see  any basis for this statement.  If what  we are
trying to
detect is a malfunctioning switch, then we need to consider the various
ways
in which a switch may malfunction before we can state that FCS enables
us to
detect the malfunctions. 

For instance,  a switch  may malfunction by  discarding frames
entirely, or
perhaps by intermittently discarding bursts of frames.  FCS would not
detect
this  sort of  malfunction.  Given  the way  switches work,  this is  a
more
likely malfunction than overwriting the frame data.  

In  the particular  case  of PW,  a  switch may  malfunction by
incorrectly
manipulating  the PSN header  during transit.   This is  a much  more
likely
malfunction than overwriting the frame data.  Carrying the customer's
FCS to
the remote edge won't detect this at all. 

FCS may be  good for detecting burst errors on a  transmission medium,
but I
see no reason to believe that  it has a high probability of detecting
errors
that might be caused by malfunctioning switches. 

> CE to CE OAM tools are not in control of the provider and IMO the
provider
> asking the customer to diagnose the service is a non-starter. 

I agree with this, but note that with the usual ethernet switching
equipment
it would be the  egress CE that would detect any FCS  failure that is
caused
by problems in the provider network.   So any specification of an option
for
carrying the customer's  FCS should REQUIRE that the  egress PE validate
the
FCS.

---

From: mark seery <[email-masked]>
Date: Tue, 25 Nov 2003 10:02:33 -0800

Thomas D. Nadeau wrote:

> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of 
> *David Allan
> *Sent:* Tuesday, November 25, 2003 10:08 AM
> *To:* [email-masked]'; Scott Kotrla
> *Cc:* [email-masked]; 'mark seery'; [email-masked]
> *Subject:* RE: [PWE3] FCS retention in PWE3
>
>  > Stewart:
>  >I think there is an aspect of your assertions that I am missing. CE 
> to CE OAM tools are not in control  
>  >of the provider and IMO the provider asking the customer to diagnose 
> the service is a non-starter.  
>  
>       It is my understanding that at least several providers today
> have mechanisms that they provide to customers that allow them
> to test the PE-CE link from the PE if they suspect there is a
> problem (or check the link's status from the PE side), so I
> am not sure it is not such a "non-starter" as you characterize it. 
> For example, in the L3 VPN case, I have seen
> end-customers (i.e.: those that buy the L3 VPN service) simply ping  
>  (using ICMP IP) betweeen the CEs  to test connectivity and/or
> verify SLA on their own. When this fails, they can either call the 
> provider or  
>  try the SPs web-based PE testing tool  which might further isolate 
> the problem
>  using LSP ping and other things within the core, as well as 
> to to  ping the PE-CE.
> The only difference would be the inclusion of VCCV/LSP ping /trace on the
> core side.
>  
>     --Tom 

Tom,

I agree that is a useful diagnostic test.

A few thoughts:
-Service providers though need to know about a problem before their 
customer knows about it (or at least at the same time)
-Intermittent problems are often hard to pin down, so historical 
information is useful.
-Historical information can also be used as trending data to catch a 
problem before it gets too serious.
-Many L3 VPNs run over a TDM circuit that has a path check, Scott is 
trying to move to Ethernet/dark fiber where there is no such capability.
-There is a certain dynamic that is created when there is a financial 
transaction between a provider and a customer. Finger-pointing is 
inevitable, and there are implications WRT SLAs and credits. Most of the 
U.S. ILECs do not offer SLAs on their fiber Ethernet services today - 
would be nice to get beyond that (even if this is just one piece of the 
puzzle).

mark

---

From: "David Allan" <[email-masked]>
Date: Tue, 25 Nov 2003 17:30:31 -0500

>I think there is an aspect of your assertions that I am missing. CE to CE
OAM tools are not in control  
 >of the provider and IMO the provider asking the customer to diagnose the
service is a non-starter.  

This assumes that ping replicates the circumstances of the problem such that
ping working is reasonably authoritative indication that the problem is not
in the CE or the PE-CE link. 
 
cheers
Dave

---

From: "David Allan" <[email-masked]>
Date: Tue, 25 Nov 2003 17:32:16 -0500

Eric:

Comments in line..  
 
> > FCS offers detection
> 
> I don't really see  any basis for this statement.  If what  
> we are trying to detect is a malfunctioning switch, then we 
> need to consider the various ways in which a switch may 
> malfunction before we can state that FCS enables us to detect 
> the malfunctions. 

FCS is not a universal panacea, however if a misbehaving switch is
corrupting information and offering it up as good, it can be detected and
will be detected as part of normal operations.

> For instance,  a switch  may malfunction by  discarding 
> frames  entirely, or perhaps by intermittently discarding 
> bursts of frames.  FCS would not detect this  sort of  
> malfunction.  Given  the way  switches work,  this is  a more 

IMO mechanisms exist (e.g. sequence numbers, packet/octet counts) to find
discard (which may also be a result of congestion).
> 
> In  the particular  case  of PW,  a  switch may  malfunction 
> by  incorrectly
> manipulating  the PSN header  during transit.   This is  a 
> much  more likely
> malfunction than overwriting the frame data.  Carrying the 
> customer's FCS to the remote edge won't detect this at all. 

No but that would manifest itself in other forms, such as sequencing
problems etc.

> FCS may be  good for detecting burst errors on a  
> transmission medium, but I see no reason to believe that  it 
> has a high probability of detecting errors that might be 
> caused by malfunctioning switches. 
> 
> > provider asking the customer to diagnose the service is a 
> non-starter.
> 
> I agree with this, but note that with the usual ethernet 
> switching equipment it would be the  egress CE that would 
> detect any FCS  failure that is caused
> by problems in the provider network.   So any specification 
> of an option for
> carrying the customer's  FCS should REQUIRE that the  egress 
> PE validate the FCS. 

That certainly would be a BCP.

Dave

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Tue, 25 Nov 2003 19:31:28 -0500

>Behalf Of David Allan
>Sent: Tuesday, November 25, 2003 5:31 PM
>To: [email-masked]'; [email-masked]; 'Scott Kotrla'
>Subject: RE: [PWE3] FCS retention in PWE3
>
>
>Hi Tom:
>
>This assumes that ping replicates the circumstances of the 
>problem such that ping working is reasonably authoritative 
>indication that the problem is not in the CE or the PE-CE link. 

	Well, for the real networks I am refering to, it
seems to be an authoritative enough indication, so I 
don't see why it isn't in this case. :P

	--Tom