Subject: RE: last call on control and ethernet drafts

From: [email-masked]
Date: Mon, 10 Nov 2003 07:54:27 -0000

I have checked with my colleagures here and BT do not agree with this view as a general architectural principle.  When dealing with simple client/server encapsulation (between any arbitrary pair of technologies) the general maxim should be to leave the client layer PDU structure alone to ensure layer independence.....so this is *any* change of the client layer, which of course also includes any compression behaviour. 

regards, Neil 

> -----Original Message-----
> From: Ali Sajassi [mailto:[email-masked]]
> Sent: 08 November 2003 03:33
> To: Luca Martini; [email-masked]
> Cc: [email-masked]; Danny McPherson
> Subject: Re: [PWE3] last call on control and ethernet drafts
> 
> 
> 
> Given that I won't be able to attend the IETF meeting this 
> time because of 
> scheduling conflict, I would like to restate my position in 
> favor of FCS 
> omission.
> 
> The main point of passing customer FCS is to be able to detect 
> errors/corruptions that may occur inside the PE/P nodes - 
> e.g., to resemble 
> a characteristic similar to a wire. FCS pass-through is one 
> way to achieve 
> this. However, it only works for EPL service and does NOT 
> work for EVPL, 
> EPLAN, and EVPLAN. A better way of addressing this issue is 
> to ensure that 
> the packets are properly protected every step of the way 
> (both inside the 
> wire between the nodes as well as through the PE/P nodes). 
> This way, we 
> ensure that we have a reliable transport mechanism that not 
> only works for 
> EPL, but also it works for other Ethernet services such as 
> EVPL, EPLAN, and 
> EVPLAN. Therefore, I am in favor of addressing this issue 
> fundamentally for 
> all Ethernet services and not just a patch work for EPL.
> 
> -Ali
> 
> At 02:19 PM 11/7/2003 -0700, Luca Martini wrote:
> >yes, i was on vacation as well.
> >
> >I have to support the omission of the FCS from the PWE3 
> encapsulation as 
> >that fit quite well with operations that IP routers have to 
> perform. ( to 
> >route IP the FCS of layer2 is always stripped and 
> regenerated ). Because 
> >of this many implementations have the FCS function in 
> hardware at the port 
> >level,which makes it quite easy to implement the current PWE3 
> >design.  This why we went down this path in the first place. 
> Today there 
> >are many implementations of the PWE3 encapsulation ( 
> draft-martini version 
> >) that are deployed and generating revenue for service 
> providers. I have 
> >not heard of any issues that could have been prevented by 
> including an FCS.
> >
> >Luca
> >
> >Stewart Bryant wrote:
> >
> >>
> >>I have been out of town for a few days, and so have only just got
> >>to this thread.
> >>
> >>The FCS topic was always planned for the Minneapolis agenda, so
> >>I propose that we (the chairs) follow Danny's plan and put together
> >>a summary of the consensus at the end of LC, post to the list, and
> >>the WG discuss/ratify when we meet on 10th. I will post the summary
> >>of the outcome to the list on the 11th.
> >>
> >>As stated below, the decision (either way) must be based on the
> >>TECHNICAL merits of the proposal, and be based on the behavior
> >>of the PW as a system.
> >>
> >>Stewart
> >>
> >>
> >>
> >>
> >>Danny McPherson wrote:
> >>
> >>>
> >>>Folks,
> >>>I don't care if you work for an equipment manufacturer, service
> >>>provider or at a fast food drive-thru, please state your positions
> >>>based technical merit (I assure you that's all they'll be weighted
> >>>on).  Where someone has (or currently!) works is completely
> >>>irrelevant!
> >>>
> >>>At this point the closest thing to consensus I've seen (yes, after
> >>>re-reviewing ALL of the April and September discussions on this
> >>>topic [again]) is that a separate draft seems to be the preferred
> >>>option.
> >>>
> >>>Even that option has pitfalls, most of which have been expressed
> >>>here (in this or the previous two threads on this subject), though
> >>>it's still seems to be the most widely supported.
> >>>
> >>>Please continue discussing the issue and either Stewart or I will
> >>>summarize what we believe to be consensus by the end of LC  on the
> >>>concerned WG documents.
> >>>
> >>>-danny
> >>>
> >>>
> >>>_______________________________________________
> >>>pwe3 mailing list
> >>>[email-masked]
> >>>https://www1.ietf.org/mailman/listinfo/pwe3
> >>>
> >>
> >>
> >>_______________________________________________
> >>pwe3 mailing list
> >>[email-masked]
> >>https://www1.ietf.org/mailman/listinfo/pwe3
> >>
> >
> >
> >
> >_______________________________________________
> >pwe3 mailing list
> >[email-masked]
> >https://www1.ietf.org/mailman/listinfo/pwe3
> 
> 
> _______________________________________________
> pwe3 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/pwe3
>

---

From: "David Allan" <[email-masked]>
Date: Mon, 10 Nov 2003 03:30:54 -0500

Add me to the list of non-attendees in Minneapolis.

Eric put it quite well when he stated that what we really want is an e2e
integrity check in all modes. At the same time providing this functionality
is impossible without modifying the client layer, which is a non starter as
a solution. So Ali's suggestion that we need a comprehensive solution for
all modes is demonstratably unachievable and carriage of FCS e2e for some
modes is the best we can do. As such we should not preclude it.

Further I would prefer that this is in the base documents, Andy Nick and I
have a draft that shows the required changes are minor.

rgds
Dave

---

From: Shahram Davari <[email-masked]>
Date: Mon, 10 Nov 2003 05:13:43 -0800

Hi,

I would like to support having an option to transport Ethernet FCS. There are 2 cases:

1) The ingress PE does not modify the ingress Ethernet packet, such as the port-based raw mode.
In this case keeping customer's FCS, provides the capability of checking the integrity of the
access+transport networks.

2) The ingress PE does modify the ingress Ethernet packet, such as the raw mode with service delimiting tag,
and the tagged mode (which may add a dummy tag). In these cases, if the ingress PE generates new FCS and
transports the packet with that FCS, then at least the integrity of the transport network (excluding
the ingress and egress PE) could be checked. 

-Shahram

>From: David Allan [mailto:[email-masked]]
>Sent: Monday, November 10, 2003 3:31 AM
>To: 'Scott Kotrla'; 'Ali Sajassi'; 'Luca Martini'; [email-masked]
>Cc: [email-masked]; 'Danny McPherson'
>Subject: RE: [PWE3] last call on control and ethernet drafts
>
>

---

From: Nick DelRegno <[email-masked]>
Date: Mon, 10 Nov 2003 10:33:27 -0600

Unfortunately I had to cancel my plans to attend as well.

Once again, for the record, I also favor the option to retain the FCS on PW
Ingress be added to the base drafts.

Nick

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of David
Allan
Sent: Monday, November 10, 2003 2:31 AM
To: 'Scott Kotrla'; 'Ali Sajassi'; 'Luca Martini'; [email-masked]
Cc: [email-masked]; 'Danny McPherson'
Subject: RE: [PWE3] last call on control and ethernet drafts

rgds
Dave

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 10 Nov 2003 13:52:14 -0500

As I have previously stated, I am also in favor of the option to retain 
Ethernet FCS.

David, Nick, and I have written a short draft that we'll submit when the 
draft site reopens that is 100% backwards compatible with the current 
signaling so you can implement it in only a subset of the PEs and they'll 
interoperate.  It's so short that I've attached it here to give people a 
chance to look at it prior to this evening's meeting.

Andy

----------

  Internet Draft                                         Andrew G. Malis
  Document: draft-malis-pwe3-fcs-retention-00.txt                Tellabs
  Expires:  May 2004                                         David Allen
                                                         Nortel Networks
                                                          Nick Del Regno
                                                                     MCI
                                                           November 2003

                     PWE3 Frame Check Sequence Retention

  Status of this Memo

     This document is an Internet-Draft and is in full conformance with
     all provisions of Section 10 of RFC2026.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF), its areas, and its working groups.  Note that
     other groups may also distribute working documents as Internet-
     Drafts.

     Internet-Drafts are draft documents valid for a maximum of six
     months and may be updated, replaced, or obsoleted by other
     documents at any time. It is inappropriate to use Internet-Drafts
     as reference material or to cite them other than as "work in
     progress."

     The list of current Internet-Drafts can be accessed at
          http://www.ietf.org/ietf/1id-abstracts.txt
     The list of Internet-Draft Shadow Directories can be accessed at
          http://www.ietf.org/shadow.html.

  Abstract

     This document defines a mechanism for preserving frame FCS through
     Ethernet, Frame Relay, and HDLC pseudowires.

  Table of Contents

     1. Specification of Requirements.................................2
     2. Overview......................................................2
     3. FCS Retention With MPLS-based Pseudowires.....................3
     4. FCS Retention With L2TPv3-based Pseudowires...................4
     5. Security Considerations.......................................4
     6. IANA Considerations...........................................4
     7. References....................................................4
     8. Authors' Addresses............................................5

  Malis et al                Expires May 2004                   [Page 1]

                            PWE3 FCS Retention             November 2003

  1. Specification of Requirements

     The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
     "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
     this document are to be interpreted as described in RFC 2119 [6].

  2. Overview

     The specifications for Ethernet, Frame Relay, HDLC, and PPP
     pseudowire encapsulation [1] [2] [3] include a mode of use where
     frames are transparently delivered across the pseudowire without
     any header or other alterations by the pseudowire ingress or egress
     Provider Edge (PE). [Note that this mode is inherent for HDLC and
     PPP Pseudowires.]

     However, these specifications all specify that the original Frame
     Check Sequence (FCS) be removed at ingress and regenerated at
     egress, which means that the frames may be subject to unintentional
     alteration during their traversal of the pseudowire from the
     ingress to the egress PE.  Thus, the pseudowire cannot be
     absolutely guaranteed to be "transparent" in nature.

     To be more precise, pseudowires, as currently defined, leave the
     payload vulnerable to undetected errors caused by the encapsulating
     network. As defined for MPLS, not only can a PW-aware device
     internally corrupt an encapsulated payload, but ANY MPLS LSR in the
     path can corrupt the encapsulated payload. In the event of such
     corruption, there is no way to detect the corruption through
     the path of the pseudowire. Further, because the FCS is calculated
     upon network egress, any corruption will pass transparently through
     ALL Layer 2 switches(Ethernet and Frame Relay) through which the
     packets travel. Only at the endpoint, assuming the corrupted packet
     even reaches the correct endpoint, can the packet be discarded, and
     depending on the contents of the packet, the corruption may not
     ever be detected.

     Not only does the encapsulation technique leave the payload
     unprotected, it also subverts the error checking mechanisms already
     in place in SP and customer networks by calculating FCS on
     questionable data.

     In a perfect network comprising perfect equipment, this is not an
     issue. However, as there is no such thing, it is an issue. SPs
     should have the option of saving overhead by yielding the ability
     to detect faults. Equally, SPs should have the option to sacrifice
     the overhead of carrying the original FCS end-to-end to ensure the
     ability to detect faults in the encapsulating network.

  Malis et al                Expires May 2004                   [Page 2]

                            PWE3 FCS Retention             November 2003

     This document defines such a mechanism to allow the ingress PE to
     retain the original frame FCS on ingress to the network, and
     relieves the egress PE of the task of regenerating the FCS.

     This in an OPTIONAL mechanism for pseudowire implementations.  For
     interoperability with systems that do not implement this document,
     the default behavior is that the FCS is removed at the ingress PE
     and regenerated at the egress PE, as specified in [1], [2], and
     [3].

  3. FCS Retention With MPLS-based Pseudowires

     When using the signaling procedures in [4], there is a Virtual
     Circuit FEC element parameter ID used to signal the desire to
     retain the FCS when advertising a VC label:

        Parameter   ID Length    Description
             0x??           2    FCS Retention Indicator

       Note: The next currently available parameter number is 0x0A [5].
       This parameter number will be added to [5] if the timing permits
       while it is still a draft, otherwise it will be assigned by IANA.
       This note is to be removed once the assignment has been made.

     The presence of this parameter ID in the VC FEC element indicates
     that the egress PE requests the ingress PE to retain the FCS for
     the VC label being advertised.  It does not obligate the ingress PE
     to retain the FCS; it is simply an indication that the ingress PE
     MAY retain the FCS.  The sender MUST NOT retain the FCS if this
     parameter ID is not present in the VC FEC element.

     Since unknown parameter IDs are silently ignored [4], backwards
     compatibility with systems that do not implement this document is
     provided by requiring that the FCS is retained ONLY if the FCS
     Retention Indicator has been included in the advertisements for
     both directions on a pseudowire.

     If the ingress PE recognizes the FCS Retention Indicator parameter
     ID, but does not wish to retain the FCS, it need only issue its own
     label mapping message for the opposite direction without including
     the FCS Retention Indicator.  This will prevent FCS retention in
     either direction.

     If PWE3 signaling [4] is not in use for a pseudowire, then whether
     or not the FCS is to be retained MUST be identically provisioned in
     both PEs at the pseudowire endpoints.

  Malis et al                Expires May 2004                   [Page 3]

                            PWE3 FCS Retention             November 2003

  4. FCS Retention With L2TPv3-based Pseudowires

     This section will be added in a subsequent revision.

  5. Security Considerations

     This mechanism enhances the security of transparent Ethernet, Frame
     Relay, and HDLC pseudowires, because the original FCS, as generated
     by the Customer Edge (CE), is included in the encapsulation.  When
     the encapsulated payload passes FCS checking at the destination CE,
     it is clear that the payload was not altered in any way during its
     transmission through the network.

     Of course, nothing comes for free; this requires the additional
     overhead of carrying the original FCS (in general, either two or
     four octets per payload packet).

     This signaling was designed to be backwards compatible and
     interoperable with systems that do not implement this document.

  6. IANA Considerations

     This document requires IANA to allocate a PWE3 Virtual Circuit FEC
     element parameter ID.

  7. References

     [1]
         Martini, L. et al, "Encapsulation Methods for Transport of
         Ethernet Frames Over IP and MPLS Networks", draft-ietf-pwe3-
         ethernet-encap-04.txt, October 2003, work in progress

     [2]
         Martini, L. et al, "Frame Relay Encapsulation over Pseudo-
         Wires", draft-ietf-pwe3-frame-relay-01.txt, July 2003, work in
         progress

     [3]
         Martini, L. et al, "Encapsulation Methods for Transport of
         PPP/HDLC Frames Over IP and MPLS Networks", draft-ietf-pwe3-
         hdlc-ppp-encap-mpls-00.txt, June 2003, work in progress

     [4]
         Martini, L. et al, "Transport of Layer 2 Frames Over MPLS",
         draft-ietf-pwe3-control-protocol-04.txt, October 2003, work in
         progress

     [5]
         Martini, L. et al, "IANA Allocations for pseudo Wire Edge to
         Edge Emulation (PWE3)", draft-ietf-pwe3-iana-allocation-02.txt,
         October 2003, work in progress

     [6]
         Bradner, S., "Key words for use in RFCs to Indicate Requirement
         Levels", RFC 2119, March 1997.

  Malis et al                Expires May 2004                   [Page 4]

                            PWE3 FCS Retention             November 2003

  8. Authors' Addresses

     Andrew G. Malis
     Tellabs
     90 Rio Robles Dr.
     San Jose, CA 95134
     Email: [email-masked]

     David Allan
     Nortel Networks
     3500 Carling Ave.
     Ottawa, Ontario, CANADA
     Email: [email-masked]

     Nick Del Regno
     MCI
     400 International Parkway
     Richardson, TX 75081
     Email: [email-masked]

  Malis et al                Expires May 2004                   [Page 5]

---

From: "Prayson Pate" <[email-masked]>
Date: Mon, 10 Nov 2003 14:17:24 -0500

I would also like to add my support for having an option to preserve
Ethernet FCS.

Prayson

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Andrew G. Malis
Sent: Monday, November 10, 2003 1:52 PM
To: Shahram Davari
Cc: PWE3 WG (E-mail)
Subject: RE: [PWE3] last call on control and ethernet drafts

Andy

<snipped to end>

---

From: Sasha Vainshtein <[email-masked]>
Date: Tue, 11 Nov 2003 02:52:09 +0200

Andy and all,
Please see a brief comment inline.

----------------------------------------------------------------------------
--------
With best regards,
                          Sasha Vainshtein
email:   [email-masked] <mailto:[email-masked]> 
phone:  +972-3-7659993 (office)
            +972-8-9254948 (home)
            +972-58-674833 (cellular)

> From: Andrew G. Malis [mailto:[email-masked]]
> Sent: Monday, November 10, 2003 10:52 PM
> To: Keith Knightson
> Cc: Prayson Pate; Shahram Davari; PWE3 WG (E-mail)
... snipped...
> 
> >It would seem to imply that in the case where it is carried 
> >it is not looked at, at the ingress.  I can't imagine seeing a false FCS
and 
> >continuing as if it were correct.
> 
> That's a good point to mention explicitly - I'll add to the 
> draft that the FCS SHOULD (or MUST?) be examined at the ingress to prevent

> errored frames that would be discarded at the receiving CE anyway from 
> wasting bandwidth.
> 
IMO, it is a MUST.

... snipped to the end

---

From: Ali Sajassi <[email-masked]>
Date: Mon, 10 Nov 2003 21:22:40 -0800

At 07:54 AM 11/10/2003 +0000, [email-masked] wrote:
>I have checked with my colleagures here and BT do not agree with this view 
>as a general architectural principle.  When dealing with simple 
>client/server encapsulation (between any arbitrary pair of technologies) 
>the general maxim should be to leave the client layer PDU structure alone 
>to ensure layer independence.....so this is *any* change of the client 
>layer, which of course also includes any compression behaviour.

Do you and your colleagues have the same view for other services that I 
mentioned (EVPL, EPLAN, and EVPLAN) or do you have this view for EPL ?

-Ali

---

From: Ali Sajassi <[email-masked]>
Date: Mon, 10 Nov 2003 21:53:52 -0800

>modes is the best we can do. As such we should not preclude it.

What I meant was instead of fixing this problem for one mode, we should try 
to fix it for all modes. And as you said if modifying the client layer is 
not an option, then the only option we are left with is to ensure that the 
forwarding of packets within the P/PE nodes are CRC protected. This way, we 
have a reliable transport for all modes.

-Ali

---

From: Scott Kotrla <[email-masked]>
Date: Tue, 11 Nov 2003 01:30:33 -0700

As a service provider writing RFPs for equipment vendors, I can very
easily require the FCS to be kept and I can do packet captures to
validate compliance in a lab setting.  Asking an equipment provider to
prove that their box is incapable of corrupting packets is a much more
daunting task, especially given that PWs may be riding across an
installed base of P routers that may not meet these "packet corruption
requirements."  Other than the minimal effect of 4 extra bytes, FCS
retention affects the endpoints only and not every P router in between.

As far as isolating the problem, a service providers can do fault
isolation using tools like loopbacks once it is know there is a problem.
If it is not known that the network is broken, there's no chance of ever
fixing it.  If equipment providers have done their jobs right, internal
corruptions of packets should never leave the equipment.  However, if
equipment gets into the network that is not performing correctly, it is
an absolute service provider requirement that it be possible to
determine that this unfortunate event has transpired.

As I have previously stated, it is possible to architect a network where
VLANs are never swapped in any service.  In such a network, service
providers shouldn't be forced to toss customer data away just because
other service providers might be doing VLAN swapping. The option to make
a service look like a Pseudowire should be there for any service
provider that has compliant PEs in the network and wants to turn it on.

Thanks,

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ali
Sajassi
Sent: Monday, November 10, 2003 11:29 PM
To: Andrew G. Malis; Keith Knightson
Cc: Prayson Pate; Shahram Davari; PWE3 WG (E-mail)
Subject: RE: [PWE3] last call on control and ethernet drafts

>That's a good point to mention explicitly - I'll add to the draft that
the 
>FCS SHOULD (or MUST?) be examined at the ingress to prevent errored
frames 
>that would be discarded at the receiving CE anyway from wasting
bandwidth.

Same error detection logic applies to within a P/PE nodes such that
while 
forwarding packets from ingress to egress ports of a PE/P node and error
is 
detected, then the packet should be discarded. If the platform cannot do

that and the error can only be detected by the CE (via FCS retention),
then 
isolating the location of the error would be an interesting task !!
If the P/PE nodes have such capability and FCS retention is still
needed, 
then I would question the value added of it ?
Have we run any test to see the probability of an undetected error via 
platforms capable of CRC-protection ? If the probability of it is 
negligible (e.g., 10**-12), does it still make sense to do it ?

-Ali

---

From: Mark Seery <[email-masked]>
Date: Tue, 11 Nov 2003 01:29:47 -0800 (PST)

Scott Kotrla <[email-masked]> wrote: 
As a service provider writing RFPs for equipment vendors, I can very
easily require the FCS to be kept.....

<snipped to the end>

Scott, please do not take my following comments as a rebuttal to your points, because I think enough people have expressed interest in this, for all types of logical transport/separation of layer arguments etc in addition to what you have said.
 
However, I would like to make the following observation for your consideration, in what ever present or future context the observation might apply:
 
At some point here we are establishing an architectural house of cards. If we can not trust the integrity of a PW, then it begs the question how will you trust any other service you offer based on IP or MPLS, and can you even trust the integrity of the PW control plane, making use of a PW data plane, and if you can't.......

---

From: [email-masked]
Date: Tue, 11 Nov 2003 11:07:09 -0000

Ali Sajassi wrote 11 November 2003 05:23
> mentioned (EVPL, EPLAN, and EVPLAN) or do you have this view for EPL ?
Hi Ali, I understand your point.  Yes EPL constrains a normally cnls client to behave in a form of p2p co-ps mode, so this is an exception rather than the rule here.  However (and I checked this with my func arch expert colleagues working on ethernet yesterday), although the FCS definitely belongs to the ethernet client, in a G.805/809 func arch sense the FCS resides in the adaptation function rather than the trail term function.  Now since a client layer's link-connections are provided server layer trails, this means that if any client PDU overhead is changed (during client->server adaptation) then any FCS present will need a linear update here.  This is no different in principle to modifying the HEC in ATM cells say to the VPI/VCI changes on different link-connections.

However, the general maxim of 'don't touch the client layer signal if it can be avoided' still always holds.

regards, Neil
<snipped to end>

---

From: Scott Kotrla <[email-masked]>
Date: Wed, 12 Nov 2003 09:06:08 -0700

My requirement is that any errors that are injected into the network by
the service provider not be allowed to leave the service provider
network.  Any PW/IP control traffic remains in the service provider
network and should be protected from errors by TCP.  If the service is
IP, which does not have a checksum for the whole packet, the only
solution at the egress of the provider's routed network would be to look
at the Layer 4 CRC.  The layer 2 tail circuits would be protected by the
end-to-end FCS which would go from the customer router to the IP service
edge.
 
Hope this makes sense.
 
Thanks,
 
Scott
 
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark
Seery
Sent: Tuesday, November 11, 2003 2:30 AM
To: Scott Kotrla
Cc: [email-masked]
Subject: RE: [PWE3] last call on control and ethernet drafts

Scott Kotrla <[email-masked]> wrote: 
As a service provider writing RFPs for equipment vendors, I can very
easily require the FCS to be kept.....
<snipped to the end>
Scott, please do not take my following comments as a rebuttal to your
points, because I think enough people have expressed interest in this,
for all types of logical transport/separation of layer arguments etc in
addition to what you have said.
 
However, I would like to make the following observation for your
consideration, in what ever present or future context the observation
might apply:
 
At some point here we are establishing an architectural house of cards.
If we can not trust the integrity of a PW, then it begs the question how
will you trust any other service you offer based on IP or MPLS, and can
you even trust the integrity of the PW control plane, making use of a PW
data plane, and if you can't.......

---

From: Mark Seery <[email-masked]>
Date: Wed, 12 Nov 2003 08:40:10 -0800 (PST)

Hi Scott, thanks for the response. this makes sense, few comments below.

--- Scott Kotrla <[email-masked]> wrote:
> network.

Makes all kinds of sense - preserving a fundamental promise/value to the
customer....

> Any PW/IP control traffic remains in the service provider

Your desire to preserve the integrity of the subscriber's payload makes sense,
I don't think that is under any debate, by anyone that I have seen. Neil's
comments about layer separation and simplification also make sense to me on a
variety of intellectual/philiosophical (sp?) levels as well.

There is a fundamental problem this entire issue raises though. For those that
have worked around routers for a long time (and not suggesting you have not)
there is a comfort and familiarity with the concept for stripping L2
headers/trailers on ingress and then reapplying them on egress (now extended by
PWE from within a system to across a network - which perhaps is what makes a
difference to you). The issue you raise, unintentionally perhaps, raises the
question of whether routing works (as compared to transport - which is
essentially what you are suggesting). There appears to be existence proofs that
routing works, so I suspect this is the crux of the response. Yes some
equipment does not support retaining the FCS, but this is an attribute of an
approach (routing) not simply a technical nit. 

The power of routing has been the ability to strip PDUs out of any layer 2
network, forward them based on information not present in the L2 header, and
then place the PDU back on to any other L2 network. It is a kind of
interworking, without the complexity of interworking (I realise this is not
interworking - just trying to make a point). In the IETF's approach to PWE, we
see an extension of this philosphy (IMHO) - and I don't say that to diminish
your concerns, merely to highlight a philisophical issue.

So we see here in PWE an approach that is kind of routing and kind of
transport, which makes for some interesting conversations, tensions, and
expectation short falls ;-) Perhaps that is what organic (non-architected)
evolution is all about, perhaps it is not, we shall see.

For the record, I understand the desire of those to get a base draft moving
forward, especially if adding the option to retain FCS is a moot point for a
large base of installed equipment, and I support the movement forward of a
separate draft on this issue.

Best,
Mark

(BTW, thanks to chairs for a great issues presentation at yesterdays meeting)

---

From: Scott Kotrla <[email-masked]>
Date: Wed, 12 Nov 2003 10:17:35 -0700

Mark,

I think the issue is that routers have never been considered
transmission equipment.  With advent of Pseudowires, that is changing,
and I would propose that the requirement for L3 routers are different
from the requirements of L3 networks that will be transporting lower
level protocols.

Thanks,

Scott

-----Original Message-----
From: Mark Seery [mailto:[email-masked]] 
Sent: Wednesday, November 12, 2003 9:40 AM
To: Scott Kotrla
Cc: [email-masked]; [email-masked]
Subject: RE: [PWE3] last call on control and ethernet drafts

> end-to-end FCS which would go from the customer router to the IP
service

Your desire to preserve the integrity of the subscriber's payload makes
sense,
I don't think that is under any debate, by anyone that I have seen.
Neil's
comments about layer separation and simplification also make sense to me
on a
variety of intellectual/philiosophical (sp?) levels as well.

There is a fundamental problem this entire issue raises though. For
those that
have worked around routers for a long time (and not suggesting you have
not)
there is a comfort and familiarity with the concept for stripping L2
headers/trailers on ingress and then reapplying them on egress (now
extended by
PWE from within a system to across a network - which perhaps is what
makes a
difference to you). The issue you raise, unintentionally perhaps, raises
the
question of whether routing works (as compared to transport - which is
essentially what you are suggesting). There appears to be existence
proofs that
routing works, so I suspect this is the crux of the response. Yes some
equipment does not support retaining the FCS, but this is an attribute
of an
approach (routing) not simply a technical nit. 

The power of routing has been the ability to strip PDUs out of any layer
2
network, forward them based on information not present in the L2 header,
and
then place the PDU back on to any other L2 network. It is a kind of
interworking, without the complexity of interworking (I realise this is
not
interworking - just trying to make a point). In the IETF's approach to
PWE, we
see an extension of this philosphy (IMHO) - and I don't say that to
diminish
your concerns, merely to highlight a philisophical issue.

So we see here in PWE an approach that is kind of routing and kind of
transport, which makes for some interesting conversations, tensions, and
expectation short falls ;-) Perhaps that is what organic
(non-architected)
evolution is all about, perhaps it is not, we shall see.

For the record, I understand the desire of those to get a base draft
moving
forward, especially if adding the option to retain FCS is a moot point
for a
large base of installed equipment, and I support the movement forward of
a
separate draft on this issue.

Best,
Mark

(BTW, thanks to chairs for a great issues presentation at yesterdays
meeting)

---

From: mark seery <[email-masked]>
Date: Wed, 12 Nov 2003 12:06:38 -0800

Scott Kotrla wrote:

>transmission equipment.  
true, worth noting, no doubt our friends at 802 have heard similar 
comments, but hard to parse in an engineering work group.

I would also observe that class 5 switches are not used as transmission 
equipment either, so we have to understand whether we are dealing with 
a) requirements issues b) familiarity issues c) architecture domain 
issues (i.e. switching vs transmission) d) organizational issues e) 
other f) all of the above; and if we merge switching and transmission 
does that require a new way of thinking, or simply replicating both 
capabilities in the same system - no simple question for the industry to 
grapple with.

no doubt though this is the unlanced boil that will pop in many forms 
during the life of this WG.

>With advent of Pseudowires, that is changing,
this is unquestionably true on many levels, specifically if the goal is 
to recreate a transmission facility from a functional, operations, and 
management perspective. my only, clumsy, point was that there is a proof 
point that you can strip an FCS, even change the PDU, and reapply the 
FCS and it works. people here are inherently aware of that proof point 
which plays in to the discussion.

---

From: Scott Kotrla <[email-masked]>
Date: Wed, 12 Nov 2003 14:02:43 -0700

Mark,

I think we are dealing with all of the issues that you mentioned.  While
you can do a lot of things on a small scale, service providers can't put
10000s of devices into a network and afford to have gaping holes in
their abilities to detect faults.  One difference between transmission
and switching has traditionally been that the size of the transmission
network greatly outnumbered the size of the switching network in terms
of number of nodes.  As switching technology becomes intertwined in the
transmission network, there is no doubt that changes must be made to
switching protocols to build scale.

Essentially, MCI's requirement is for an end-to-end FCS for detecting
faults in the transmission/switching network.  All MCI RFP's dealing
with Ethernet transmission/switching will have this requirement.  I
believe that any service provider who looks at the ramifications of not
providing an end-to-end guarantee against corruption (bounded by the
CRC32 capabilities) will comes to the same conclusions that MCI has.
Many service providers are ambivalent today in this area for many
reasons:

1. They have not looked at anything beyond Private Line Ethernet
services
2. They are not aware of the ramifications of not providing an
end-to-end FCS
3. They haven't looked forward to a time in the future where the
switching protocols become part of the transmission network

I will stand by the assertion that all service providers will ultimately
demand these capabilities until other service providers say otherwise.

As long as this is the case, any equipment provider that wants to sell
to service providers (and I assume this is any equipment provider
involved in the 802.1ad project) will ultimately have to provide these
capabilities in their equipment, whether it is part of the standard or
not.  I would think it would be in everybody's best interest to find out
whether or not this is really a service provider requirement (and to
make sure that those service providers who say it is not have really
looked at the issue).  If it is a requirement, it would seem that
standardizing it is the only logical course of action, so that equipment
providers can begin designing the capabilities into their equipment.

Thanks,

Scott

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of mark
seery
Sent: Wednesday, November 12, 2003 1:07 PM
To: Scott Kotrla
Cc: [email-masked]; [email-masked]
Subject: Re: [PWE3] last call on control and ethernet drafts

grapple with.

---

From: Mark Seery <[email-masked]>
Date: Wed, 12 Nov 2003 13:39:16 -0800 (PST)

(I may have had HTML reenabled previously by mistake - my apologies if I did). 
 
" <snip> Scott Kotrla <[email-masked]> wrote: 

Mark,

I think we are dealing with all of the issues that you mentioned. While
you can do a lot of things on a small scale, service providers can't put
10000s of devices into a network and afford to have gaping holes in
their abilities to detect faults. <<

So I think the question on some people's minds (mine included, I think Ali also
alluded to this) is whether e2e reuse of FCS helps the SP detect fault, or
merely enables the subscriber to detect fault. I assume both are goals? I also
assume that you prefer not to enable the subscriber to do so unless you can as
well ;-) Once again my clumsy articulation was the difference between payload
integrity and FCS integrity.
 
" <snip> 3. They haven't looked forward to a time in the future where the
switching protocols become part of the transmission network "

I don't know what assumptions the WG makes WRT this issue (would be interested
to hear an opinion from the chairs), what assumptions 802 makes, or what
assumptions people in general make. But I suspect this is a salient point, and
the subconscious long-term pretext/foundation of challenges we have in people
from different backgrounds holding hands across the aisle.

---

From: [email-masked]
Date: Wed, 12 Nov 2003 22:45:52 -0000

Scott please see below.......and speaking as service provider I am with you totally on this stuff.

regards, Neil

Scott Kotrla wrote 12 November 2003 21:03
> faults in the transmission/switching network.
>
>  All MCI RFP's dealing

NH=> Scott, I'd go further than this and generalise your remarks here to say that any layer network technology must have its own independent OAM that can automatically detect all failures and automatically take the right consequent actions.  We need this to deliver quality services to our customers and drive down our opex.  The FCS of ethernet belongs to the ethernet layer's characteristic information and thus can be considered to be an intrinsic part of its OAM functionality.  Further, a layer network's characteristic information (which includes its OAM functionality) must be transparently preserved in any client->server encapsulations.

{aside - The above reasons are precisely why, esp looking ahead at possibly using MPLS as converged co-ps core, I was instrumental in drafting Y.1710 and Y.1711 (MPLS OAM requirements and mechanisms respectively).  This is the only place I know where all defects are fully specified in terms of automatic detection and handling.  And for similar reasons there is also now great activity in enhancing ethernet's OAM capability in IEEE, MEF and ITU.}
<snipped to end>

---

From: [email-masked]
Date: Wed, 12 Nov 2003 23:05:20 -0000

Scott,
Scott Kotrla wrote 12 November 2003 22:13
 
> 
> The requirement would actually be to check the Path FCS on 
> egress before
> handing off to the customer and not to pass packets that are 
> corrupt.  I
> sent out another email that describes the functionality of various
> interfaces in detail, but in short:
> 
> 1. On ingress the customer FCS is checked, and either the packet is
> tossed and the appropriate counter is incremented or the packet is
> modified with new headers and new service FCS is added after the
> customer FCS
> 
> 2. On egress after the packet is returned to its original 
> condition with
> service tag and service FCS removed, the customer FCS is checked, and
> either the packet is tossed and the appropriate counter is incremented
> or the packet is sent to the customer.
> 
> The ingress counters tell me if the customer was sending corrupt data
> and the egress counters tell me if the service provider network
> corrupted the packet.
> 
> This seems extremely simple and extremely clean to me.
NH=> Sounds right to me.
  
> People have said
> that this method doesn't tell you where the error is,
NH=> And nor should it...see next comment.

> and I would be
> interested to understanding the costs of checking both the 
> customer and
> service FCS at every ingress port in the middle.  My guess is 
> that it is
> cost prohibitive and not backwards compatible.  
> 
> I think everyone can agree that once you have identified that 
> there is a
> problem in the service provider network, troubleshooting the 
> problem is
> feasible, whereas if you can't determine if a problem is a customer
> problem or a service provider problem, troubleshooting a 
> problem is next
> to impossible because there are too many variables.
NH=> Precisely so.  Detection comes before diagnosis.....these are not the same issues.  The former needs to be simple and automatically operate (in G.805/809 parlance) at the trail termination points.  Once the problem has been detected then one can wheel out more sophisticated diagnosis tools to delve into the network.  IMO this is all just common sense.

regards, Neil
<snipped>

---

From: Scott Kotrla <[email-masked]>
Date: Thu, 13 Nov 2003 07:08:45 -0700

Let me start by clarifying something that may be leading to some
confusion.  MCI is pushing for an end-to-end FCS in both 802.1ad and in
Pseudowires.  Think of stacked VLANs spokes running across a network of
switches into a Pseudowire Core connected via a network of routers.  I
think we have agreed in PWE3 that carrying an end-to-end FCS should be a
service provider option and the drafts will be modified accordingly.  I
have now moved on to pushing for the same option in 802.1ad.  Since
there were still discussions going on in PWE3 when I started posting to
the 802.1 listserv earlier this week, I started copying 802.1 on my PWE3
replies since my response were valid to both groups. 

I completely agree that it is theoretically possible to construct
switches that are not anymore likely to corrupt packets than a CRC32 is
to not catch a corrupt packet. 

It is also possible to build a network that can never be misconfigured
and that never has any problems.  This argument could be used to wish
away all OAM tools.  If one builds the perfect network, a lot of things
become a lot easier and a lot of tools would never be used.

Being able to detect errors in switches is a requirement from MCI.
However, MCI is not convinced that every piece of gear ever deployed in
the network will always meet those requirements, even if the equipment
providers promise that they do.

Equipment has always had bugs and it will always have bugs.  Even if the
equipment design makes unknown packet corruption highly improbable, the
reality of some implementations will be holes that permit unknown packet
corruption.  Convincing me that no equipment provider that claims to
meet the requirement of unknown packet corruption could ever get a box
into the network that doesn't meet that requirement will be very tough.
There are too many variables.  I completely agree with service provider
rights to save 4 bytes and not turn the feature on, but I stand by my
assertion that any service provider who understands the risks will be
willing to sacrifice 4 bytes.

Regarding hardware costs, 802.1ad interfaces are likely going to require
new hardware.  I am highly suspicious of the assertion that stacking 4
bytes on ingress and doing an extra CRC check on egress on new 802.1ad
hardware will have any significant impact on cost.  I agree that adding
this feature on existing hardware might be cost prohibitive in some
cases.

As far as the importance of detecting end-to-end errors versus
diagnosing where the errors are happening, I think I have stated this
several times already.  If I don't know that my network is running
clean, how do I troubleshoot a customer call?  I have to guarantee and
be able to demonstrate beyond a reasonable doubt that my network is
clean, and trying to walk the customer through the internal workings of
a dozen boxes is much more difficult and less convincing than showing
them egress FCS error counters.  If I know there is a problem, I have a
lot of tools at my disposal, depending on the seriousness of the
problem.  First of all, I can root cause my FCS errors to figure out
which boxes/interfaces all affected customer traffic are going through.
If enough customer packets are being corrupted that the service is
unusable, I can do intrusive monitoring and loopbacks.  If service is
not being impacted so severely, I can diagnose the problem during a
maintenance window.  If the equipment is reliable as it should be, I
shouldn't be doing this often anyway.

Thanks,

Scott

-----Original Message-----
From: mark seery [mailto:[email-masked]] 
Sent: Thursday, November 13, 2003 1:17 AM
To: Ali Sajassi
Cc: Scott Kotrla; [email-masked]; [email-masked]
Subject: Re: [PWE3] last call on control and ethernet drafts

Ali,

Sorry to see this is testing your endurance threshold (as measured by 
double !s), and anything I am doing to contribute to that.

I understand your frustration at people throwing solutions at you and 
not requirements - this is obviously an oft-expressed angst by engineers

in many different contexts. However, I think it is natural for 
experienced operators to offer their opinions on a solution, and can 
help sometimes to flush out the issues. I do agree that to deviate to 
far from a focus on the requirements is a dangerous path to travel in 
the long-term, so good to bring it back to that, but given the general 
scarcity of input from carriers, a little leeway might be constructive.

As I stated in my last email, I do see how a single FCS can work, based 
on the draft from andy et el. I have seen you consistently suggest that 
solving the problems for all modes is desired - and I agree. I have seen

others say this is not achieveable. Unfortunately I missed your proposal

on how it is achieveable, and the technical critique from others on how 
it is not - I'll look back at the april/september entrails to re-educate

myself.

While I agree that solving for all modes is best, and ultimately the 
test of whether PW is a good layer - or at the least a consistent layer,

if people seem to be focused heavily on the EPL case, that might be 
suggestive of where their interest lay.

Personally, I think stating a requirement such as "like a transmission 
facility" is a good starting point, but ultimately does provide 
engineers anything constructive to design towards, it would be nice if 
someone could turn that general requirement in to something more 
specific, then all your comments like "probability of undetectable 
errors" would come more into focus.

I am not sure I agree with your observations about loopbacks. It is not 
clear to me that it is impossible to construct non-intrusive tests with 
sufficient samples. I obviously missed that explanation as well. In 
addition, there is a distinction between the process of detecting an 
error and the process of diagnosing one - which is a point that has been

made by others. I missed that distinction in your comments.

I am not sure I understand your comments about IEEE 802.1ad 
compatibility. Even though I can see how one FCS might be sufficient, if

a PW used two, I am not sure how this creates a compatibility problem as

I don't forsee an "existing" bridge being in the middle of a PW.

I do think the issue about cost is something I would have preferred the 
carrier representatives to have picked up on and commented on - but they

probably can not give an intelligent response without a meaningful 
estimate of cost difference. If their answer is its your job to get the 
costs in line, we aint doing it with this functionality, well then I 
suggest that should probably spark a set of offline discussions. And the

cost discussion is obviously not just pertinent to the double FCS case, 
the zero FCS case clearly provides for an interesting set of discussions

between incumbent vendors and customers (one that is beyond me how I 
personally would gain by promoting, but it is clear to me how it is a 
worthy discussion for some set of agents in the value chain to have).

---

From: "Matt Squire" <[email-masked]>
Date: Thu, 13 Nov 2003 10:45:09 -0500

> this feature on existing hardware might be cost prohibitive in some

I haven't really looked into detailed cost derivatives for such a feature, however it does change the architectural model pretty significantly.  In all of the products on which I've worked in my life, there has not been a way to do FCS checking on egress.  The MAC performs an FCS check on ingress and counts the FCS errors, often discarding the errored frame for you (unless you configure the MAC to pass errored frames). 

On egress, the FCS may be recalculated depending if VLAN tags or priority bits have been inserted/deleted/modified.  In many cases, the MAC or some other hardware computes the FCS for you and appends it.  
 
An architecture where FCS is checked at egress is a non-trivial modification to how things work today.  Not saying its not possible, but the MAC, which does this for many people today, couldn't be used in the way its designed and currently implemneted.  

- Matt

---

From: Scott Kotrla <[email-masked]>
Date: Thu, 13 Nov 2003 09:08:28 -0700

Matt,

I would think this would be a function that would occur between the
frame manipulation of 802.1ad and passing the frame to the MAC.  I
assume that the MAC is not what does the 802.1ad VLAN stacking/swapping
functionality.  Is this assumption not true?

I take from your message that doing the egress FCS check is the concern
and not appending the new FCS at ingress?  If so, you could make doing
the egress FCS check optional.  If turned off, the error detection
wouldn't be confined to the service provider network, but at least the
frames would all get tossed at the ingress to the customer network and
it would be clear that there is a problem.  802.3ah OAM could
potentially send an alarm message back to the service provider that
there are excessive errors on the wire once customer CPE supports this
feature.

Maybe you have some insight into what MAC chips typically do with the
FCS.  How often is it tossed after the FCS check and how often is it
passed through the system end-to-end, with recalculation only if their
have been changes?

Thanks,

Scott

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Matt Squire
Sent: Thursday, November 13, 2003 8:45 AM
To: Scott Kotrla; [email-masked]; Ali Sajassi
Cc: [email-masked]; [email-masked]
Subject: [802.1] RE: [PWE3] last call on control and ethernet drafts

> this feature on existing hardware might be cost prohibitive in some

I haven't really looked into detailed cost derivatives for such a
feature, however it does change the architectural model pretty
significantly.  In all of the products on which I've worked in my life,
there has not been a way to do FCS checking on egress.  The MAC performs
an FCS check on ingress and counts the FCS errors, often discarding the
errored frame for you (unless you configure the MAC to pass errored
frames). 

On egress, the FCS may be recalculated depending if VLAN tags or
priority bits have been inserted/deleted/modified.  In many cases, the
MAC or some other hardware computes the FCS for you and appends it.  
 
An architecture where FCS is checked at egress is a non-trivial
modification to how things work today.  Not saying its not possible, but
the MAC, which does this for many people today, couldn't be used in the
way its designed and currently implemneted.  

- Matt

---

From: "Matt Squire" <[email-masked]>
Date: Thu, 13 Nov 2003 11:45:26 -0500

> functionality.  Is this assumption not true?

Thats certainly true.

This functional block, which is pretty much HW, doesn't exist in most product architectures. I'm not saying it can't be added, but its a new egress HW block that doesn't cost much in C?COGS but which does change the fundamental architecture of what happens where and why.  

> there are excessive errors on the wire once customer CPE supports this

That's my primary concern from an implementation and architectural perspective.  Can't speak for anyone else.  

On ingress, I can imagine how it could be done on non-ASIC implementations.  Of course how I imagine it happening might conflict internal error detection mechanisms (e.g. if you treat a packet with an FCS as not having an FCS, then its just data and you can add one at egress, but then its difficult to use that FCS internally for detecting internal corruption).  

But all implementations are different and I can also see how it would make life difficult for some.  

Personally, I'm torn.  I understand your concerns.  But I also understand Ali's reasoning that it may not buy you anything.  Your basic premise is that equipment can't be trusted:(  As an equipment vendor, I think I might cry...

> passed through the system end-to-end, with recalculation only if their

(based on recollection, haven't verified with spec reading)

Officially, I believe the MAC discards FCS errored frames on ingress.  Unofficially, most MACs provide mechanisms to pass up FCS errored frame (really needed in system debugging).  

- Matt

---

From: Heiles Juergen <[email-masked]>
Date: Fri, 14 Nov 2003 17:15:33 +0100

Scott,

I agree that a network operator needs a end-to-end supervision of the service.
For the pwe3 case I therefore understood the proposal to include the FCS as the original definition didn't support it.

For the 802.1ad case I have problem to understand the need for an additional FCS as I think the existing FCS can be used for the end-to-end service supervision. May be it is as misunderstanding of the Ethernet process on my side.

The Ethernet frame from the customer has a FCS. At the ingress UNI all frames with wrong FCS are dropped. So you forward only frames with correct FCS into the network. When you add now an additional FCS it is not different from the already existing FCS, both indicate good frames. 802.1ad now manipulates the frame by adding the provider tag. This is similar to what .1q bridges are already doing today. These changes have to be reflected in the FCS. 

802.1q says the following:
NOTE-There are two possibilities for recreating a valid FCS. The first is to generate a new FCS by algorithmically
modifying the received FCS, based on knowledge of the FCS algorithm and the transformations that the frame has
undergone between reception and transmission. The second is to rely on the normal MAC procedures to recalculate the
FCS for the outgoing frame. The former approach may be preferable in terms of its ability to protect against increased
levels of undetected frame errors. ISO/IEC 15802-3, Annex G, discusses these possibilities in more detail. The
frame_check_sequence parameter of the Enhanced Internal Sublayer Service (7.1) is able to signal the validity, or otherwise,
of the FCS; an unspecified value in this parameter in a data request indicates to the transmitting MAC that the
received FCS is no longer valid, and the FCS must therefore be recalculated.

For a end-to-end supervision the first approach has to be used. It means that for the outgoing FCS the incoming FCS is used and it is modified according to the frame modifications. This means that errors that occur within the bridge are covered and will be detected. 802.1d Annex G provides examples on how this can be achieved.

One question to the Ethernet community: What is the usual FCS procedure for .1q bridges modification or new calculation? What would it be for .1ad bridges?

The frame now passes through the network and comes to the egress UNI or may be dropped before due to a FCS error. Both, the original and the new FCS have the same coverage as they start with a no error value at he ingress UNI and are checked at the egress UNI. So the additional FCS provides no additional information in my view, as the standard FCS also covers internal errors of a bridge.
Only in case incoming customer frames with wrong FCS are not dropped at the ingress UNI and forward into the provider network the additional FCS is useful as in this case the network coverage of both FCS would be different.

Scott you mentioned that the egress counter tells you if the service provider network corrupted the packet. This counter will not give you the whole picture as Ethernet frames could be dropped at intermediate nodes in case of wrong FCS. The egress counter tells you only the frames with wrong FCS on the last link or you have to disable drop in case of wrong FCS at the intermediate notes.

Regards

Juergen

> -----Ursprungliche Nachricht-----
> Von: Scott Kotrla [mailto:[email-masked]]
> Gesendet: Donnerstag, 13. November 2003 15:09
> An: [email-masked]; 'Ali Sajassi'
> Betreff: RE: [PWE3] last call on control and ethernet drafts
> 
>

---

From: Norman Finn <[email-masked]>
Date: Mon, 17 Nov 2003 10:58:13 +0100

Scott,

I conclude from your arguments that bridges live in a fantasy world, and
don't actually work.  You're clearly wrong.

Had you said simply, "Bridges don't work like routers," you'd have been correct.

The reason that routers get away with poor error control is they they mostly
run over bridges and physical Ethernets, which have excellent error detection.
When you run routers over bridges over PW without 32-bit CRCs, the circle is
removed one further layer from reality, and threatens to fail.

-- Norm