Subject: [rbridge] FTag

From: Eric.Gray at marconi.com (Gray, Eric)
Date: Fri, 20 Oct 2006 15:05:14 -0400

Silvano,

	I was working on the replies to you analysis below, thinking
I would not complete them this week.  As you can see, I had a bit
more time than I expected to have and was able to complete my
response earlier than I thought. 

	As a first observation, whenever someone says "this is not
hardware friendly", my gut reaction is to ask "for what hardware"?

	As a second observation, although the number of MPLS labels
may change, the Ethernet encapsulations defined for PWE3 already
uses a 32-bit alignment assumption independent of Ethernet frame
alignment either "below" or "above" the MPLS SHIM.  Consequently,
there is definitely hardware in existence, in development or both
that deals with this alleged "alignment issue."

--> -----Original Message-----
--> From: Silvano Gai [mailto:sgai at nuovasystems.com] 
--> Sent: Wednesday, October 18, 2006 12:04 PM
--> To: Gray, Eric
--> Cc: rbridge at postel.org
--> Subject: RE: [rbridge] FTag
--> 
--> 
--> 
--> Eric,
--> 
--> Let me depict you what happens when you insert one MPLS 
--> label, [or] a 1Q option or a 1S option
--> 
--> Original:
--> +--------------------------------+
--> |               DA               |
--> +----------------+---------------+
--> |       DA       |     SA        |
--> +----------------+---------------+
--> |               SA               |
--> +--------------------------------+
--> |    Ethertype   |   Payload     |
--> +--------------------------------+
--> |                                |
--> |         ...........            |
--> 
--> 
--> Original + Opt (where Opt can be MPLS, 1Q or 1S)
--> +--------------------------------+
--> |               DA               |
--> +----------------+---------------+
--> |       DA       |     SA        |
--> +----------------+---------------+
--> |               SA               |
--> +--------------------------------+
--> |    Ethertype   |      Opt      |
--> +--------------------------------+
--> |      opt       |   Payload     |
--> +--------------------------------+
--> |                                |
--> |         ...........            |
--> 
--> The HW that perform the insertion/removal does not need to 
--> realign the payload that is in its buffer.
--> 

This is an interesting, but inefficient, storage scheme in the
first place.  With this storage scheme, the same DA and/or SA
values will be stored in uncompressed format hundreds or even
thousands of times in any sizable buffer.

I think it is unrealistic to assert that this is the only way
that this information might be stored in a buffer.  If it is 
not stored in this precise fashion, then the below observations
are invalidated.

--> 
--> Now Let's consider the encapsulation/decapsulation function 
--> by looking at the Original frame + TRILL header (as proposed 
--> by the current draft)

Let's be clear about what is proposed by the current draft.

The encapsulation below is not - in any way - proposed by the
current draft as it does not assume that there will be a 16-bit
"slop factor" as you indicate with "????????" in your picture.

--> +--------------------------------+
--> |         Outer DA               |
--> +----------------+---------------+
--> |    Outer DA    |  Outer SA     |
--> +----------------+---------------+
--> |         Outer SA               |
--> +--------------------------------+
--> |Ethertype=TRILL |   Trill Shim  |
--> +--------------------------------+
--> |  Trill Shim    |  ????????     |
--> +--------------------------------+
--> |               DA               |
--> +----------------+---------------+
--> |       DA       |     SA        |
--> +----------------+---------------+
--> |               SA               |
--> +--------------------------------+
--> |    Ethertype   |   Payload     |
--> +--------------------------------+
--> |                                |
--> |         ...........            |

If this was proposed by "the current draft", then it should be
put forward as "an example only" in later versions.  An equally 
valid example may be either of the following:

 +--------------------------------+
 |         Outer DA               |
 +----------------+---------------+
 |    Outer DA    |  Outer SA     |
 +----------------+---------------+
 |         Outer SA               |
 +--------------------------------+
 |Ethertype=TRILL |   Trill Shim  |
 +--------------------------------+
 |  Trill Shim    |      DA       |
 +--------------------------------+
 |               DA               |
 +----------------+---------------+
 |               SA               |
 +----------------+---------------+
 |       SA       |   Ethertype   | 
 +--------------------------------+
 |            Payload             |
 +                                +
 |          ...........           |

OR

 +--------------------------------+
 |         Outer DA               |
 +----------------+---------------+
 |    Outer DA    |  Outer SA     |
 +----------------+---------------+
 |         Outer SA               |
 +--------------------------------+
 |Ethertype=TRILL |   Trill Shim  |
 +--------------------------------+
 |  Trill Shim    |      DA       |
 +--------------------------------+
 |               DA               |
 +----------------+---------------+
 |               SA               |
 +----------------+---------------+
 |       SA       |     0x8100    |
 +--------------------------------+
 | P |  VLAN ID   |   Ethertype   | 
 +--------------------------------+
 |            Payload             |
 +                                +
 |          ...........           |

And there are others.  Note that this is a depiction of what a
frame would look like "on the wire".  It is inappropriate for 
us to make assumptions about what it would look like in memory
for any specific implementation.

In any case, when it goes on the wire, the 16-bit "field" shown
in your example would not be present unless there was reason to
have it. 

--> 
--> If you don't have the 16 bits indicated by ???????, the 
--> hardware will need to realign the inner frame. The frame 
--> is not on the wire. The frame sits in the [frame] buffer 
--> and the memory bandwidth of the packet buffer is today 
--> the MOST important consideration in port ASIC design.

>From the above discussions, you should be able to see the following:

1) It is inappropriate to make assumptions about how frames are 
   stored in memory prior to being put on the wire as we are not
   here to standardize that.

2) It is likely that any such assumptions will be incorrect in
   at least some implementations.

Based on this, your otherwise reasonable argument is invalidated
as it is based on inappropriate, and incorrect, assumptions about
implementations.

--> 
--> More inline
--> 
--> > -----Original Message-----
--> > From: Gray, Eric [mailto:Eric.Gray at marconi.com]
--> > Sent: Wednesday, October 18, 2006 4:57 AM
--> > To: Silvano Gai
--> > Cc: rbridge at postel.org
--> > Subject: RE: [rbridge] FTag
--> > 
--> > Silvano,
--> > 
--> > Let me answer your points in order, leaving out irrelevant issues:
--> > 
--> > <Your Point> N*32 is correct, when including Ethernet options/shims
--> >              (e.g. .1Q or .1S).  This is hardware friendly, because
--> > 		 alignment doesn't change going from one header to another.
--> > 
--> > There are a few different "Ethernet" encapsulations, once we start to
--> > include Ethernet options and the shims associated with those options.
--> > 
--> > First, and I believe most important, the "shim" we are talking about
--> > (in TRILL) is not an Ethernet option (or associated shim).  It's a
--> > "shim" as a separate "layer."  Otherwise, it would be inappropriate
--> > for us to do this work here (in TRILL, as part of the IETF).
--> 
--> From the above picture it should be clear that you should 
--> consider the TRILL header as a whole, and the TRILL header 
--> as only one form of Ethernet encapsulation.

Actually, what should be clear is that the intent is explicitly 
NOT to define an aggregate header type, combining a specific 
Ethernet header and TRILL SHIM.  Not only would doing so limit
the usefulness of RBridges in general, it would also involve our 
doing work that is strictly out of scope in the IETF.

What we need to do is recognize that both tunnel, and native, 
encapsulations may be any defined Ethernet encapsulation that
is supported by the interfaces of any specific implementation.

--> 
--> > 
--> > Second, "Ethernet" headers do not all align the same.
--> > 
--> 
--> Agreed, but that is not my concern, My concern is to not 
--> change their alignment due to the insertion/removal of the 
--> TRILL header.

However, it is my concern.  A key reason for using a layer
model in networking protocols is to separate functions and
their definitions in ways that isolate the workings of one
layer from the changes in another.

--> 
--> > Currently, we are talking about having a single Ethertype defined to
--> > distinguish inter-RBridge traffic from other types of Ethernet frames.
--> > If your implementation produces 32-bit aligned Ethernet frames, using
--> > 802.1Q, then it will similarly produce 32-bit aligned frames - prior
--> > to considering the TRILL shim - using the new Ethertype.
--> > 
--> > We are not proposing a change in any Ethernet frame/header format.
--> 
--> Neither do I.

This is semantics.  You are clearly proposing that we should 
define a new type of Ethernet header - calling it a TRILL
Ethernet header - and that is not what we're supposed to be
doing.

--> 
--> > 
--> > The reference to 802.1Q is slightly disingenuous, because 802.1Q
--> > effectively displaces the existing Ethertype by 32 bits.  802.1Q
--> > "shims" include the Ethertype, because they replace the original
--> > one with 0x8100 and have to include the original in the "shim."
--> > 
--> > I assume that a similar observation could be made about 802.1S.
--> 
--> Yes and a similar observation can be made for MPLS, but all these three
--> do not change the alignment of the fields that follow.

And, as you may have noted from the above, if you're talking 
about frame alignment in a buffer, this depends a lot on the
specifics of any implementation.

As I pointed out above, there is already a "proof of concept"
example in PWE3 Ethernet encapsulation.

--> 
--> > 
--> > In TRILL, we are not replacing the original Ethertype, we are adding
--> > a new encapsulation.  The original Ethertype stays in the original
--> > - re-encapsulated, or tunneled - Ethernet frame.  Therefore, there's
--> > no reason to consider the Ethertype to be part of the TRILL shim.
--> 
--> As I told before you need to consider the whole TRILL header, i.e.
--> 6 bytes of DA
--> 6 bytes of SA
--> 2 bytes of Ethertype
--> 4 bytes of shim
--> ----------------------
--> 18 bytes

Yes, you have asserted this, and I have refuted it.

Hence we are in disagreement on that point and we should let other
people make their own assertions (and refutations) and see how things 
turn out.

--> 
--> Which is 2 bytes short of N*32, i.e. it is not HW friendly.
--> 
--> For this reason I propose that Ethertype + shim should be 
--> N*32 to be HW
--> friendly.
--> 
--> 
--> > 
--> > <Your Point> TRILL should adopt this model for its shim header - the
--> >              Ethertype should be included as part of computing "shim"
--> >              size.
--> > 
--> > In TRILL, we are not replacing the original Ethertype, we are adding
--> > a new encapsulation.  The original Ethertype stays in the original
--> > - re-encapsulated, or tunneled - Ethernet frame.  Therefore, there's
--> > no reason to consider the Ethertype to be part of the TRILL shim.
--> > 
--> > <Your Point> TRILL should optimize the Ethernet case that will cover
--> >              99% of TRILL applications.
--> > 
--> > Coverage is slightly more complete than that.  Considering all of the
--> > layer 2 encapsulations that currently come under an Ethernet umbrella,
--> > TRILL applications are 100% included.
--> > 
--> > I think we are essentially in agreement on this point.
--> > 
--> > <Your Point> TRILL bridges will be implemented mainly in hardware and
--> >              the encapsulation and decapsulation functions need to be
--> >              HW friendly.
--> > 
--> > Okay.  I was not making any other point - except to say that alignment
--> > is usually a software concern, especially in initial design.
--> > 
--> > Let me be blunt, alignment does not factor into initial hardware
design,
--> > but is an issue with hardware re-use complexity.  Bits on a wire are
not
--> > "aligned" - you simply collect them as they arrive, serially.
--> 
--> I agree that bits on the wire are not aligned, but you need to buffer
--> the incoming frame (or at least part of it) until you have enough fields
--> to perform your lookup and decide what to do with the frame. From that
--> point on the frame is not on the wire, it is in the packet buffer and
--> alignment is crucial at the performance we are talking about.

This is perhaps the most specious part of the argument.  While a huge
majority of switch builders use essentially similar buffering models,
at an architectural level, the details of how and where the frames are 
buffered is very much a part of the "secret sauce" of implementations.

Given the number of potential ways of doing input buffering, and the
likelihood that we are going to be able to get enough data on specific
implementations to perform any sort of meaningful statistical analysis,
I see no reason to assign any greater validity to the assumption that
frame alignment is important than to the assumption that it is not.

--> 
--> I hope this time I made my point more clear

It did.  I had thought you were talking about the importance of 
alignment in framing hardware, and you clarified that you were
talking about buffering.

This doesn't change the fact that I think you're wrong, but it 
does help me to understand exactly what you're wrong about.  :-)

--> 
--> -- Silvano
-- [SNIP] --