Subject: [rbridge] In-order delivery not an issue?

From: gvsm <[email-masked]>
Date: Sat, 4 Oct 2008 05:28:52 +0530

Hi,

Does not present Ethernet by STP families give inorder delivery of
frames for a flow?
Will not this be compromised with this(TRILL) solution or is it
assumed implementation would take care of it?

Thanks,
venkatg
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: Radia Perlman <[email-masked]>
Date: Mon, 06 Oct 2008 00:39:30 -0700

TRILL ought to do in-order delivery, except with an occasional out of
order packet in very rare cases. The 802 solutions also, with low
probability, will occasionally reorder packets. Perhaps one could argue 
that the probability
might be slightly larger with TRILL, but in either case it will be "very 
rare".

Certainly we took the desire for in-order delivery into account in the 
design of TRILL.

Radia

---

From: Dinesh G Dutt <[email-masked]>
Date: Mon, 06 Oct 2008 07:33:50 -0700

One place where TRILL will out-of-order a frame where a 802.1Q won't is for
the case of an unknown unicast vs known unicast. If the first frame is
flooded and the subsequent frame is not, it is possible that they may get
out of order as they follow different paths. With spanning tree, all frames
follow the same path because there is only one path.

I have not run into any customer who thought that this was an issue when I
explicitly raised this point with them.

Dinesh
P.S: I've fixed my emailer to not send emails to rbridge with my personal 
a/c. Apologies for the double posting.

-- 
We make our world significant by the courage of our questions and by
the depth of our answers.                               - Carl Sagan

---

From: Joe Touch <[email-masked]>
Date: Mon, 06 Oct 2008 07:41:57 -0700

> might be slightly larger with TRILL, but in either case it will be "very 

Agreed.

That has been in the problem statement as an assumed requirement from
the beginning of the WG, FWIW. See section 3.1.

Joe

---

From: " G.Venkatasubramaniyan" <[email-masked]>
Date: Mon, 6 Oct 2008 22:29:48 +0530

resending my previous mail ...
Thanks,
venkatg
---------- Forwarded message ----------
From: Venkatsubramaniyan G, TLS-Chennai <[email-masked]>
Date: Mon, Oct 6, 2008 at 7:21 PM
Subject: RE: [rbridge] In-order delivery not an issue?
To: [email-masked], Radia Perlman <[email-masked]>
Cc: gvsm <[email-masked]>

Radia,

Assuming a traffic flow (of given SRC, DST and QoS) happens through
multiple equal cost paths in Rbridge cloud, I think TRILL cannot not
offer guarantees that the frames at egress are in the same order as in
ingress in Layer-2.  The individual paths can have different traffic
conditions and reaction mechanisms which may not give in-order behavior,
which could largely vary.

At present it is assumed this is left for ES ('s higher layer) to handle
out-of-order packets.

If this needs to be handled inside TRILL, there may be a need for
seq-numbering the packets for each flow inside the TRILL cloud. Is it
not?
It may also need an elaborate session management and control protocols
between ingress R-Bridge and egress R-Brdge, so that in-order is
guaranteed.

BTW, do not 802 solutions in *steady state* give in-order delivery for a
given flow?

Thanks a lot,
Venkatg

-----Original Message-----
From: Radia Perlman [mailto:[email-masked]]
Sent: Monday, October 06, 2008 1:10 PM
To: gvsm
Cc: [email-masked]; Venkatsubramaniyan G, TLS-Chennai
Subject: Re: [rbridge] In-order delivery not an issue?

rare".

Radia

> assumed implementation would take care of it?

DISCLAIMER:
-----------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and
intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its
affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily
reflect the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure,
modification, distribution and / or publication of
this message without the prior written consent of the author of this
e-mail is strictly prohibited. If you have
received this email in error please delete it and notify the sender
immediately. Before opening any mail and
attachments please check them for viruses and defect.

-----------------------------------------------------------------------------------------------------------------------
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: "Caitlin Bestler" <[email-masked]>
Date: Mon, 6 Oct 2008 13:40:07 -0400

> From: [email-masked] [mailto:[email-masked]]
On
> Behalf Of Dinesh G Dutt
> Sent: Monday, October 06, 2008 7:34 AM
> To: Radia Perlman
> Cc: [email-masked]; [email-masked]; gvsm
> Subject: Re: [rbridge] In-order delivery not an issue? (resending with
> my correct addr)
> 
> One place where TRILL will out-of-order a frame where a 802.1Q won't
is
> for

The shift from unknown to known is the one case where TRILL could have
an out of ordering scenario that spanning tree alone would not have.

But it should also be noted that TRILL will *reduce* the frequency of
spanning tree bridges forgetting addresses because those bridges will
see fewer addresses.

Benefits from reduction in excess flooding would far outweigh any
extreme
corner case re-ordering potential that TRILL may create.

---

From: James Carlson <[email-masked]>
Date: Mon, 6 Oct 2008 14:39:49 -0400

G.Venkatasubramaniyan writes:
> ingress in Layer-2.

Absent events that cause any of the intermediate nodes to change their
forwarding database, I would expect something like that is true.

However, those events can occur, and the existence of them has nothing
to do with the identification of a given {SRC,DST,QoS} triplet.  If
some node in the middle of the network decides that the best path to
get to DST no longer goes through downstream node X but instead goes
to Y, then packets previously sent to X will race those now being sent
to Y.  If the source packet rate and switching time are 'short'
compared to the time delta between the two paths, then some packets
are bound to arrive out of order.

There's little we can do about that short of deliberately dropping
packets.  It gives us only "rare" and "unintentional" reordering
characteristics.

>  The individual paths can have different traffic

Yes; exactly.

> seq-numbering the packets for each flow inside the TRILL cloud. Is it

I think that'd be a terrible mistake.  It's something we could handle
with options, if it were necessary, but I suspect that worrying about
the narrow cases where this can happen in various networks and trying
to 'fix' all of them is just taking a flamethrower to a gnat of a
problem.  It might be better to worry about the countryside that's
scorched in the process.

(I've seen a variant of this used in other protocols: if the sender
[original encapsulator] *knows* that the encapsulated message can
tolerate reordering, then don't bother numbering it.  If the sender
doesn't know, or knows that it cannot tolerate reordering, then add a
number.  That way, most protocols don't pay the price in overhead and
loss, but [unfortunately] all implementations do pay it in
complexity.)

> BTW, do not 802 solutions in *steady state* give in-order delivery for a

If one can assume *steady-state*, then pretty much everything except a
deliberately malicious node provides in-order delivery.

If we are permitted to assume that, then RBridges are every bit as
non-reordering as are any of the standard 802 bridges.

-- 
James Carlson, Solaris Networking              <[email-masked]>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: "Caitlin Bestler" <[email-masked]>
Date: Mon, 6 Oct 2008 15:11:04 -0400

> conditions and reaction mechanisms which may not give in-order
> behavior, which could largely vary.
> 
> At present it is assumed this is left for ES ('s higher layer) to
> handle out-of-order packets.
> 

I do not believe that is an accurate characterization of Appendix C.
There it states that the mechanism(s) used to ensure that conversations
are not mis-ordered by ECMP is "out of scope".

   When multipathing is used, frames that follow different paths will be
   subject to different delays and may be re-ordered.  While some
   traffic may be order/delay insensitive, typically most traffic
   consists of flows of frames such the re-ordering within a flow is
   damaging. How to determine flows or what granularity flows should
   have is beyond the scope of this document but, as an example, under
   many circumstances it would be safe to consider all the frames
   flowing between a particular pair of end station ports to be a flow.

It might be useful to explicitly state that any flow considered to be 
a "conversation" for 802.3ad purposes SHOULD be similarly treated by
any RBRidge implementing multi-pathing. But such a statement is only
making explicit what I think is already clearly the intent.

---

From: "Donald Eastlake" <[email-masked]>
Date: Mon, 6 Oct 2008 15:32:59 -0400

See below

> which could largely vary.

TRILL does not define what a flow is. If you believe that frames with
the same source and destination MAC address and priority should be
delivered in order, then you would use that as the definition of a
flow and send all the frames in that flow over the same equal cost
path in an RBridge campus just as you would send them all over the
same link in the case of link aggregation. Although TRILL supports
equal cost multipath, it does not fully specify it and the granularity
of flows is one aspect left to the implementer.

> between ingress R-Bridge and egress R-Brdge, so that in-order is

I do not see any need for this to be "handled inside TRILL".

> BTW, do not 802 solutions in *steady state* give in-order delivery for a

Yes, as does TRILL.

> Venkatg

Donald

--
=============================
Donald E. Eastlake 3rd   +1-508-634-2066 (home)
155 Beaver Street
Milford, MA 01757 USA
[email-masked]
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: "Thomas, Matthew R" <[email-masked]>
Date: Mon, 6 Oct 2008 21:54:02 +0100

Donald,
 
I also agree,
 
The phrasing I think is well handled as it is,
 
MT
 
The nice thing about LS protocols is that you get a certain load sharing anyway as you have multiple differing src - dst trees anyway.

________________________________

From: [email-masked] on behalf of Donald Eastlake
Sent: Mon 06/10/2008 20:32
To: G.Venkatasubramaniyan
Cc: [email-masked]
Subject: Re: [rbridge] Fwd: In-order delivery not an issue?

See below

Yes, as does TRILL.

Donald

---

From: Dinesh G Dutt <[email-masked]>
Date: Mon, 06 Oct 2008 14:32:54 -0700

One more point. The TCP/IP RFCs don't talk about maintaining order 
either, but products do it. IIRC, even IEEE's link aggregation doesn't 
specify how to maintain order when spreading frames across the different 
links of a link agg group.

Dinesh

---

From: Dinesh G Dutt <[email-masked]>
Date: Mon, 06 Oct 2008 14:30:51 -0700

Venkat,

Think about how in-order delivery guarantees are offered within IP 
today. Even TCP does not like dealing with out of order frames without 
suffering loss of throughput.

What is done is that we send all frames associated with a flow (and 
different products have different definitions of flow, ranging from {DA, 
SA, VLAN} to the full 5 TCP/IP tuple and more) on the same path thereby 
guaranteeing in order delivery for that flow, in steady state. AFAIK, 
there is no requirement for the network to maintain order across flows.

Dinesh

---

From: James Carlson <[email-masked]>
Date: Mon, 6 Oct 2008 17:30:30 -0400

> making explicit what I think is already clearly the intent.

I'd support that.  Given the amount of wiggle room the 802.3ad
'conversation' definition allows -- it's just the frames between
stations that must maintain order, without necessarily restricting how
one identifies such frames -- I'd even say this could be a safe MUST.

---

From: Joe Touch <[email-masked]>
Date: Mon, 06 Oct 2008 15:39:14 -0700

Dinesh G Dutt wrote:
> either, but products do it.

TCP/IP does talk about NOT requiring order (sometimes referred to as
'sequencing') to be maintained, and explicitly require tolerating it at
the network layer:

791 - sec 1.2
793 - sec 1.5
1122 = sec 1.1.2
1812 - sec 2.2.1

Agreed that these RFCs don't focus on efficiencies afforded when packets
arrive in the order sent, but that's not what is being discussed here.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 06 Oct 2008 17:30:50 -0700

> suffering loss of throughput.

Let's be a little more careful with words here:

	- the Internet does not rely on in-order "guarantees"

	- some protocols react better to out-of-order than others
		e.g., UDP does just fine, as should TCP w/SACK

	- the impact of out of order depends on how spread-out
	the reordering is in some cases
		e.g., TCP should react to small reorderings
		due to multipath fairly well within a RTT;
		it will be more bursty, but should not cut
		the window down just because packets arrive
		out of order

Joe

---

From: ravikumarb <[email-masked]>
Date: Tue, 7 Oct 2008 10:17:17 +0530

Dinesh,

What you are saying is perfectly fine with IP and TCP. But Ethernet is a layer 2 (link layer) protocol. In steady state, Ethernet always delivers packets in order. There are occasional packet losses but not out-or-order packet delivery.

802.3ad mandates in-order delivery for packets belonging to a conversation. In similar lines, TRILL can maintain in-order deliver *only* for packets belonging to a CONVERSATION (by taking the same path), which in our case can loosely be defined as packets from a Source to Destination machine.

Regards,
Ravi

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Dinesh G Dutt
Sent: Tuesday, October 07, 2008 4:01 AM
To: G.Venkatasubramaniyan
Cc: [email-masked]
Subject: Re: [rbridge] Fwd: In-order delivery not an issue?

Venkat,

Dinesh

**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely 
for the use of the addressee(s). If you are not the intended recipient, please 
notify the sender by e-mail and delete the original message. Further, you are not 
to copy, disclose, or distribute this e-mail or its contents to any other person and 
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken 
every reasonable precaution to minimize this risk, but is not liable for any damage 
you may sustain as a result of any virus in this e-mail. You should carry out your 
own virus checks before opening the e-mail or attachment. Infosys reserves the 
right to monitor and review the content of all messages sent to or from this e-mail 
address. Messages sent to or from this e-mail address may be stored on the 
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***

---

From: James Carlson <[email-masked]>
Date: Tue, 7 Oct 2008 11:02:12 -0400

> What you are saying is perfectly fine with IP and TCP. But Ethernet is a layer 2 (link layer) protocol. In steady state, Ethernet always delivers packets in order. There are occasional packet losse…

Agreed, though we're also talking about bridging here, and out-of-
order is at least remotely possible when failures and reconfigurations
are considered.

As long as we don't have to consider recovery from "failure," with
"failure" defined as anything that changes the computed paths in the
network, I think requiring in-order delivery is fine.  In fact, it'd
likely require extra work to achieve out-of-order under those
conditions.

> 802.3ad mandates in-order delivery for packets belonging to a conversation. In similar lines, TRILL can maintain in-order deliver *only* for packets belonging to a CONVERSATION (by taking the same…

802.3ad does mandate it for a conversation, and it seems reasonable
that for the narrow case of ECMP we should recommend it to the extent
possible, but there are multiple issues here.  One is that 802.3ad
doesn't quite define what constitutes a "conversation" except in
circular terms via the definition in 1.4.94: you must keep those
packets in order because they're packets that are kept in order.

(It even seems to equivocate on that point by setting one of the goals
as a "low risk of duplication or misordering," and not just "none.")

The underlying issue, though, is that it's really not feasible to
mandate an absolute here.  We don't have the same simple peer-to-peer
signaling situation that is in 802.3ad, so the same type of "Marker
PDU" mechanism is impractical; the streams are split across multiple
paths, not just multiple links.  Instead, for folks who implement
multipath at all (it's not a required part of the implementation),
they'll need to do whatever they feel is necessary to satisfy the
markets they go into.  That might mean avoiding or disabling ECMP, or
selecting a single path among the available ones on which to send all
"unknown" type packets, or perhaps applying some of the insights in
RFC 2992 to minimize pain.

Given that implementors *must* understand their markets and the
intended use of what they're producing and any additional constraints
that may impose in order to produce competent products (the RFCs alone
are never enough), and that multiple reasonable answers are possible
here, I don't think it's a good idea to nail this issue down in this
document.  A warning about the possible effects of ill-considered
reordering behavior might be reasonable, but constraining the solution
set doesn't sound like a good plan to me, particularly in an IETF
document.

---

From: gvsm <[email-masked]>
Date: Fri, 17 Oct 2008 06:58:35 +0530

I think I have not drafted my question properly and hence restarting
the discussion.  Apologies ...

[1] I am going through a book on Network convergence which lists TRILL
as one of the enabling technologies and more specifically towards the
multipath offering of the protocol [Ref-1] and this is my source of
interest.

[2]I am aware of scope of TRILL. It has clearly stated that this would
introduce frame reordering.

   Multipathing of multi-destination frames through alternative
   distribution tree roots and ECMP (Equal Cost MultiPath) of unicast
   frames are supported (see Appendix C). Multipathing may introduce
   frame reordering as can differing frame priorities or changes in
   network topology.

[3] I think the frame ordering, if its not in the scope of TRILL, the
protocol needs to be extensible to support additional features of this
kind. I am not right? In my understanding Link aggregation works only
at Link level and may not be suitable enough for multipath. I think
though in theory it can be possible the use of Slow Protocols
Multicast address would prohibit in present form. Will this multicast
packets forwarded? I dont think so. There are studies of new protocols
to overcome this limitation[Ref-2].

All I was asking that is the protocol should be extensible for
features like this, something with "options".

Hope I am clear this time.

Thanks a lot,
venkatg

Ref-1 : Data center Networks and Fiber Channel over Ethernet by Silvano Gai
Ref-2 : High speed, Short Latency Multipath Ethernet Transport for
Interconnections, [
http://doi.ieeecomputersociety.org/10.1109/HOTI.2008.13&ei=d-j3SJ2pAaa0qAOS3tHtCg&sig2=UQ4ZsidRizrOTow70xQqdQ&ct=w
]

791 - sec 1.2
793 - sec 1.5
1122 = sec 1.1.2
1812 - sec 2.2.1

Joe

>> From: Venkatsubramaniyan G, TLS-Chennai <venkatg at hcl.in>
>> To: rbridge at postel.org, Radia Perlman <Radia.Perlman at sun.com>
>> Cc: gvsm <g.venkatasubramaniyan at gmail.com>
>>
>>
>>
>> Radia,
>>
>> From: Radia Perlman [mailto:Radia.Perlman at sun.com]
>> Cc: rbridge at postel.org; Venkatsubramaniyan G, TLS-Chennai
>>> rbridge at postel.org
>> rbridge at postel.org

---

From: "Donald Eastlake" <[email-masked]>
Date: Fri, 17 Oct 2008 10:47:37 -0400

See below...

>   frame reordering as can differing frame priorities or changes in

It says "may" not "will". It depends what your standard is for
re-ordering. As stated in Appendix C, if you are going to do ECMP, you
have to decide what your criteria is for a flow. Frames in different
flows can be re-ordered. Frames in the same flow would not normally be
re-ordered. Although this thread has somewhat wandered into Layer-3
areas, I thought this was made clear in earlier answers.

> [3] I think the frame ordering, if its not in the scope of TRILL, the

Where does the spec say that frame ordering is out of scope for TRILL?
I don't think that the definition and granularity of ECMP flows being
out of scope is the same as frame ordering being out of scope. There
was considerable discussion of frame ordering in the early days of
TRILL.

> kind. I am not right? In my understanding Link aggregation works only

Exactly what features?

> at Link level and may not be suitable enough for multipath. I think

I don't know that much about link aggregation but I believe it
currently is actually not an 802.1 protocol but a 802.3 protocol that
only works across multiple single hop equal speed physical paths.
Furthermore, you have to have it implemented in the ports at both ends
of the physical paths.

> packets forwarded? I dont think so. There are studies of new protocols

Indeed, no 802.1 conformant customer bridge will forward any frame
with a destination address in the range 01-80-C2-00-00-00 to
01-80-C2-00-00-0F. If it doesn't understand the protocol the frame
refers to, it is required to drop the frame. If it does understand it,
it processes the frame. It never forwards such frames. There are
exceptions for provider bridges and the like.

> features like this, something with "options".

Again, exactly what "features"?

> Hope I am clear this time.

Well, I don't understand exactly what you want. Can you be more specific?

Sorry,
Donald

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Dinesh G Dutt wrote:
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkjqk5IACgkQE5f5cImnZrtgNgCdF3sxWNIS2ZgMmWGuL5PV/25z
> hF0AoKKoRZwwy7mDdhBo3VbwLCE3386g
> =nMhu
> -----END PGP SIGNATURE-----

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 17 Oct 2008 09:11:49 -0700

gvsm wrote:
> [3] I think the frame ordering, if its not in the scope of TRILL, 

Normally frame ordering is an issue left to the specific 802.1 LAN
that TRILL PDUs are sent over. That is, TRILL trusts the 802.1 and
the lower L2 layer (typically 802.3) to do nothing to introduce
frame mis-ordering within a "conversation" as defined in 802.3ad.

The extent to which TRILL should address is this is with the context
of Appendeix C. *If* an RBRidge is using ECMP (or any form of
multipathing) then it SHOULD avoid splitting any single 802.3ad
conversation over two paths.

That is something that anyone implementing multipathing probably
would have done anyway, but the standard would be more complete
if this was stated.

But on a given path, avoidance of mis-ordering should be left
to the 802.1 and lower layers. TRILL should deal with this *only*
to the extent that when it does multi-pathing it involves multiple
802.1 clouds -- and obviously two distinct clouds can make any
guarantees about how their timing will compare.

If someone wanted to provide multipathing within the context of
a single conversation, whether it involved multiple 802.1 clouds
or not, they would need to create a protocol that provided the
intended sequence information for a reassembly function at the
destination. That protocol could certainly be a TRILL extension,
but there is no real need to discuss such a possible extension
in the base protocol.

-----
Caitlin Bestler
[email-masked] - http://www.asomi.com/CaitlinBestlerResume.pdf
_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: James Carlson <[email-masked]>
Date: Fri, 17 Oct 2008 12:36:43 -0400

> Furthermore, you have to have it implemented in the ports at both ends

That's correct.  In addition, it uses Marker PDUs to try to make sure
that order is preserved when there are link changes within the group.
(Or at least it _can_ use such PDUs; not all implementations include
the feature.)

It's rather different from ECMP, though superficially similar.

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 17 Oct 2008 10:52:23 -0700

> It's rather different from ECMP, though superficially similar.

802.3ad requires that a given "conversation" (a sequence of frames
for which ordering is desired) not be shifted from one link to
another link without taking *some* mechanism to prevent re-ordering. 
Marker PDUs are defined as a standard method of achieving this goal,
but it is explicitly not the *only* solution.

Similarly, any specific ECMP strategy for RBridges would have to
ensure that moving a conversation from one path to another path
did not cause re-ordering. Specific solutions can be documented
in later specifications. There is no need to include them in the
base protocol, simply not moving them except after a path becoming
unavailable is sufficient with some simple timing heuristics.
Anyone who wants more dynamic load balancing even within a
single conversation can propose a specific algorithm in a
later draft.

These are fundamentally the same, its just switching paths
rather than switching links, and because paths are longer
than links the timing is on a different scale. But the fact
that so many link aggregating bridges do *not* use the Marker
PDUs is a strong argument on why the TRILL base protocol does
not need to specify their equivalent. The base protocol is
already a large document, we should not force extra material
into it that can be handled in supplemental specifications.

-----
Caitlin Bestler

---

From: James Carlson <[email-masked]>
Date: Fri, 17 Oct 2008 14:02:06 -0400

> but it is explicitly not the *only* solution.

True.

> did not cause re-ordering.

Note that the same problem occurs when IS-IS recalculates a new path
for a given destination.  It's not a special problem of ECMP -- though
we seem to be talking about it that way -- it's inherent to datagram
routing when there may be topology changes.

> Specific solutions can be documented

I agree ... and this also points towards a "quality of implementation"
issue rather than something that needs to be specified as part of the
base protocol.

> into it that can be handled in supplemental specifications.

I don't want us to do anything like that, either.  I pointed it out to
show that they drove well out of their way to address this problem,
while we are not.

---

From: James Carlson <[email-masked]>
Date: Fri, 17 Oct 2008 13:57:54 -0400

> conversation over two paths.

That's obvious enough that it ought not need to be said.  In other
words, I think that if you don't know at least that much, then you're
going to have trouble all over the place; ECMP is probably the least
of your worries.  The RFC should be a protocol spec, not a tutorial on
networking.

However, that's not sufficient in comparison to 802.3ad, because you
*can* get reordering within a single conversation without ever
splitting it across two paths.  This case can occur if you move
(forced or otherwise) a given conversation from one path to another:
the packets sent on the 'old' and 'new' paths race each other to the
destination, and you may have some trivial reordering as a result
until steady state returns.  (Max delay across the network)

Of course, the same thing happens if you don't have ECMP and you have
a change in the IS-IS layer that causes you to choose a different path
for an entire destination.

In 802.3ad, they use a Marker PDU to fix this problem (since the
underlying links may have some buffering that ends up behaving the
same way as delays in those separate ECMP paths), but we don't have
that.  We're stuck with trivial cases (all relating to link and
RBridge state transitions) that can introduce modest amounts of
reordering in some cases.

But so what?

> would have done anyway, but the standard would be more complete

Maybe ... though I think stating the obvious tends to bloat the
document and introduce other risks.  (Such as a mistaken belief that
the RFC is "all" that you would ever need to know in order to
implement.)

> but there is no real need to discuss such a possible extension

Agreed; a sequencing mechanism is a mistake we can certainly make
later.  ;-}