Subject: Re: Nr in data packets

From: Vernon Schryver <[email-masked]>
Date: Sun, 23 Jan 2000 23:28:10 -0700 (MST)

> From: "W. Mark Townsley" <[email-masked]>

> > You are mistaken about the requirements of the upper layers.  Many
> > protocols carried in PPP frames do not tolerate out-of-order packets.
>
> That is true, which is why we can turn on sequencing when necessary. 

Who is that "we," Kemo Sabe?
"We" cannot be end users configuring their PC's, since they don't
understand out-of-order from in-order.  It cannot even be system
administrators employed by ISP's, since many of them understand the issue
no better than typical end users (i.e. not at all), and the administrators
who do understand will have neither the time to figure out whether sequence
numbers are necessary, nor the tools to determine that necessity.

> ...
> We all know that, by design, IP may reorder packets. I see no reason why
> it is L2TP's business to correct this for tunneled PPP/IP frames.

That is a straw man.  No one said anything about correcting the
order of already mis-ordered IP frames.  
Never mind that in real life, out-of-order UDP/IP packets, out-of-order
TCP/IP segments, and out-of-order IP fragments are all rare.

>                                                                   If the
> reordering of IP packets being detrimental to some broken stacks is
> cause for enabling sequence mode by L2TP, then why limit L2TP traffic to
> this new-found benefit for IP?  We should start using L2TP in sequence
> mode for all IP transmissions. L2TP becomes the packet sequencing
> protocol for the Internet! This is of course obsurd, but it illustrates
> the point that, when carrying IP within L2TP/PPP over IP, it is safe to
> not bother with sequencing...  

I don't understand that either, even after discounting the hyperbolic 
rhetoric

It's true that some PC code is so broken that it does not merely
suffer retransmissions and closed congestion windows from out-of-order
TCP/IP segments, but goes wheels-up.  If you were saying that such
broken systems should not constrain L2TP, I would agree.

It is well known among that miss-ordering TCP/IP packets harms RFC
compliant TCP/IP implementations by significantly reducing throughput,
significantly increasing latency, and increasing wasted bandwidth.  Out
of order TCP/IP segments are required to trigger duplicate ACKs, and
duplicate ACKs cause both retransmissions and partially close the sender's
TCP congestion window.  None of this has anything to do with "broken
stacks," but with the opposite, complaint, good TCP code.

As I said before, a major reason that RFC 1990 MP is far more popular than
"load balancing" or "BF&I" multilink despite the vastly increased
complexity and reduced raw throughput and latency of MP is that TCP/IP
throughput suffers due to the inevitable mis-ordering of TCP/IP segments.
In other words, if it made sense in the real world to have L2TP not
preserve order, then it would also make sense to transmit IP packets in
MP bundles without MP headers.  Do you know of any routers that do that,
and if not, why not?

Unless you define "safe" as reducing performance and increasing wasted
bandwidth, allowing L2TP to mis-order IP packets is a not safe.

> ...
> In fact, there are some cases where sequence checking can actually be
> dangerous on an IP link. Consider the following actual occurance that we
> witnessed at a recent bakeoff. One implementation consisently reordered
> fragmented L2TP packets. This UDP/IP stack was operating properly as IP
> stacks are certainly allowed to reorder packets (and fragmentation is a
> common cause of this).

Since when is IP fragmentation of mis-ordered UDP/IP packets?  Either I
don't understand what you mean, or you are wrong.  Are you assuming that
a lost IP fragment in a first UDP transmission combined with a UDP
retransmission can produce a delayed UDP packet?  If so, perhaps in non-RFC
compliant UDP stacks+applications that cheat on changing the IP ID when
the application sends a new UDP packet (e.g. a retransmission), but I
think such systems are rare.

>                        L2TP Sequence checking was enabled by the
> receiving peer, thus dropping all out of order packets just as you
> suggest. The end result was that no fragmented L2TP packets were ever
> received. 

So one of the L2TP peers was badly broken and was transmitting packets
out of order?  And you infer somehow that preserving order is bad?
That reasoning does not make sense to me.

> At any rate, yes there are indeed cases where preservation of order was
> an important underlying assumption for PPP. For these, L2TP has a method
> for enabling (and disabling) sequence numbers at any time during the
> life of the session. 

The standard problem of standards committee protocols is bloat with
optional features and switches switches that no one will ever use because
they cannot be used in real life.  As far as I can see, any switch that
allows L2TP to mis-order either PPP or upper layer packets is a classic
example of a standards committe misfeature and bloat.  In real life, the
very few if any circumstances in which mis-ordered L2TP frames can be
turned on cannot or will be determined by the people who would have to be
doing the determining.

Vernon Schryver    [email-masked]

---

From: "W. Mark Townsley" <[email-masked]>
Date: Mon, 24 Jan 2000 03:37:40 -0500

> Who is that "we," Kemo Sabe?

Those who implement the protocol. The LNS is in control of sequencing.
Thus the LNS can then make a determination of whether sequencing is
necessary via heuristics, not necessarily via config. So, the protocol
allows for the programmers to design in sequence checking when they deem
necessary. The "dialup user" does not have to be cognizant of this at
all.

> numbers are necessary, nor the tools to determine that necessity.

I have more faith in the network admins than you do. I have worked with
many very bright network admins that certainly understand what packet
ordering is and is not. Perhaps it is a misleading artifact of who my
customer base is, but when I refer to a "user", I am often referring to
a customer who actually understands networking (e.g. *not* the average
DUN user at his windows box, but the network admin configuring the
router/server/etc). 

> order of already mis-ordered IP frames.

Then I misunderstood you.

> broken systems should not constrain L2TP, I would agree.

Yes. I was refuting using broken TCP/IP stacks as a design constraint on
L2TP.

> bandwidth, allowing L2TP to mis-order IP packets is a not safe.

I don't believe L2TP should mis-order packets, but it should not have to
go out of its way to preserve the order of already mis-ordered IP
frames.

> don't understand what you mean, or you are wrong.  Are you assuming that

I don't think you understood. I was referring to fragmentation of an IP
packet transmitted from the client. The circumstance causes the two IP
fragments, each framed as individual PPP packets, to be repeatedly
misordered exiting the LAC (due to fragmentation of one of the two
already fragmented packets at the LAC). It was not actually L2TP that
was misordering the packets, it was the IP stack on the LAC. It did so
because of fragmentation of one of the two initial IP fragments.

Perhaps more detail would help: The smaller of the two IP fragments
coming from the PPP client was getting switched out before the
fragmentation of the larger packet could occur, causing the L2TP packet
to be consistently out of order from the LAC. The key here is that
everyone was operating within spec, and the ultimate result was that the
the packet would never, ever, be received on the other end. Basic
sequence checking of IP assumes that reordering is a random phenomena,
when in fact it is not. IP sequence checking in the face of a consistent
packet misordering situation results in an unrecoverable denial of
service (unless a route change or some such actually restores the order
by chance). 

> That reasoning does not make sense to me.

No one was broken. Preserving order by dropping packets caused a bad
problem.

> they cannot be used in real life.  As far as I can see, any switch that

There is at least some notion of removal of features that never get used
as the document advances the standards track.

What you say about committee-born protocols is true in many regards, and
L2TP is niether immune nor alone in this arena. The upside in our case
is a fairly large number of interoperable implementations, and general
industry acceptance. 

> turned on cannot or will be determined by the people who would have to be

Either the code or the network admins will be doing this. I would not
expect the "average dialup user" to understand the implications of
packet sequencing. 

- Mark