Subject: Re: Nr in data packets

From: Vernon Schryver <[email-masked]>
Date: Tue, 25 Jan 2000 11:14:00 -0700 (MST)

> From: "W. Mark Townsley" <[email-masked]>

> So, a router interface should queue all incoming IP packets while it
> takes a large one and splits it into one or more IP fragments? 

Not exactly.  A router must be largely FIFO for a given UDP or TCP
stream.  An occassional reordering is tolerable and unavoidable,
but consistent re-ordering is broken, cannot be tolerated, and will
eventually be rejected by the market.

Why have some of Cisco routers for many years tried to keep a given TCP
stream associated with a given output port when load balancing over equal
paths?  Why is the standard Cisco response to observations about apparent
inconsistenties or flapping in paths reported by traceroute or ping -R
(LSRR) that ICMP packets get special handling but that TCP/IP packets
generally see a constant path?  Why go to that significant trouble and
expense to associate TCP streams with output ports (or routes) except to
avoid IP re-ordering, and why both about IP re-ordering?  My guess is that
some people at Cisco know of the Bad Things that happen with significant
packet reo-ordering.

Again, if you think that significant re-ordering TCP/IP packets by PPP
routers is tolerable, why doesn't Cisco support "load balancing" or "BF&I"
PPP multilink better?  BF&I requires less processing by the sending
router and an enormous amount less by the receiver than RFC 1990.  BF&I
also has better throughput and latency.

> ...
> The bottom line is this: L2TP data packets carrying PPP/IP is ultimately
> an IP packet within an IP packet. Applying sequencing checking to a
> tunneled IP packet would surely make a the original IP architects
> shudder just as much as reordering PPP frames does for a PPP guru. Each
> are fundamental, opposing, assumptions for each protocol, respectively. 

Maybe you're right, but I bet not.  Shall we ask some?

By the way, do you remember when a lot of IP packets of what is now called
the Internet were carried on x.25 circuits?  My vague recollection is also
that x.25 has a sequence number, albeit rather small.

> Together, we have identified cases where checking packet sequence and
> not checking packet sequence can break the tunneled connection. We could
> compare pathological examples all day. In practice, I have seen only one
> of these occur in a repeatable fashion. If you want to avoid this, my
> recommendation is to not check sequencing on tunneled IP packets. 

I don't understand.  Are you recommending that most L2TP boxes turn off
sequence numbers, only that broken box that can't IP fragment sanely,
or only that broken box and only some of the time?  If the last, when?

Aren't you still patching one pathology, a router that consistently
mis-orders some packets, with another, worse pathology, turning
off L2TP sequencing?

You have not explicitly addressed my implicit question, but I have
inferred what might not be your answer.  Let me make it explicit.
Do you recommend under any circumstances turning off L2TP sequencing for
non-IP packets such as bridging, IPX and PPP CP packets on that wonderful
box that consistently re-orders when IP fragmenting?

Vernon Schryver    [email-masked]

---

From: "W. Mark Townsley" <[email-masked]>
Date: Tue, 25 Jan 2000 16:35:18 -0500

> some people at Cisco know of the Bad Things that happen with significant

It has more to do with what types of packets have optimized switching
paths, and which do not. Further, whether the packet is locally destined
or being switched through the router, and even which interface it is
being switched to. For example, since traceroute is a series of pings to
each node, the packet ends up in a different switching path when the
node is first discovered vs. being switched through to the next node.

Vernon, I'm not disputing that preserving order for a TCP connection is
not a Good Thing. Let me ask, at what point do you suggest we stop
peeking into a packet to determine the flow? I doubt any router looks
inside UDP, then L2TP, then PPP, then IP, then TCP to try and keep the
flow ordered (or even the UDP packet to preserve L2TP flow alone). At
some point, the cost of all of this lookup becomes more painful than the
gain. I would be happy to see someone take as an action item researching
this and providing the results. Even if it is proven that we should
perform all this flow checking, it still cannot be guaranteed (as it
certainly could be configured off).

> Maybe you're right, but I bet not.  Shall we ask some?

Go ahead. I would think that they would consider trying to sustain order
as noble when possible, but checking and dropping packets that were out
of order on receipt as a problem. The latter is what I am concerned
about in L2TP.

> that x.25 has a sequence number, albeit rather small.

Checking for out of sequence packets and dropping them is different than
providing a reliable transport. With the latter you may have some
windowing problems between the reliable transports' operation on top of
one another. The former means that there may be packets which never get
delivered.

A reproducible packet reordering circumstance + IP sequence checking =
packets that never will be received. I don't like the result of this
equation.

Take away either of the items in the sum above, and all packets will be
received. I cannot guarantee that all IP stacks will only have rare,
random, reordering. Or, that they will peek into my UDP packet to
determine L2TP flow.

> mis-orders some packets, with another, worse pathology, turning

I don't think that L2TP operating without sequencing in some
circumstances is a pathology. Also, I'm not 100% convinced that a router
consistently reordering under certain situations is a pathology either.

> box that consistently re-orders when IP fragmenting?

At risk of you trashing the text in RFC2661, I will give you my answer
(from section 5.4):

   The LNS may initiate disabling of sequencing at any time during the
   session (including the first data message sent). It is recommended
   that for connections where reordering or packet loss may occur,
   sequence numbers always be enabled during the initial negotiation
   stages of PPP and disabled only when and if the risk is considered
   acceptable. For example, if the PPP session being tunneled is not
   utilizing any stateful compression or encryption protocols and is
   only carrying IP (as determined by the PPP NCPs that are
   established), then the LNS might decide to disable sequencing as IP
   is tolerant to datagram loss and reordering.

Thus, if you are running IP only (and are running over an IP network as
opposed to L2TP over ATM or some such), then I support disabling
sequence checking after the CPs have finished. For other protocols,
particularly stateful compression, encryption and PPP CPs, I recommend
enabling sequence numbers. That said, most implementations I see today
simply have sequencing off from the start.