Subject: RE: Few questions on L2TP..

From: Vipin Jain <[email-masked]>
Date: Tue, 13 Mar 2001 23:16:06 -0800

Mark,

Agree with most of your ideas. I wanted to add the following:
Querying or probing peer about your state is different from asking peer to
reset sequence number and retransmit control packets which has not been
acknowledged. Following can be fairly easily implemented and backward
compatible as well:

* After fail over/shutdown on LNS (or LAC) when redundant one takes over, it
initiates a packet on tunnel with Ns=0, Nr=0 and "reset sequence bit" set.
* LAC (or LNS) responds with first acknowledged packet with Ns=0 Nr=0 and
"reset sequence bit" set
* LNS (or LAC) responds to it with Ns=0 Nr=0 without "reset sequence bit"
set.
* After having sent a packet with Ns=0 and Nr=0 and "reset sequence bit"
set, if LAC or LNS receives a packet with Ns or Nr with NOT EQUAL to ZERO it
considers it out of sequence.
* They can then start exchanging control packets as they were before.
* Data plane can use the same logic if it wishes.

thanks,
vipin

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Tuesday, March 13, 2001 10:51 AM
To: Vipin Jain
Cc: 'Keyur Parikh'; Andy Koscinski; [email-masked]
Subject: Re: Few questions on L2TP..

I don't disagree that a signalling mechanism could reduce the overhead
of state checkpoints for failover operations. However, it is also true
that it is *possible* to be fully redundant without changes to the
protocol (this has been proven by example from at least one vendor I
know of, though I do not know personally at what level of scale). 

It is certainly easier and quite likely more scalable to perform
failover if you can query your peer for information. If there is enough
interest and participation to support this I think it is a very good
idea. However, if we are going to change the protocol, I think that
there are more extensible ways to do this than setting a bit in the
header that effectively overloads the meaning of one or more existing
control messages.

To illustrate, here are a couple of techniques I came up with off the
top of my head while out jogging this afternoon:

1. If you must have a single message "probe," then define a new message
type that is always sent with Ns = 0, Nr = 0 and tunnel ID = 0. For
existing implementations, this matches the transport level signature of
an SCCRQ and should be immediately passed from the transport layer to
the L2TP protocol layer for processing. Here, the new message type will
either be rejected, or accepted if the new feature is supported. Within
the message you could request whatever information you wanted, for
whatever tunnels you wanted, via specific AVPs. You are limited only by
the 1500 byte maximum size of a control message (which may come into
consideration if for scaling purposes you wish to concatenate
information for each posible open tunnel within a single message). There
are obvious security considerations to this method if you are sending
this message over an unsecured transport. 

2. A more secure, and perhaps a little more considerate, solution would
be to go ahead and setup a new control connection with the peers you
wish to resync with. After the connection is established, you can send
control messages and AVPs asking for resync information for whatever
tunnels you choose. This is actually quite similar to #1 except that it
has some protection by allowing tunnel authentication to occur before
revealing everything the box knows about its current state to a peer. It
also preserves the existing control connection setup logic that we all
have in place now.

Other improvements when operating in a failover-capable mode might
include:

- Advertisement at tunnel setup of the ability for a peer to announce
certain state after a failover, this includes what types of values it
has the ability to respond with when requested. This allows the ultimate
level of flexibility and backwards compatibility as the peer can decide
how much state it must keep on backup nodes and how often checkpoints
must occur based upon its peer's ability. Of course, this flexibilty and
compatibility adds complexity for us programmers, but our users would
probably be better served by something like this as it does not require
both sides to participate to be fully redundant on one side -- it just
affords better scalability when the other side participates. Also, it
scales to values beyond simply Ns/Nr synchronization. Other values that
are difficult to checkpoint, such as connect time, byte/packet counters,
etc. all could be very important in a full-featured redundant system.

- Leaving sessions persistent in the face of an abnormal control
connection disconnect in hopes that a backup control connection will be
forthcoming - either from the same box (e.g. after a reboot) or another
box waiting in the wings. We would have to be very careful here so as
not to keep unwanted sessions hanging around (for security, billing, and
resource allocation considerations). 

As I said, these are just a couple I thought of off the bat. I'm sure
that there are other techniques as well, perhaps one that is better than
either of these. 

- Mark

Vipin Jain wrote:
> 
> Have been following this thread for a while..
> 
> I have to agree with Keyur's idea of having bit in common-header, which
can
> be examined by control plane (may be data plane if it wishes to incur the
> overhead of examining every single packet) to see if peer seeks to reset
the
> sequence numbers. I believe there are some protocols that already do this.
> It helps them to have planned shutdown etc., which may not be truly
relevant
> in L2TP. Basic idea still remains same: to have no impact on the end
users,
> should something happen.
> 
> May be Mark argues that it is not required because an implementation can
> smartly get away with saving states and sequences of various
> tunnels/sessions. IMHO, it doesn't harm to provision this in protocol! Can
> be backward compatible too.
> 
> thanks,
> -- vipin
> 
> -----Original Message-----
> From: Keyur Parikh [mailto:[email-masked]]
> Sent: Friday, March 09, 2001 11:18 AM
> To: W. Mark Townsley; Keyur Parikh
> Cc: Andy Koscinski; [email-masked]
> Subject: RE: Few questions on L2TP..
> 
> -----Original Message-----
> From: W. Mark Townsley [mailto:[email-masked]]
> Sent: Wednesday, March 07, 2001 2:17 PM
> To: Keyur Parikh
> Cc: Andy Koscinski; [email-masked]
> Subject: Re: Few questions on L2TP..
> 
> Keyur Parikh wrote:
> >
> > ok, I am glad I included the excerpt from the RFC, I assumed you were
> aware
> > of the background.  While on clarification, you should also clarify what
> the
> > LNS should do, if it had enabled sequencing, but the LAC is sending data
> > messages without the inclusion of the Ns (i.e drop the call, disable
> > sequencing on its end etc..)
> 
> OK.
> 
> >
> > comment on your following statement
> > "
> > An implementation that is performing any sort of reordering MUST be able
> > to handle packet loss. This should be clear from RFC2661.
> >
> > This includes loss of a single packet, 1000 packets, 65535 packets, or
> > more. If you stop at a given sequence number, switchover, and pickup at
> > 0, at worst you look like you dropped a bunch of packets, or sent a few
> > duplicates. Your tunnel peer may discard some packets on receipt, queue
> > some packets waiting for the "lost" one, etc. but the numbers should
> > sync up fairly quickly."
> >
> > KRP - ok, so what you are saying is that it is ok to lose  up to 65k
data
> > packets, but a sequence number resetting mechanism is unwarranted.
> > Interesting!
> 
> Precisely. If you *were* to draw a line at how much percieved data loss
> had occured at the LNS (or LAC) was acceptable, where would you draw it?
> Fairly impossible, given the variety of data rates for L2TP and
> considering that even an "in-order" packet is really just *perceived* to
> be in order. Even without failover, you could have flushed exactly
> 65,535 packets down the tubes somewhere in transit and never have even
> realized it! Unreliable data sequencing is just that, unreliable. It's a
> shot at *trying* to get packets in order within a window, or at least
> detecting that they were out of order within that window.
> 
> KRP - I disagree with your premise. Data loss due an occasional "blip" in
> the network is acceptable but data loss due lack of protocol support is
not
> acceptable.  IMO, the goal of the protocol specification should be to
> specify a mechanism by which a best effort is made to minimize data loss
in
> this situation.  Keep in mind the amount of data loss or more precisely
the
> "maximum time of service outage" requirement can vary a lot among carriers
> based on the types of SLA they provide. So you can't make assumptions
about
> what is acceptable or not.
> >
> > Also on the point of expanding the protocol for explicit signaling -
IMO,
> > inband mechanism with a bit in the common header is more efficient. This
> > way, any control message (call control, tunnel control or Hello) can be
> used
> > for this purpose.  Obviously, all the race conditions have to be worked
> out.
> 
> I think there are even more ways to skin this cat....
> KRP - Well I am glad that you finally agree that "this cat needs to be
> skinned". The option that you provided (the obvious one) to keep the
standby
> in perfect sync with the primary has following problems:
>     -  in the wireless space, we have to support hundreds of setup and
> teardown per second - and keeping the standby in perfect sync with control
> and specially data sessions at wirespeed - just does not work very well.
>     - the second problem is that this solution does not fit very well with
> the 1 to N redundancy model.
> >
> > -----Original Message-----
> > From: W. Mark Townsley [mailto:[email-masked]]
> > Sent: Wednesday, March 07, 2001 11:07 AM
> > To: Keyur Parikh
> > Cc: Andy Koscinski; [email-masked]
> > Subject: Re: Few questions on L2TP..
> >
> > Keyur Parikh wrote:
> > >
> > > >Repeat after me: "Data messages do not need synchronizing", "Data
> > > >messages do not need synchronizing", "Data messages do not need
> > > >synchronizing"...
> > >
> > > OK, then how do you explain the following from RFC2661 (section 5.4)
> > >
> > > "
> > > Unlike the L2TP control channel, the L2TP data channel does not use
> > >    sequence numbers to retransmit lost data messages. Rather, data
> > >    messages may use sequence numbers to detect lost packets and/or
> > >    restore the original sequence of packets that may have been
reordered
> > >    during transport.  The LAC may request that sequence numbers be
> > >    present in data messages via the Sequencing Required AVP (see
Section
> > >    4.4.6). If this AVP is present during session setup, sequence
numbers
> > >    MUST be present at all times. If this AVP is not present,
sequencing
> > >    presence is under control of the LNS. The LNS controls enabling and
> > >    disabling of sequence numbers by sending a data message with or
> > >    without sequence numbers present at any time during the life of a
> > >    session. Thus, if the LAC receives a data message without sequence
> > >    numbers present, it MUST stop sending sequence numbers in future
data
> > >    messages. If the LAC receives a data message with sequence numbers
> > >    present, it MUST begin sending sequence numbers in future outgoing
> > >    data messages. If the LNS enables sequencing after disabling it
> > >    earlier in the session, the sequence number state picks up where it
> > >    left off before.
> > >
> > >    The LNS may initiate disabling of sequencing at any time during the
> > >    session (including the first data message sent). It is recommended
> > >    that for connections where reordering or packet loss may occur,
> > >    sequence numbers always be enabled during the initial negotiation
> > >    stages of PPP and disabled only when and if the risk is considered
> > >    acceptable. For example, if the PPP session being tunneled is not
> > >    utilizing any stateful compression or encryption protocols and is
> > >    only carrying IP (as determined by the PPP NCPs that are
> > >    established), then the LNS might decide to disable sequencing as IP
> > >    is tolerant to datagram loss and reordering.
> > > "
> > > The above clearly indicates that the Ns value for each end is present
at
> > the
> > > other end to re-order and detect lost packets. Also, if the LNS
disables
> > it
> > > and then re-enables it, the sequence number MUST start from where it
> left
> >
> > There is no "MUST" in this sentence:
> >
> > "If the LNS enables sequencing after disabling it earlier in the
> > session, the sequence number state picks up where it left off before."
> >
> > But, having seen this text again here, I certainly agree that it would
> > not hurt for it to be clarified a bit overall. I will put this on my
> > list of things to do for l2tpbis. Thanks.
> >
> > - Mark
> >
> > > off.  THIS IS ALL STATEFUL INFORMATION THAT NEEDS TO BE RESET (I used
> the
> > > word resynch to really mean reset). I am fully aware of the difference
> in
> > > the use of the sequence numbers for control and data channel.
> > >
> > > -----Original Message-----
> > > From: W. Mark Townsley [mailto:[email-masked]]
> > > Sent: Tuesday, March 06, 2001 8:22 PM
> > > To: Keyur Parikh
> > > Cc: Andy Koscinski; [email-masked]
> > > Subject: Re: Few questions on L2TP..
> > >
> > > > Just a thought: How about ensuring that your hot standby router has
> full
> > > > state before even acknowledging the control message from your
primary?
> > > > This doesn't require any changes to the protocol.
> > > >
> > > > KRP - This is not a very scalable solution.  Specially for the
session
> > > data
> > >
> > > Repeat after me: "Data messages do not need synchronizing", "Data
> > > messages do not need synchronizing", "Data messages do not need
> > > synchronizing"...
> > >
> > > > messages. IMO, the RFC2661 should state what the LNS MUST do when it
> > > > receives session data messages without Ns when it was enabled by the
> > LNS.
> > > I
> > >
> > > Sequencing on data packets within any tunneling protocol is a tricky
> > > subject.
> > >
> > > IMHO, the LNS SHOULD accept them. One could argue that the LNS should
> > > have this wiggle room if it "knows" that sequencing is vital for the
> > > particular state of the protocol being carried or not. I suppose it
> > > would not hurt to make this more clear in the spec, though there are
no
> > > easy answers here.
> > >
> > > > agree with andy that some mechanism should be recommended for
> resetting
> > > the
> > > > stateful information between LAC and LNS.  How about assigning a bit
> in
> > > the
> > > > common header for this purpose. Anytime either end want to reset the
> > > > applicable sequence numbers they can set this bit in the control
> > message.
> > >
> > > And then you have to check this bit for every single data packet just
to
> > > enable the failover feature? Probably not a good choice. Error
recovery
> > > should not slow down your data packet performance.
> > >
> > > > This should trigger the other end reset its internal variables for
the
> > > > control and all the data session of the tunnel.  Again this is just
an
> > > > initial thought.
> > >
> > > If we have the freedom to modify the protocol, this can probably all
be
> > > done with explicit signalling. Also, you could decouple the tunnel
> > > control connection and sessions within if you wish. Bottom line, we
> > > didn't consider this when designing the protocol at the beginning. If
we
> > > are going to modify the protocol to handle semi-full state failover,
> > > then we should step back and approach it from a fresh perspective.
> > >
> > > - Mark

---

From: "W. Mark Townsley" <[email-masked]>
Date: Wed, 14 Mar 2001 11:50:46 -0500

Vipin Jain wrote:
> acknowledged. 

OK, so you are suggesting that we tell a peer to *start over* at a new
sequence number. Make sense, except that (1) you have to ensure that
there are no local messages that you may have acknowledged but did not
transfer state of to your backup node yet (possible, but if we really
want to talk about scaling then this can be limiting), (2) this requires
your peer to poke down into its transport level unsent queue and
resequence messages (nasty layer violation), (3) this is not at all
extensible, (4) most importantly, this is NOT backwards compatible
(illustrated below).

> Following can be fairly easily implemented and backward
> compatible as well:
> 

An implementation that does not support the reset bit could consider
this a valid "future" packet and simply drop it. Again, run your logic
with Ns=100, Nr=100 and tell me if it works in a backwards compatible
manner. There is NOTHING SPECIAL about Ns=0 and Nr=0 to the L2TP
transport unless the TunnelID=0 as well (the zero TunnelID being the
more important of these fields).

Further, an implementation MAY update its resend queue with the Nr
present in an out of order packet (see the l2tpbis draft). So, your Nr=0
could end up inadverdantly acknowledging a lot of packets. I know of at
*least* one widely deployed implementation that does this, so get ready
for some hard to figure out bug reports if you deploy this non-standard
extension in a multivendor environment!

> * LNS (or LAC) responds to it with Ns=0 Nr=0 without "reset sequence bit"

The following steps are not completely clear to me, but it doesn't
matter as the step above breaks backwards compatibility already.

> * Data plane can use the same logic if it wishes.

There is no concept of "replying" on the data plane, so I don't see how
this same logic can be applied. Also, treating Ns=0 as somehow "special"
on the data plane is as incidious as it is on the control plane. Nr on
the data plane is just a reserved field, you cannot use it for anything.

If you just want to set an "R-bit" ala the old flow control days to
reset your requence numbers, fine, I guess that would work to signal
your peer to accept an out of order packet (assuming it is even
checking). But, this adds a bit to the data plane and makes the data
header even more complex than it already is. I still think for the data
plane you could just start over at 0 and let the peer drop a couple of
packets (again, assuming it is even checking sequence numbers).

---

From: Ignacio Goyret <[email-masked]>
Date: Wed, 14 Mar 2001 10:18:49 -0800

I normally hate jumping into a heated discussion, but here are my $.02:

Abusing Ns=0 or adding a "reset" bit is just a very bad idea. It makes
L2TP so much more vulnerable to attacks. It also negates any authentication
that may have been negotiated between the two endpoints: how can I verify
that the request to resync is from a bona fide source? Finally, it is a hack
that has been demonstrated to fail (guaranteed!) in some cases.

A *MUCH* better solution is to add new commands to the protocol so that
the failed end can obtain all the resync information over a _new_ tunnel,
as Mark suggested. Whether the remote end supports or not these new
commands can easily be determined by a judicious use of the M-bit or
by using a different number in AVP 2, or even with a new AVP (resync capable)
added to the SCCRQ and SCCRP. This is a significantly more robust design
since it will allow you to select different strategies depending on the
capabilities of the remote endpoint.

If you were to implement something like the Ns=0 hack, I think you have a
duty to warn your customers that your box is not 2661-compliant and that
this non-compliance may lead to unexplained behavior and failures.

---

From: Bill Palter <[email-masked]>
Date: Wed, 14 Mar 2001 10:28:17 -0800

> extensible, (4) most importantly, this is NOT backwards compatible

  I worked on an implementation that supported failover, and played around 
with a few different methods of trying to trick peers into resyncing, I tested
it at several bakeoffs and basicly we found that for the data channel nothing
really needed to be done. starting over at a random number worked as well
as keeping state, when we had flow control (before it was removed) we would
just set the R bit and everything was happy.

  The control channel was a different story, the only thing that worked well 
was to keep state across all the possible backup nodes, anything else risked
loosing control messages that might have been important to the state of the 
tunnel. It was not very difficult to identfy the key locations in the protocol
that case state to need to be distributed.

Bill Palter