Subject: Few questions on L2TP..

From: Keyur Parikh <[email-masked]>
Date: Mon, 5 Mar 2001 16:55:31 -0500

1) Since there is no protocol support to re-synchronize the tunnel and data
message sequence number - how does the LAC go about recovering a tunnel and
call session after a switchover?  For tunnel control channel will the
following work?
        LAC                            LNS
                Hello (Ns,Nr=0)
           ------------------------------>
                ZLB (Ns=x,Ny=y)
            <-----------------------------
The LAC can then determine its state variables based on the ZLB.
 
For sessions, what happens if the switched in component at the LAC does not
set the sequence numbers, but the call was using sequence number prior to
failure. Will the LNS clear the call?
 
2)Is there a another method for synchronizing the negotiated LCP options at
the LAC? is snooping the only alternative? 
 
regards,
keyur

Keyur Parikh
Megisto Systems, Inc.
Ph: 301-444-1723
email: [email-masked] <mailto:[email-masked]>

---

From: Andy Koscinski <[email-masked]>
Date: Wed, 7 Mar 2001 23:54:26 -0800

responses to Mark prefixed "andyk>" below.

> Andy Koscinski wrote:
> >
> >In my understanding this dicussion is about how to
> >support a "hot standby" LAC, which was not addressed
> >by the rfc 2661, but is a relevant question, IMHO.
> >
> >mark> 5. I assume by "switchover" you mean some sort of 
> active failover
> >technique. This is, in fact, outside the scope of RFC2661.
> >andyk> yes
> >
> >more comments inline, marked "andyk"
> >
> > > > 1) Since there is no protocol support to re-synchronize the
> > > tunnel and
> > > > data message sequence number -
> > > >    how does the LAC go about recovering a tunnel and call
> > > session after
> > > > a switchover?
> > > >
> > > >    For tunnel control channel will the following work?
> > > >
> > > >        andyk> in the standard rfc2661 it would not work,
> > > because the Ns
> > > > and Nr
> > > >        between the LAC and the LNS would not 
> synchronize after the
> > > > switch over
> > > >        and the tunnel would go down.
> > > >
> > > >
> > > >                LAC                            LNS
> > > >                        Hello (Ns =0 ,Nr=0)
> > > >                        ZLB (Ns=x,Nr =y)
> > > >        andyk> yes, in the above msg exchange, the LAC and
> > > the LNS could
> > > > synchronize
> > > >        their msg sequence counters  (and Amber Networks L2TP
> > > > implementation does,
> > > >        which is an extension to the rfc2661).
> > > > >KRP - andy, I agree in theory it should work.
> > >
> > > This could be dangerous.
> > >
> > > It seems that you are "probing" with a Hello message that 
> you think is
> > > invalid, and expecting the peer to send a ZLB ack in response
> > > and obtain
> > > information from this. In the nominal case, this could 
> work, but you
> > > have to consider the edge conditions as well. You don't 
> want to design
> > > something that works "most" of the time, particularly when a
> > > failure can
> > > bring down an entire tunnel with thousands of sessions within it.
> >
> >andyk> this was just an initial thought. I think we can design
> >something which works in all cases. Do you think we need an 
> extension to
> >the rfc 2661 to support a hot standby LAC?
> >
> > >
> > > - The Ns sequence space is a wrapping 16-bit value. It 
> begins with 0,
> > > but after that there is no signifigance to 0 other than it
> > > occurs again
> > > some time in the future... Probably beyond your testing
> > > procedures so it
> > > "looks" like this works fine. Think of choosing, say, 100 
> for your Ns
> > > Hello probe and think about whether your method will always work.
> >
> >andyk> yes, we know that. please note, that Ns AND Nr = 0, for which
> >the probability of occuring in the same Hello msg is extremely low.
> >All what such Hello (Ns=0, Nr=0) message could mean is, that
> >the LAC and LNS synchronize their tunnel control sequence counters
> >(and do not bring a tunnel down), which does not hurt anybody.
> 
> Here is an example of hurt: one side of your tunnel has sent 
> out a bunch
> of ICRQs, you acknowledged them with the probing Hello 

andyk> what you call here the probing Hello is a representation
of a new explicit signalling to request a sync, which you propose in 
one other email. In my proprietory prototype this hello (Ns=0,Nr=0) 
would not ack any ICRQ's. The ICRQ's would sit in the transmit 
queue ready to go again. 

> but never
> actually received the messages. So, the side that sent the ICRQs is
> still waiting for the ICRPs, perhaps forever. There is no 
> guarantee that
> an implementation will timeout and resend these ICRQs 

andyk> in my prototype (not a standard rfc 2661), these ICRQ's
will be re-sent, because they not have been acked yet.

> unless the
> protocol being tunneled has some sort of timeout or keepalive that
> resets the link. This is just an example, the same would apply to any
> control message which an implementation expects was delivered to L2TP,
> not just acknowledged by the transport (which is just about all of
> them).
> 
> In general, the probability argument just means that it will be that
> much more difficult to discover the problem when it occurs in a
> customer's network.

andyk> I don't think there is a problem. In case a hello (Ns=0, Nr=0)
occurs in a natural sequence (no "switchover"), both tunnel ends
will resync (there was no need for that). The minimal "negative" 
impact of such overlap will be, that a few ICRQ's from your example will
not be acked and will go again. The receiving side can handle message
repetition, thou. The next ctrl message after the Hello (Ns=0,Nr=0) 
will ack the ICRQ's.

> 
> But let's assume for a moment that this is OK.
> 
> First, consider the simple and perhaps likely case of bringing up
> sessions on a tunnel, no sessions brought down, and all you 
> are doing is
> sending keepalives. Since both sides start sequence numbers at 0, and
> both sides send the same number of messages, Ns and Nr very well may
> track with one another. If both sides of the link are 
> configured to send
> hellos at the same interval, and neither performs any optimizations
> about not sending hellos if one was received, then based on 60 sec
> intervals you have a shot at sending a "real" Hello Ns=0 Nr=0 packet
> every 45 days or so. Less for the first hit depending on the number of
> sessions you bring up.
> 
> Second (and more importantly), it is fully acceptable for an
> implementation to inspect the Nr of a packet for its own retransmit
> queue, even when the Ns is not the value it wants. So, everytime you
> probe, you are very likely telling your peer: "I have received all
> packets you sent me up to 0." 

andyk> upon reception of Hello (Ns=0,Nr=0) my prototype would not assume, 
that all packets up to 0 have been received. 

> Which means, depending on your
> implementation, all messages between 32767 and 65535 could get
> acknowledged via your probe (depending on your 
> implementation). Messages
> beneath this value do not, and your peer will retransmit them as you
> would like it to do. The end result is that *sometimes* lost messages
> will get retransmitted after the 0,0 Hello probe, and *sometimes* they
> will not. 
> You can imagine how difficult indeterminate behavior such as
> this would be to debug in the field.
> 
> >
> >The thought here was, that a mechanism is needed to synchronize
> >the tunnel control message counters between the LAC and LNS after
> >a "switchover" to a hot standby LAC happened. If you do not like
> >Hello (Ns=0, Nr=0), what's better in your opinion?
> 
> Just a thought: How about ensuring that your hot standby 
> router has full
> state before even acknowledging the control message from your primary?
> This doesn't require any changes to the protocol.

andyk> yes this is an alternative, which I have choosen not to persue,
because IMO it's not scalable and has a negative performance imapct.

> 
> >
> > >
> > > - The only way to be sure you are sending an Ns that will be
> > > dropped and
> > > acknowledged is to send Ns equal to the last acknowledged
> > > packet for the
> > > tunnel. However, I *think* your intended application 
> prohibits this as
> > > it is coming up after a "switchover" and has no sequence
> > > number state.
> >
> >andyk> yes, the "hot standby" is talking after a "switchover"
> >and has no sequence number state, but does have the tunnel id
> >and session id of all active tunnels and sessions
> >and the question is how to keep the hello's
> >going on in the known tunnels, so they'll stay up after the 
> "switchover".
> 
> More than just hellos, but additional calls over the tunnel. I am
> guessing your particular application doesn't care about this so much
> though... e.g. you typically bring up all your tunnels and sessions at
> the beginning and don't expect any further activity beyond the
> keepalive. All l2tp tunnels are not like this though.

andyk> my proprietory prototype accepts a loss of a session or tunnel in
a transient state. such session or tunnel request will be repeated
after a time out. 
> 
> >
> > >
> > > - The Nr of zero in your probe could inadverdantly acknowledge
> > > outstanding packets on the tunnel (packets that you have 
> not recevied,
> > > but your peer will think you have received).
> > >
> > > - A peer does not have to respond to a Hello message with a ZLB.
> > >
> > > - Even if the Hello message does trigger a ZLB, a "real"
> > > control message
> > > could be received before the ZLB is (packets may be reordered in
> > > transit).
> > >
> > > - Think about what happens if/when both sides try to sync with one
> > > another.
> > >
> > > There are no requirements for "timeouts" of states in 
> L2TP. The L2TP
> > > state machine very much depends on the reliable control
> > > channel beneath
> > > it to deliver ALL packets. I see the possibililty for 
> packets to be
> > > erroneously acknowledged here, which places additional 
> requirements on
> > > the L2TP state machine.
> >
> >andyk> all of your cases thought about and clearly we cannot 
> implement
> >a "hot standby" LAC taking an orthodox rfc 2661 position. 
> But with some
> >minor modifications, IMHO we could.
> 
> Yes, I expect you could, but BOTH sides may have to participate,
> depending on how much state you are willing to save.

andyk> yes, BOTH sides have to participate. I am affraid
I did not say this earlier, thx for pointing out this omission.

> 
> This is all just a matter of where you draw the line between 
> state that
> must be saved on your standby box and the state which you plan on
> recreating at switchover. If you can save ALL state, then you can run
> with an orthodox RFC2661 today. 

andyk> yes (but taking a performance hit).

> If you can change the  protocol at will,
> you could get away with very little saved protocol state. It's just a
> matter of where you draw the line. 

andyk> I think the minimum requirement is to save the state of all
tunnels and sessions one time after they fully came up (and to update
the hot standby after they cleared).

> How much information you 
> can request
> from your peer about connectivity when you come to life and takeover.
> 

-andy

---

From: Keyur Parikh <[email-masked]>
Date: Fri, 9 Mar 2001 14:17:58 -0500

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Wednesday, March 07, 2001 2:17 PM
To: Keyur Parikh
Cc: Andy Koscinski; [email-masked]
Subject: Re: Few questions on L2TP..

Keyur Parikh wrote:
> 
> ok, I am glad I included the excerpt from the RFC, I assumed you were
aware
> of the background.  While on clarification, you should also clarify what
the
> LNS should do, if it had enabled sequencing, but the LAC is sending data
> messages without the inclusion of the Ns (i.e drop the call, disable
> sequencing on its end etc..)

OK.

> 
> comment on your following statement
> "
> An implementation that is performing any sort of reordering MUST be able
> to handle packet loss. This should be clear from RFC2661.
> 
> This includes loss of a single packet, 1000 packets, 65535 packets, or
> more. If you stop at a given sequence number, switchover, and pickup at
> 0, at worst you look like you dropped a bunch of packets, or sent a few
> duplicates. Your tunnel peer may discard some packets on receipt, queue
> some packets waiting for the "lost" one, etc. but the numbers should
> sync up fairly quickly."
> 
> KRP - ok, so what you are saying is that it is ok to lose  up to 65k data
> packets, but a sequence number resetting mechanism is unwarranted.
> Interesting!

Precisely. If you *were* to draw a line at how much percieved data loss
had occured at the LNS (or LAC) was acceptable, where would you draw it?
Fairly impossible, given the variety of data rates for L2TP and
considering that even an "in-order" packet is really just *perceived* to
be in order. Even without failover, you could have flushed exactly
65,535 packets down the tubes somewhere in transit and never have even
realized it! Unreliable data sequencing is just that, unreliable. It's a
shot at *trying* to get packets in order within a window, or at least
detecting that they were out of order within that window.

KRP - I disagree with your premise. Data loss due an occasional "blip" in
the network is acceptable but data loss due lack of protocol support is not
acceptable.  IMO, the goal of the protocol specification should be to
specify a mechanism by which a best effort is made to minimize data loss in
this situation.  Keep in mind the amount of data loss or more precisely the
"maximum time of service outage" requirement can vary a lot among carriers
based on the types of SLA they provide. So you can't make assumptions about
what is acceptable or not.
> 
> Also on the point of expanding the protocol for explicit signaling - IMO,
> inband mechanism with a bit in the common header is more efficient. This
> way, any control message (call control, tunnel control or Hello) can be
used
> for this purpose.  Obviously, all the race conditions have to be worked
out.

I think there are even more ways to skin this cat....
KRP - Well I am glad that you finally agree that "this cat needs to be
skinned". The option that you provided (the obvious one) to keep the standby
in perfect sync with the primary has following problems:
    -  in the wireless space, we have to support hundreds of setup and
teardown per second - and keeping the standby in perfect sync with control
and specially data sessions at wirespeed - just does not work very well.
    - the second problem is that this solution does not fit very well with
the 1 to N redundancy model. 
> 
> -----Original Message-----
> From: W. Mark Townsley [mailto:[email-masked]]
> Sent: Wednesday, March 07, 2001 11:07 AM
> To: Keyur Parikh
> Cc: Andy Koscinski; [email-masked]
> Subject: Re: Few questions on L2TP..
> 
> Keyur Parikh wrote:
> >
> > >Repeat after me: "Data messages do not need synchronizing", "Data
> > >messages do not need synchronizing", "Data messages do not need
> > >synchronizing"...
> >
> > OK, then how do you explain the following from RFC2661 (section 5.4)
> >
> > "
> > Unlike the L2TP control channel, the L2TP data channel does not use
> >    sequence numbers to retransmit lost data messages. Rather, data
> >    messages may use sequence numbers to detect lost packets and/or
> >    restore the original sequence of packets that may have been reordered
> >    during transport.  The LAC may request that sequence numbers be
> >    present in data messages via the Sequencing Required AVP (see Section
> >    4.4.6). If this AVP is present during session setup, sequence numbers
> >    MUST be present at all times. If this AVP is not present, sequencing
> >    presence is under control of the LNS. The LNS controls enabling and
> >    disabling of sequence numbers by sending a data message with or
> >    without sequence numbers present at any time during the life of a
> >    session. Thus, if the LAC receives a data message without sequence
> >    numbers present, it MUST stop sending sequence numbers in future data
> >    messages. If the LAC receives a data message with sequence numbers
> >    present, it MUST begin sending sequence numbers in future outgoing
> >    data messages. If the LNS enables sequencing after disabling it
> >    earlier in the session, the sequence number state picks up where it
> >    left off before.
> >
> >    The LNS may initiate disabling of sequencing at any time during the
> >    session (including the first data message sent). It is recommended
> >    that for connections where reordering or packet loss may occur,
> >    sequence numbers always be enabled during the initial negotiation
> >    stages of PPP and disabled only when and if the risk is considered
> >    acceptable. For example, if the PPP session being tunneled is not
> >    utilizing any stateful compression or encryption protocols and is
> >    only carrying IP (as determined by the PPP NCPs that are
> >    established), then the LNS might decide to disable sequencing as IP
> >    is tolerant to datagram loss and reordering.
> > "
> > The above clearly indicates that the Ns value for each end is present at
> the
> > other end to re-order and detect lost packets. Also, if the LNS disables
> it
> > and then re-enables it, the sequence number MUST start from where it
left
> 
> There is no "MUST" in this sentence:
> 
> "If the LNS enables sequencing after disabling it earlier in the
> session, the sequence number state picks up where it left off before."
> 
> But, having seen this text again here, I certainly agree that it would
> not hurt for it to be clarified a bit overall. I will put this on my
> list of things to do for l2tpbis. Thanks.
> 
> - Mark
> 
> > off.  THIS IS ALL STATEFUL INFORMATION THAT NEEDS TO BE RESET (I used
the
> > word resynch to really mean reset). I am fully aware of the difference
in
> > the use of the sequence numbers for control and data channel.
> >
> > -----Original Message-----
> > From: W. Mark Townsley [mailto:[email-masked]]
> > Sent: Tuesday, March 06, 2001 8:22 PM
> > To: Keyur Parikh
> > Cc: Andy Koscinski; [email-masked]
> > Subject: Re: Few questions on L2TP..
> >
> > > Just a thought: How about ensuring that your hot standby router has
full
> > > KRP - This is not a very scalable solution.  Specially for the session
> > data
> >
> > Repeat after me: "Data messages do not need synchronizing", "Data
> > messages do not need synchronizing", "Data messages do not need
> > synchronizing"...
> >
> > > messages. IMO, the RFC2661 should state what the LNS MUST do when it
> > > receives session data messages without Ns when it was enabled by the
> LNS.
> > I
> >
> > Sequencing on data packets within any tunneling protocol is a tricky
> > subject.
> >
> > IMHO, the LNS SHOULD accept them. One could argue that the LNS should
> > have this wiggle room if it "knows" that sequencing is vital for the
> > particular state of the protocol being carried or not. I suppose it
> > would not hurt to make this more clear in the spec, though there are no
> > easy answers here.
> >
> > > agree with andy that some mechanism should be recommended for
resetting
> > the
> > > stateful information between LAC and LNS.  How about assigning a bit
in
> > the
> > > common header for this purpose. Anytime either end want to reset the
> > > applicable sequence numbers they can set this bit in the control
> message.
> >
> > And then you have to check this bit for every single data packet just to
> > enable the failover feature? Probably not a good choice. Error recovery
> > should not slow down your data packet performance.
> >
> > > This should trigger the other end reset its internal variables for the
> > > control and all the data session of the tunnel.  Again this is just an
> > > initial thought.
> >
> > If we have the freedom to modify the protocol, this can probably all be
> > done with explicit signalling. Also, you could decouple the tunnel
> > control connection and sessions within if you wish. Bottom line, we
> > didn't consider this when designing the protocol at the beginning. If we
> > are going to modify the protocol to handle semi-full state failover,
> > then we should step back and approach it from a fresh perspective.
> >
> > - Mark

---

From: Keyur Parikh <[email-masked]>
Date: Wed, 14 Mar 2001 08:45:30 -0500

comments with KRP

-----Original Message-----
From: Vipin Jain [mailto:[email-masked]]
Sent: Wednesday, March 14, 2001 2:16 AM
To: 'W. Mark Townsley'; Vipin Jain
Cc: 'Keyur Parikh'; Andy Koscinski; [email-masked]
Subject: RE: Few questions on L2TP..

Mark,

Agree with most of your ideas. I wanted to add the following:
Querying or probing peer about your state is different from asking peer to
reset sequence number and retransmit control packets which has not been
acknowledged. Following can be fairly easily implemented and backward
compatible as well:

* After fail over/shutdown on LNS (or LAC) when redundant one takes over, it
initiates a packet on tunnel with Ns=0, Nr=0 and "reset sequence bit" set.
* LAC (or LNS) responds with first acknowledged packet with Ns=0 Nr=0 and
"reset sequence bit" set
* LNS (or LAC) responds to it with Ns=0 Nr=0 without "reset sequence bit"
set.
* After having sent a packet with Ns=0 and Nr=0 and "reset sequence bit"
set, if LAC or LNS receives a packet with Ns or Nr with NOT EQUAL to ZERO it
considers it out of sequence.
* They can then start exchanging control packets as they were before.
* Data plane can use the same logic if it wishes.
KRP - Actually, I was thinking the control channel message at either end
would trigger the data sessions of the tunnel  to reset the sequence numbers
- no need to go through the same mechanism.
KRP - Agree, my proposal was to reset the sequence numbers to 0.  To avoid
doing this for every tunnel between the LAC and LNS, there needs to be an
indication to reset the sequence numbers for all tunnels.

regards,
keyur

thanks,
vipin

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Tuesday, March 13, 2001 10:51 AM
To: Vipin Jain
Cc: 'Keyur Parikh'; Andy Koscinski; [email-masked]
Subject: Re: Few questions on L2TP..

I don't disagree that a signalling mechanism could reduce the overhead
of state checkpoints for failover operations. However, it is also true
that it is *possible* to be fully redundant without changes to the
protocol (this has been proven by example from at least one vendor I
know of, though I do not know personally at what level of scale). 

It is certainly easier and quite likely more scalable to perform
failover if you can query your peer for information. If there is enough
interest and participation to support this I think it is a very good
idea. However, if we are going to change the protocol, I think that
there are more extensible ways to do this than setting a bit in the
header that effectively overloads the meaning of one or more existing
control messages.

To illustrate, here are a couple of techniques I came up with off the
top of my head while out jogging this afternoon:

1. If you must have a single message "probe," then define a new message
type that is always sent with Ns = 0, Nr = 0 and tunnel ID = 0. For
existing implementations, this matches the transport level signature of
an SCCRQ and should be immediately passed from the transport layer to
the L2TP protocol layer for processing. Here, the new message type will
either be rejected, or accepted if the new feature is supported. Within
the message you could request whatever information you wanted, for
whatever tunnels you wanted, via specific AVPs. You are limited only by
the 1500 byte maximum size of a control message (which may come into
consideration if for scaling purposes you wish to concatenate
information for each posible open tunnel within a single message). There
are obvious security considerations to this method if you are sending
this message over an unsecured transport. 

2. A more secure, and perhaps a little more considerate, solution would
be to go ahead and setup a new control connection with the peers you
wish to resync with. After the connection is established, you can send
control messages and AVPs asking for resync information for whatever
tunnels you choose. This is actually quite similar to #1 except that it
has some protection by allowing tunnel authentication to occur before
revealing everything the box knows about its current state to a peer. It
also preserves the existing control connection setup logic that we all
have in place now.

Other improvements when operating in a failover-capable mode might
include:

- Advertisement at tunnel setup of the ability for a peer to announce
certain state after a failover, this includes what types of values it
has the ability to respond with when requested. This allows the ultimate
level of flexibility and backwards compatibility as the peer can decide
how much state it must keep on backup nodes and how often checkpoints
must occur based upon its peer's ability. Of course, this flexibilty and
compatibility adds complexity for us programmers, but our users would
probably be better served by something like this as it does not require
both sides to participate to be fully redundant on one side -- it just
affords better scalability when the other side participates. Also, it
scales to values beyond simply Ns/Nr synchronization. Other values that
are difficult to checkpoint, such as connect time, byte/packet counters,
etc. all could be very important in a full-featured redundant system.

- Leaving sessions persistent in the face of an abnormal control
connection disconnect in hopes that a backup control connection will be
forthcoming - either from the same box (e.g. after a reboot) or another
box waiting in the wings. We would have to be very careful here so as
not to keep unwanted sessions hanging around (for security, billing, and
resource allocation considerations). 

As I said, these are just a couple I thought of off the bat. I'm sure
that there are other techniques as well, perhaps one that is better than
either of these. 

- Mark

Vipin Jain wrote:
> 
> Have been following this thread for a while..
> 
> I have to agree with Keyur's idea of having bit in common-header, which
can
> be examined by control plane (may be data plane if it wishes to incur the
> overhead of examining every single packet) to see if peer seeks to reset
the
> sequence numbers. I believe there are some protocols that already do this.
> It helps them to have planned shutdown etc., which may not be truly
relevant
> in L2TP. Basic idea still remains same: to have no impact on the end
users,
> should something happen.
> 
> May be Mark argues that it is not required because an implementation can
> smartly get away with saving states and sequences of various
> tunnels/sessions. IMHO, it doesn't harm to provision this in protocol! Can
> be backward compatible too.
> 
> thanks,
> -- vipin
> 
> From: Keyur Parikh [mailto:[email-masked]]
> Sent: Friday, March 09, 2001 11:18 AM
> To: W. Mark Townsley; Keyur Parikh
> Subject: RE: Few questions on L2TP..
>

---

From: Keyur Parikh <[email-masked]>
Date: Wed, 14 Mar 2001 13:33:54 -0500

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Wednesday, March 14, 2001 11:51 AM
To: Vipin Jain
Cc: 'Keyur Parikh'; Andy Koscinski; [email-masked]
Subject: Re: Few questions on L2TP..

Vipin Jain wrote:
> acknowledged. 

OK, so you are suggesting that we tell a peer to *start over* at a new
sequence number. Make sense, except that (1) you have to ensure that
there are no local messages that you may have acknowledged but did not
transfer state of to your backup node yet 
>KRP - session information should be saved before an ack is sent out.
(possible, but if we really
want to talk about scaling then this can be limiting), (2) this requires
your peer to poke down into its transport level unsent queue and
resequence messages (nasty layer violation),
>KRP - why can't you just flush the queue of current messages, and
re-transmit them which the new sequence number. I think this should be an
implementation issue.
 (3) this is not at all
extensible, (4) most importantly, this is NOT backwards compatible
(illustrated below).
>KRP - If we increase the protocol version for this change, then a peer can
use this information to identify whether or not it can initiate this
exchange. When talking to version 1.0, the peer MUST not use this method for
resyncing.  I think, these are initial thoughts, and obviously enough
support must provided in the specification and the implementation to make
this robust. I guess defining a separate message for this will work also,
the only problem I had with your #1 was the scaling issue. I would rather
start the sequence number with 0. IMHO,  a single message should be designed
such that I can reset a single tunnel(with sessions that have sequencing
enabled) or all the tunnel belonging to an endpoint. 

> Following can be fairly easily implemented and backward
> compatible as well:
> 

An implementation that does not support the reset bit could consider
this a valid "future" packet and simply drop it. Again, run your logic
with Ns=100, Nr=100 and tell me if it works in a backwards compatible
manner. There is NOTHING SPECIAL about Ns=0 and Nr=0 to the L2TP
transport unless the TunnelID=0 as well (the zero TunnelID being the
more important of these fields).

Further, an implementation MAY update its resend queue with the Nr
present in an out of order packet (see the l2tpbis draft). So, your Nr=0
could end up inadverdantly acknowledging a lot of packets. I know of at
*least* one widely deployed implementation that does this, so get ready
for some hard to figure out bug reports if you deploy this non-standard
extension in a multivendor environment!

> * LNS (or LAC) responds to it with Ns=0 Nr=0 without "reset sequence bit"

The following steps are not completely clear to me, but it doesn't
matter as the step above breaks backwards compatibility already.

> * Data plane can use the same logic if it wishes.

There is no concept of "replying" on the data plane, so I don't see how
this same logic can be applied. Also, treating Ns=0 as somehow "special"
on the data plane is as incidious as it is on the control plane. Nr on
the data plane is just a reserved field, you cannot use it for anything.

If you just want to set an "R-bit" ala the old flow control days to
reset your requence numbers, fine, I guess that would work to signal
your peer to accept an out of order packet (assuming it is even
checking). But, this adds a bit to the data plane and makes the data
header even more complex than it already is. I still think for the data
plane you could just start over at 0 and let the peer drop a couple of
packets (again, assuming it is even checking sequence numbers).

> > > > The above clearly indicates that the Ns value for each end is
present
> at
> > > > And then you have to check this bit for every single data packet
just
> to
> > > > > control and all the data session of the tunnel.  Again this is
just
> an
> > > > didn't consider this when designing the protocol at the beginning.
If
> we

---

From: Keyur Parikh <[email-masked]>
Date: Wed, 14 Mar 2001 13:36:14 -0500

-----Original Message-----
From: Bill Palter [mailto:[email-masked]]
Sent: Wednesday, March 14, 2001 1:28 PM
To: W. Mark Townsley
Cc: Vipin Jain; 'Keyur Parikh'; Andy Koscinski; [email-masked]
Subject: Re: Few questions on L2TP.. 

> transfer state of to your backup node yet (possible, but if we really
> resequence messages (nasty layer violation), (3) this is not at all

  I worked on an implementation that supported failover, and played around 
with a few different methods of trying to trick peers into resyncing, I
tested
it at several bakeoffs and basicly we found that for the data channel
nothing
really needed to be done. starting over at a random number worked as well
as keeping state, 
>KRP - does this mean that in all the bakeoff tests you did, the peer did
not support sequencing or re-ordering for data messages? I would be
interested to know how many vendors provide re-ordering support.
when we had flow control (before it was removed) we would
just set the R bit and everything was happy.

  The control channel was a different story, the only thing that worked well

was to keep state across all the possible backup nodes, anything else risked
loosing control messages that might have been important to the state of the 
tunnel. It was not very difficult to identfy the key locations in the
protocol
that case state to need to be distributed.

Bill Palter

---

From: Keyur Parikh <[email-masked]>
Date: Tue, 13 Mar 2001 16:21:18 -0500

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Tuesday, March 13, 2001 1:51 PM
To: Vipin Jain
Cc: 'Keyur Parikh'; Andy Koscinski; [email-masked]
Subject: Re: Few questions on L2TP..

>KRP question on security (for non-IPSEC operation)- since the tunnel
disconnect message is not authenticated, what stops a "man in the middle" to
snoop control channel messages between LAC and LNS, collect all the
necessary information and form a StopCCN message to either end.  Am I
missing something?

- Mark

can
the
relevant
users,

---

From: Vipin Jain <[email-masked]>
Date: Wed, 14 Mar 2001 22:07:20 -0800

Mark,

comments inline.. with <<vipin>>

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Wednesday, March 14, 2001 8:38 PM
To: Vipin Jain
Cc: 'Ignacio Goyret'; 'Keyur Parikh'; Andy Koscinski; [email-masked]
Subject: Re: Few questions on L2TP..

Vipin Jain wrote:
> 
> Ignacio, Mark:
> 
> I have commented on your ideas in following with <<vipin>> tags..
> 
> From: Ignacio Goyret [mailto:[email-masked]]
> Sent: Wednesday, March 14, 2001 10:19 AM
> To: W. Mark Townsley
> I normally hate jumping into a heated discussion, but here are my $.02:
> 
> Abusing Ns=0 or adding a "reset" bit is just a very bad idea. It makes
> L2TP so much more vulnerable to attacks. It also negates any
authentication
> that may have been negotiated between the two endpoints: how can I verify
> that the request to resync is from a bona fide source?
> 
> <<vipin>>
> Regarding security, as keyur also pointed out, how do you stop a person
> from sending a StopCCN on a tunnel; In other words, you are advocating
> something like "password digest" to authenticate every message coming from
> peer,
> like how radius does it.
> <<vipin>>

A StopCCN attack is a Denial of Service attack. Even so, in a blind
situation you have to guess your peer's tunnel ID and the proper Ns for
the tunnel. The "reset bit" proposal negates the second item here,
reducing the values one has to guess down to just 16 bits.

<<vipin>>
agree that guess work is more when somebody wants to attack using StopCCN.
But snooping just one valid control packet makes it equally easier.
Nevertheless, here is a better suggestion:
Why not include challenge response AVP, which was used in SCCRP or SCCN,
while sending this message with "reset bit"..
<<vipin>>

> 
> Finally, it is a hack that has been demonstrated to fail (guaranteed!) in
> some cases.
> 
> <<vipin>>
> Not really, if we ensure an implementation following today's rfc2661
> legitimately drops the packet because of one of the following:
> - the version number of newer packet is different
> - the message contains an AVP which it doesn't like
> - the message has some header field which it thinks is invalid).

Changing the version number in the header for one new feature is a bit
drastic. Header bits are precious.
<<vipin>>
agree. using version number would be too drastic..
I was just validating in my mind if it would be feasible at all :-)
Do you have any thoughts on how it would make today's rfc2661 implementation
drop the packet but still convey the information. Like set L bit to 'off'
for a control packet?
<<vipin>>

> 
> Where as, a newer implementation of L2TP could interpret it correctly and
> get something sensible out of it.
> <<vipin>>
> 
> A *MUCH* better solution is to add new commands to the protocol so that
> the failed end can obtain all the resync information over a _new_ tunnel,
> as Mark suggested. Whether the remote end supports or not these new
> commands can easily be determined by a judicious use of the M-bit or
> by using a different number in AVP 2, or even with a new AVP (resync
> capable)
> added to the SCCRQ and SCCRP. This is a significantly more robust design
> since it will allow you to select different strategies depending on the
> capabilities of the remote endpoint.
> 
> <<vipin>>
> Relying peer to give all information over a new tunnel has following
> problems:
> - it invites a significant amount of traffic during resync and probably
> that's when you would want to avoid all un-necessary traffic flowing
between
> you and peer as there is already a dozen of other things going on when a
> failover occurs.

Let's see.

First, assume that there is only one tunnel between a given LAC and LNS:

- If you consider a hot standby case, the control connection could
already be setup between the standby and the peer, waiting for a state
update in case of a failover. One control connection should not take up
considerable resource, and you peer has the bonus of knowing that your
standby is ready and waiting in the wings.

- If you consider a restoration after reboot case, sending 3 packets for
tunnel setup is likely insignificant to the time it took to reload the
box.
<<vipin>>
I thought Ignacio mentioned downloading whole databases of sessions and
tunnels over from peer. hence..
<<vipin>>

As a more pathological example, suppose I have 2000 tunnels between an
LAC and an LNS and one side goes down. With the "reset bit" method, I
have to send 2000 packets of some sort with the reset bit to resync.
With the #2 proposal I sent earlier, I bring up one control connection
and can obtain whatever state information I wish to compact within a
single control message exchange, potentiall one message for all 2000
tunnels (Ignacio pointed out to me that RFC 2661 removed the 1500 byte
control message limitation somewhere along the way, so if you don't mind
your control messages potentially being fragmented by IP, then you could
packet one message chock full of whatever you wish... I still think it
is probably prudent to keep it under 1500 in separate messages though).

> - more serious problems like peer can propagate incorrect information to
> you.

If your peer is going to lie to you then you have much bigger problems
at hand.

> 
> Asking peer to restart sequencing (using "resync bit" was just one way to
do
> it) is much more cleaner.
> <<vipin>>
> 
> If you were to implement something like the Ns=0 hack, I think you have a
> duty to warn your customers that your box is not 2661-compliant and that
> this non-compliance may lead to unexplained behavior and failures.
> <<vipin>>
> As of today, nobody should claim it will be 2661 compliant..

Interesting accusation.

> IMHO, that's doesn't mean we should stop trying to work on a cleaner way
of
> solving problems.

Using bits in the header (so far you are suggesting at least two, one
for the "reset" bit and one for the header version increment) for new
features that can be just as well served with control messages and AVPs
is not a good use of the few header bits we have. 
<<vipin>>
header version increment is not something I recommend.. that was just a
thought..
yes "reset" bit is something I would suggest and the only change.
<<vipin>>

> <<vipin>>
> 
> thanks,
> vipin
> 
> At 11:50 AM 3/14/01 -0500, W. Mark Townsley wrote:
> >Vipin Jain wrote:
> >>
> >> Mark,
> >>
> >> * After fail over/shutdown on LNS (or LAC) when redundant one takes
over,
> >> set, if LAC or LNS receives a packet with Ns or Nr with NOT EQUAL to
ZERO

---

From: Andy Koscinski <[email-masked]>
Date: Thu, 15 Mar 2001 21:58:14 -0800

Bill Palter wrote:

> tunnel. It was not very difficult to identfy the key 
> locations in the protocol

Bill, are you saying that the L2TP implementation you tested was
updating all backup nodes with EVERY Ns, Nr change in 
the control plane? How many L2tp tunnels
do you think such solution can support?

thx
-andy

> Sent: Wednesday, March 14, 2001 10:28 AM
> To: W. Mark Townsley
> with a few different methods of trying to trick peers into 
> resyncing, I tested
> it at several bakeoffs and basicly we found that for the data 
> channel nothing
> as keeping state, when we had flow control (before it was 
> removed) we would
> tunnel. It was not very difficult to identfy the key 
> locations in the protocol

---

From: Andy Koscinski <[email-masked]>
Date: Thu, 15 Mar 2001 23:02:48 -0800

Gentelmen:

my comments inline with "andyk"

> Sent: Wednesday, March 14, 2001 10:07 PM

andyk> agreed, the security was a problem, but I find this a good idea.

> > Finally, it is a hack that has been demonstrated to fail 

andyk> I disagree with Ignacio. how did you demonstrate, it fails?

> > - the message has some header field which it thinks is invalid).
 
andyk> I agree with vipin here.

> > as Mark suggested. 

andyk> I think this is not a better, but worse solution.
I doubt it works for more than 500 tunnels at all.
if every tunnel terminates at a different node, this solution
requires the failed node to establish a new synchronization tunnel 
for every tunnel which existed before the fail over.
This creates a big burst of traffic after the failure and therefore
dramatically reduces the scalability of this solution. 

> > Whether the remote end supports or not these new

andyk> I think you've got a point, vipin. It's not only the L2TP who
wants to resync. all modules want to resync as quickly as possible.

> tunnel setup is likely insignificant to the time it took to reload the

andyk> please think about thousands of tunnels and how to minimize
the traffic after the fail over. while the failed node is busy establishing
the new sync tunnels, it has to respond (time is limited) to messages 
on existing tunnels. the responses to
to the existing tunnels will compete for resources with the creation
of the new tunnels. Should the responses to messages arriving on existing
tunnels have priority over generating new tunnels? otherwise we
might end up with many new tunnels, but could loose old, good tunnels
we wanted to keep up in the first place.

> is probably prudent to keep it under 1500 in separate 

andyk> IMO 2000 tunnels and more is the scale to think about.
maybe it should work for all 64k tunnels possible? 

> > As of today, nobody should claim it will be 2661 compliant..

andyk> any proprietary extension of the rfc 2661 with the resync signal
(Ns=0, Nr=0 or any other explicit resync request signalling), 
does not claim to be compliant with the rfc 2661. I think this is obvious.
Such an extension serves the purpose to demonstrate, that
a scalable hot standby concept without keeping full state can be implemented
with
a lesser effort, than a full state hot standby. IMO it is trivial 
to configure the proprietory extension off, and stay rfc 2661 compliant.

> > >> Querying or probing peer about your state is different 
> from asking peer
> > to
> > >> initiates a packet on tunnel with Ns=0, Nr=0 and "reset 
> sequence bit"
> > set.
> > >> * LAC (or LNS) responds with first acknowledged packet 
> with Ns=0 Nr=0
> and
> > >> "reset sequence bit" set
> > >
> > >> * LNS (or LAC) responds to it with Ns=0 Nr=0 without 
> "reset sequence
> bit"
> > >> set.
> > >
> > >> * After having sent a packet with Ns=0 and Nr=0 and 
> "reset sequence
> bit"

---

From: Andy Koscinski <[email-masked]>
Date: Fri, 16 Mar 2001 12:55:46 -0800

4 new comments marked "andyk>" below.

> From: Keyur Parikh [mailto:[email-masked]]
> Sent: Friday, March 16, 2001 12:10 PM
> To: Andy Koscinski; Vipin Jain; 'W. Mark Townsley'
> Cc: 'Ignacio Goyret'; Keyur Parikh; [email-masked]
> I think there is a general consensus that the RFC should be 
> extended for this support. 

andyk> this is the outcome of this discussion, IMO.

> I am ok with a explicit signaling message,

andyk> me, too.

> but I don't agree
> with #1 and #2 solutions proposed by Mark.  Here are couple of hybrid
> proposal.
> 
> 1. (if protocol version number can be incremented)
> -The common header parameter for this message such as: tunnel 
> id, Ns, Nr
> should be set 0.
> -make a new message type = sync request
> -create new AVP -  sync info (this may be hidden for authentication)
> - the new avp should contain following:
> 	- indication to reset all tunnels attached to this endpoint OR
> 	- a specific list of tunnels to reset to attached to 
> this endpoint.
>  the receiving end should reset the sequence numbers for the 
> control channel
> and the all the associated data sessions (if sequencing 
> enabled) of each
> specified tunnel.  The receiving end should format a sync 
> response message
> with sync info. The sync info should contain the list of 
> tunnels which were
> reset or an indication that all tunnels between the peers are reset.
> 
> 2. if protocol version needs to stay the same.
> 
> - the requester sends SCCRQ with the AVP - sync info.  
> The M bit in the AVP must be 1.

andyk> which tunnel id to use? if this is a new tunnel id,
than the max number of tunnels is reduced down from 64k.

> - The responder, if upgraded with the change should send StopCCN as a
> response to the SCCRQ and include the AVP-sync info 
> containing a list of
> tunnel which were reset or an indication that all the tunnels 
> between the
> peers were reset.
> - if the responder is an older version, then the responder 
> will terminate
> the tunnel without the new AVP. In this case, the requester 
> falls back to
> the original implementation (which may be re-establishing the tunnel)
> 
> In both cases, the requester will discard any packets from 
> peer until a
> response is received. 

andyk> to discard all pre-fail ctrl messages is good, but I think it 
is better to send a sync request back to the source of the discarded 
ctrl message, which should lower the risk that such tunnel will 
time out and go down.

> The discarded requests will get 
> re-transmitted, once
> the peers enter normal operation.
> 
> regards,
> keyur
>  
> 
> 
> 
> 
> From: Andy Koscinski [mailto:[email-masked]]
> Sent: Friday, March 16, 2001 2:03 AM
> To: Vipin Jain; 'W. Mark Townsley'

---

From: Vipin Jain <[email-masked]>
Date: Fri, 16 Mar 2001 15:16:08 -0800

Keyur,

comments inline .. with <<<<<vipin>>>>>

-----Original Message-----
From: Keyur Parikh [mailto:[email-masked]]
Sent: Friday, March 16, 2001 12:10 PM
To: Andy Koscinski; Vipin Jain; 'W. Mark Townsley'
Cc: 'Ignacio Goyret'; Keyur Parikh; [email-masked]
Subject: RE: Few questions on L2TP..

I think there is a general consensus that the RFC should be extended for
this support. I am ok with a explicit signaling message, but I don't agree
with #1 and #2 solutions proposed by Mark.  Here are couple of hybrid
proposal.

1. (if protocol version number can be incremented)
-The common header parameter for this message such as: tunnel id, Ns, Nr
should be set 0.
-make a new message type = sync request
-create new AVP -  sync info (this may be hidden for authentication)
- the new avp should contain following:
	- indication to reset all tunnels attached to this endpoint OR
	- a specific list of tunnels to reset to attached to this endpoint.
 the receiving end should reset the sequence numbers for the control channel
and the all the associated data sessions (if sequencing enabled) of each
specified tunnel.  The receiving end should format a sync response message
with sync info. The sync info should contain the list of tunnels which were
reset or an indication that all tunnels between the peers are reset.

- the requester sends SCCRQ with the AVP - sync info.  The M bit in the AVP
must be 1.
- The responder, if upgraded with the change should send StopCCN as a
response to the SCCRQ and include the AVP-sync info containing a list of
tunnel which were reset or an indication that all the tunnels between the
peers were reset.
- if the responder is an older version, then the responder will terminate
the tunnel without the new AVP. In this case, the requester falls back to
the original implementation (which may be re-establishing the tunnel)

<<<<<vipin>>>>>
Problem here: Let's say requester has newer version and responder has older
version.
Requester can deal with some pieces of redundancy (by keeping track of
sessions etc.) but it would like to try out (just try not bring down the
tunnel itself) and see if responder can do "newer version" things or not.
Your proposal says if requester decides to send SCCRQ with "sync info" AVP
then it may take risk of having peer to bring down the tunnel.
<<<<<vipin>>>>>

In both cases, the requester will discard any packets from peer until a
response is received. The discarded requests will get re-transmitted, once
the peers enter normal operation.

regards,
keyur

-----Original Message-----
From: Andy Koscinski [mailto:[email-masked]]
Sent: Friday, March 16, 2001 2:03 AM
To: Vipin Jain; 'W. Mark Townsley'
Cc: 'Ignacio Goyret'; 'Keyur Parikh'; Andy Koscinski; [email-masked]
Subject: RE: Few questions on L2TP..

Gentelmen:

---

From: Vipin Jain <[email-masked]>
Date: Fri, 16 Mar 2001 15:19:34 -0800

I would favor "resetting sequence numbers" than obtaining database of
tunnels/sessions which are up from peer.

-- vipin

-----Original Message-----
From: Keyur Parikh [mailto:[email-masked]]
Sent: Friday, March 16, 2001 1:42 PM
To: W. Mark Townsley; Keyur Parikh
Cc: Andy Koscinski; Vipin Jain; 'Ignacio Goyret'; [email-masked]
Subject: RE: Few questions on L2TP..

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Friday, March 16, 2001 4:27 PM
To: Keyur Parikh
Cc: Andy Koscinski; Vipin Jain; 'Ignacio Goyret'; [email-masked]
Subject: Re: Few questions on L2TP..

A careful analysis will show that your #2 is the same as my #1 from
earlier except you use an AVP within an SCCRQ and StopCCN to identify
what is essentially a new message, and I just use a new message type.
Note that it has the same security problems I identified in my #1
proposal.
- Mark

KRP - Mark, the major difference is that you wanted to retrieve information
from the peer (i.e. sequence numbers), I am proposing to just reset them.
Other major difference is that I am including an option to reset all tunnels
belonging to the peer, this way we don't have exchange 1000s of such
messages. I hope these differences are clear to you. In any case, I said
these were hybrid proposals i.e. the explicit signaling is something you and
andyk mentioned.
If I hide this AVP or exchange challenge AVP, why is it a security threat? 

- Mark

---

From: Keyur Parikh <[email-masked]>
Date: Mon, 19 Mar 2001 10:22:50 -0500

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Friday, March 16, 2001 6:34 PM
To: Keyur Parikh; Andy Koscinski; Vipin Jain; 'Ignacio Goyret';
[email-masked]

> Note that it has the same security problems I identified in my #1

I thought you were suggesting that state information be returned in the
StopCCN vs. back to the "resync" again.

<KRP> No, my premise from beginning is to have an effective method of
resuming the operation, thus resetting the seq numbers.
I think "resync" alone is not an extensible solution for full-featured
failover.
<KRP> May be. Lets discuss those.