Subject: RE: Few questions on L2TP..

From: Keyur Parikh <[email-masked]>
Date: Tue, 6 Mar 2001 18:08:06 -0500

comment with KRP

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Tuesday, March 06, 2001 5:33 PM
To: Andy Koscinski
Cc: Keyur Parikh; [email-masked]
Subject: Re: Few questions on L2TP..

Andy Koscinski wrote:
> 
> In my understanding this dicussion is about how to
> support a "hot standby" LAC, which was not addressed
> by the rfc 2661, but is a relevant question, IMHO.
> 
> mark> 5. I assume by "switchover" you mean some sort of active failover
> technique. This is, in fact, outside the scope of RFC2661.
> andyk> yes
> 
> more comments inline, marked "andyk"
> 
> > > 1) Since there is no protocol support to re-synchronize the
> > tunnel and
> > > data message sequence number -
> > >    how does the LAC go about recovering a tunnel and call
> > session after
> > > a switchover?
> > >
> > >    For tunnel control channel will the following work?
> > >
> > >        andyk> in the standard rfc2661 it would not work,
> > because the Ns
> > > and Nr
> > >        between the LAC and the LNS would not synchronize after the
> > > switch over
> > >        and the tunnel would go down.
> > >
> > >
> > >                LAC                            LNS
> > >                        Hello (Ns =0 ,Nr=0)
> > >                   ------------------------------>
> > >                        ZLB (Ns=x,Nr =y)
> > >                    <-----------------------------
> > >        The LAC can then determine its state variables based
> > on the ZLB.
> > >
> > >        andyk> yes, in the above msg exchange, the LAC and
> > the LNS could
> > > synchronize
> > >        their msg sequence counters  (and Amber Networks L2TP
> > > implementation does,
> > >        which is an extension to the rfc2661).
> > > >KRP - andy, I agree in theory it should work.
> >
> > This could be dangerous.
> >
> > It seems that you are "probing" with a Hello message that you think is
> > invalid, and expecting the peer to send a ZLB ack in response
> > and obtain
> > information from this. In the nominal case, this could work, but you
> > have to consider the edge conditions as well. You don't want to design
> > something that works "most" of the time, particularly when a
> > failure can
> > bring down an entire tunnel with thousands of sessions within it.
> 
> andyk> this was just an initial thought. I think we can design
> something which works in all cases. Do you think we need an extension to
> the rfc 2661 to support a hot standby LAC?
> 
> >
> > - The Ns sequence space is a wrapping 16-bit value. It begins with 0,
> > but after that there is no signifigance to 0 other than it
> > occurs again
> > some time in the future... Probably beyond your testing
> > procedures so it
> > "looks" like this works fine. Think of choosing, say, 100 for your Ns
> > Hello probe and think about whether your method will always work.
> 
> andyk> yes, we know that. please note, that Ns AND Nr = 0, for which
> the probability of occuring in the same Hello msg is extremely low.
> All what such Hello (Ns=0, Nr=0) message could mean is, that
> the LAC and LNS synchronize their tunnel control sequence counters
> (and do not bring a tunnel down), which does not hurt anybody.

Here is an example of hurt: one side of your tunnel has sent out a bunch
of ICRQs, you acknowledged them with the probing Hello but never
actually received the messages. So, the side that sent the ICRQs is
still waiting for the ICRPs, perhaps forever. There is no guarantee that
an implementation will timeout and resend these ICRQs unless the
protocol being tunneled has some sort of timeout or keepalive that
resets the link. This is just an example, the same would apply to any
control message which an implementation expects was delivered to L2TP,
not just acknowledged by the transport (which is just about all of
them).

In general, the probability argument just means that it will be that
much more difficult to discover the problem when it occurs in a
customer's network. 

But let's assume for a moment that this is OK.

First, consider the simple and perhaps likely case of bringing up
sessions on a tunnel, no sessions brought down, and all you are doing is
sending keepalives. Since both sides start sequence numbers at 0, and
both sides send the same number of messages, Ns and Nr very well may
track with one another. If both sides of the link are configured to send
hellos at the same interval, and neither performs any optimizations
about not sending hellos if one was received, then based on 60 sec
intervals you have a shot at sending a "real" Hello Ns=0 Nr=0 packet
every 45 days or so. Less for the first hit depending on the number of
sessions you bring up. 

Second (and more importantly), it is fully acceptable for an
implementation to inspect the Nr of a packet for its own retransmit
queue, even when the Ns is not the value it wants. So, everytime you
probe, you are very likely telling your peer: "I have received all
packets you sent me up to 0." Which means, depending on your
implementation, all messages between 32767 and 65535 could get
acknowledged via your probe (depending on your implementation). Messages
beneath this value do not, and your peer will retransmit them as you
would like it to do. The end result is that *sometimes* lost messages
will get retransmitted after the 0,0 Hello probe, and *sometimes* they
will not. You can imagine how difficult indeterminate behavior such as
this would be to debug in the field.

> 
> The thought here was, that a mechanism is needed to synchronize
> the tunnel control message counters between the LAC and LNS after
> a "switchover" to a hot standby LAC happened. If you do not like
> Hello (Ns=0, Nr=0), what's better in your opinion?

Just a thought: How about ensuring that your hot standby router has full
state before even acknowledging the control message from your primary?
This doesn't require any changes to the protocol.

KRP - This is not a very scalable solution.  Specially for the session data
messages. IMO, the RFC2661 should state what the LNS MUST do when it
receives session data messages without Ns when it was enabled by the LNS.  I
agree with andy that some mechanism should be recommended for resetting the
stateful information between LAC and LNS.  How about assigning a bit in the
common header for this purpose. Anytime either end want to reset the
applicable sequence numbers they can set this bit in the control message.
This should trigger the other end reset its internal variables for the
control and all the data session of the tunnel.  Again this is just an
initial thought. 
> 
> >
> > - The only way to be sure you are sending an Ns that will be
> > dropped and
> > acknowledged is to send Ns equal to the last acknowledged
> > packet for the
> > tunnel. However, I *think* your intended application prohibits this as
> > it is coming up after a "switchover" and has no sequence
> > number state.
> 
> andyk> yes, the "hot standby" is talking after a "switchover"
> and has no sequence number state, but does have the tunnel id
> and session id of all active tunnels and sessions
> and the question is how to keep the hello's
> going on in the known tunnels, so they'll stay up after the "switchover".

More than just hellos, but additional calls over the tunnel. I am
guessing your particular application doesn't care about this so much
though... e.g. you typically bring up all your tunnels and sessions at
the beginning and don't expect any further activity beyond the
keepalive. All l2tp tunnels are not like this though.

> 
> >
> > - The Nr of zero in your probe could inadverdantly acknowledge
> > outstanding packets on the tunnel (packets that you have not recevied,
> > but your peer will think you have received).
> >
> > - A peer does not have to respond to a Hello message with a ZLB.
> >
> > - Even if the Hello message does trigger a ZLB, a "real"
> > control message
> > could be received before the ZLB is (packets may be reordered in
> > transit).
> >
> > - Think about what happens if/when both sides try to sync with one
> > another.
> >
> > There are no requirements for "timeouts" of states in L2TP. The L2TP
> > state machine very much depends on the reliable control
> > channel beneath
> > it to deliver ALL packets. I see the possibililty for packets to be
> > erroneously acknowledged here, which places additional requirements on
> > the L2TP state machine.
> 
> andyk> all of your cases thought about and clearly we cannot implement
> a "hot standby" LAC taking an orthodox rfc 2661 position. But with some
> minor modifications, IMHO we could.

Yes, I expect you could, but BOTH sides may have to participate,
depending on how much state you are willing to save.

This is all just a matter of where you draw the line between state that
must be saved on your standby box and the state which you plan on
recreating at switchover. If you can save ALL state, then you can run
with an orthodox RFC2661 today. If you can change the protocol at will,
you could get away with very little saved protocol state. It's just a
matter of where you draw the line. How much information you can request
from your peer about connectivity when you come to life and takeover.

> 
> >
> > > WMT:
> > >
> > > 1. The control messages and data messages have different
> > sequence number
> > > spaces, there is no need to synchronize them.
> > >
> > > >KRP May be you misunderstood the question.  The
> > synchronization is between
> > > the LAC and LNS for control channel and data session
> > sequence number, not
> > > between the control channel and individual data channel.
> 
> andyk> I second KRP.
> 
> andyk> "data sequence number sync:"
> > You cannot rely on the data Nr to have anything valid in it.
> > Anyway, all
> > you need to do to synchronize data packets after losing state
> > is to pick
> > a number and start counting. The peer's window, timer, or whatever
> > sequence sanity checking it is performing will overflow and
> > pickup with
> > the current sequence numbers.
> 
> andyk> agreed with Mark (I have not argued otherwise).
> I think KRP was concerned how to turn off the payload sequencing
> in LAC after "switchover".

Currently, it's always up to the LNS to turn off sequencing. The LAC is
a slave.

> 
> >
> > >
> > > 2. ZLBs are never used for data sequencing.
> > > >KRP But ZLB does contain the Ns and Nr which can be used
> > by the LAC to
> > > derive its own sequence numbers.
> 
> andyk> I second KRP

And by doing so, you say it is OK to have falsely acknowledged control
packets on your tunnel.

> 
> > >
> > > 3. A Hello message can never have Ns of zero.
> > >
> > > 4. On the data channel, Nr is always ignored.
> > > >
> > > 5. I assume by "switchover" you mean some sort of active failover
> > > technique. This is, in fact, outside the scope of RFC2661.
> 
> andyk> (5) = yes
> >
> > > My point in all this is that why is there no support
> > > in the RFC2661 for re-synching the control and data session
> > sequence numbers
> > > after such a condition.  This will also help avoid proprietary
> > > implementation and make the protocol more robust.
> >
> > There is a fundamental assumption that all control packets will be
> > delivered reliably. Period.
> > At first glance, this "resyncing" seems to
> > break this under some conditions.
> 
> andyk> assuming we want L2TP redundancy in the future, may be we
> need to revisit this fundamental assumption. IMHO a synchronization
> of the tunnel control message sequence counters between LAC and LNS
> might be usefull after a "switchover" to keep the tunnels up, which is
> a concept new to the rfc 2661.

Agreed.

> 
> >
> > Again, there is no need to "resync" data messages.
> > >
> > >        andyk> the rfc 2661 requires LAC to continue including the
> > > sequence numbers, but
> > >        in my reading does not explicitly require the LNS to
> > clear the
> > > call after the LAC misbehaved?
> > >
> > >        I think it would be better if the LAC or LNS could
> > ask during the
> > > L2TP session establishment
> > >        not to use payload msg sequencing (currently the LAC
> > can only ask
> > > to use payload msg sequencing).
> > >
> > > WMT: The LNS is under control of sequence numbers as,
> > traditionally, it
> > > is the side that knows whether they are necessary or not (it is
> > > terminating the PPP session and knows whether a CP is being
> > negotiated,
> > > compression or encryption is active, what protocols are
> > being carried,
> > > etc.). Sequence number presense is not signalled on the
> > control channel
> > > as you need to be able to turn sequence numbers on and off
> > in-band, it
> > > may be "too late" if you have to wait on a control message.
> > >
> > > This is clearly not the best design. It was hacked in as a
> > comprimise to
> > > get RFC2661 ratified without much change to all of the
> > running code that
> > > we had. C'est la vie.
> 
> andyk> these are the arguments justified by the PPP in L2TP.
> but now we consider protocols other than PPP in L2TP,
> e.g. Frame Relay.

Agreed.

> 
> > >
> > > If we still need to conquer sequencing at the L2TP level, what I
> > > *really* really should exist, is a small, wrapping sequence
> > number that
> > > is present at all times on packets being sent. It can then
> > be ignored or
> > > attended to on the receiving side as necessary. In
> > hardware, stuffing in
> > > a simple value incremented for each packet is probably just
> > as fast as
> > > checking a bit to see if you need to put a sequence number
> > or a zero in
> > > place. Also, this eliminates all signalling on the subject.
> > >
> > > This begs the question though, should we have a fixed
> > header option that
> > > could be negotiated up front? That would certainly be more hardware
> > > friendly as well. Fixed headers are never easy for
> > committees to decide
> > > upon, but we could certainly try.
> > >
> andyk> I agree and share your concern for the hardware implementation.

---

From: "W. Mark Townsley" <[email-masked]>
Date: Tue, 06 Mar 2001 20:21:51 -0500

> KRP - This is not a very scalable solution.  Specially for the session data

Repeat after me: "Data messages do not need synchronizing", "Data
messages do not need synchronizing", "Data messages do not need
synchronizing"...

> receives session data messages without Ns when it was enabled by the LNS.  I

Sequencing on data packets within any tunneling protocol is a tricky
subject.

IMHO, the LNS SHOULD accept them. One could argue that the LNS should
have this wiggle room if it "knows" that sequencing is vital for the
particular state of the protocol being carried or not. I suppose it
would not hurt to make this more clear in the spec, though there are no
easy answers here.

> applicable sequence numbers they can set this bit in the control message.

And then you have to check this bit for every single data packet just to
enable the failover feature? Probably not a good choice. Error recovery
should not slow down your data packet performance.

> control and all the data session of the tunnel.  Again this is just an

If we have the freedom to modify the protocol, this can probably all be
done with explicit signalling. Also, you could decouple the tunnel
control connection and sessions within if you wish. Bottom line, we
didn't consider this when designing the protocol at the beginning. If we
are going to modify the protocol to handle semi-full state failover,
then we should step back and approach it from a fresh perspective. 

- Mark