Subject: RE: rfc2661: drop out of order (within rx-window) packets..

From: "Vipin Jain" <[email-masked]>
Date: Wed, 22 Aug 2001 13:39:48 -0700

Ignacio,

> >IMO, that's not scalable. Any thoughts? 
> 
> Yes: open one control connection per call. Your call rate
> will no longer be limited by the L2TP retransmission rate.
Agree; But then usually that's not efficient and scalable..
It will be inefficient because the amount of piggybacking that goes in
multiple sessions within a tunnel will be much higher than one session per
tunnel. With large number of sessions (consequently tunnels) it could be
significant.
It will not be scalable because we limit total number of sessions between
peers to 65k as opposed to 65k (tunnels) times 65k (sessions within each
tunnel).

> >Its easy to drop the packet and require retransmission
> >but we should probably get rid of this restriction and
> >have peer queue them.
> 
> A better alternative is to use better timeout algorithms or move
> the control connection to TCP, which handles retransmission much
> better.
Again, absolutely agree. If we were to behave like TCP then why not enqueue
packets (occasional drops are okay) on the receiving end. I wonder what
would be the performance of TCP it were dropping all out of order in window
packets!
Better retransmission can augment this to behave more close.

thanks,
-- vipin

---

From: "Vipin Jain" <[email-masked]>
Date: Wed, 22 Aug 2001 16:49:57 -0700

Greg,

I assume you are not arguing that "control packet rx windowing" as specified
in l2tp rfc is a waste. In other words you agree that a node is suppose to
enqueue the packets, in general, that are out of order so receive windowing
can work as intended. Because setting window size of "1" is equivalent of
saying that windowing is not enabled at all..

> I think the starting retry timer recommendation is 1 second.  
> You can back it off.
You mean decrease it? Backing it off will worsen the situation.
Anyways, decreasing the timeout, IMO, might be more stress (processing wise)
on the implementation without any significant benefit.. because the fact
still remains: sender will have to retransmit each and every packet, upon
expiry of a timer. Besides, there is a limit to which you can decrease this
timer because otherwise you run into a situation where you would end up
sending duplicates too early.

> Also, close your window on retransmission.  That will help 
> you recover when  your peer is dropping out of order packets.
Two points:
[1]
I wish it was easy to find out if peer is intentionally not queuing the
packets! Because there can be genuine network drops for a period of time
that can lead to similar conclusions. 

[2]
On closing a window to solve the problem: What if peer's window size is
3000? So you are saying retransmit all of them in one (or a couple)
re-transmits to try to close the window? And by luck if any one of them gets
dropped by the network peer will drop the rest of them anyways. And
retransmit them?

thanks,
-- vipin

> --
> Greg

---

From: Klausberger Walter <[email-masked]>
Date: Thu, 23 Aug 2001 14:17:18 +0200

> -----Original Message-----
> From: Paul W. Howard [mailto:[email-masked]]
> Sent: Wednesday, August 22, 2001 9:45 PM
> To: Ignacio Goyret
> Cc: Vipin Jain; [email-masked]' '
> Subject: Re: rfc2661: drop out of order (within rx-window) packets..
> 
> 
> > At 11:28 AM 8/22/01 -0700, Vipin Jain wrote:
> > >
> > >Some input for the folks who are working on new l2tp rfc:
> > >
> > >Today's l2tp spec, rfc 2661 section 5.8. para 5, mentions:
> > >" Messages arriving out of order may be queued for in-order
> > > delivery when the missing messages are received, or they
> > > may be discarded requiring a retransmission by the peer"
> > >
> > >Further, same section, rfc recommends that retry time for
> > >a packet should be 1 second. If an implementation chooses
> > >to drop out of order packet as opposed to queue them, it
> > >would take 1 second for a control packet to be transmitted
> > >(once a control message is lost in the network).
> > >
> > >In such situations, if we get incoming call rate of more
> > >than 1 call request per second we will never be able to
> > >recover from that situation?
> > >
> Hmmm...I'd suggest that this presents scalability problems of it's own
> as you now double the number of packets requires to setup each session
> (since you are setting up a tunnel for every session) and you are
> increasing the increasing the amount of traffic from 1 hello per hello
> interval to 'n' hellos per hello interval (given 'n' sessions are now
> over 'n' tunnels instead of over 1 tunnel).
> 
> >
> It might be better with TCP handling the reliability, but this would
> increase the load on the system as we would now have both L2TP and TCP
> doing reliable transmission.  If we were to drop the 
> reliability of the
> L2TP control channel, then we wouldn't have reliability for L2TP over
> other transports such as ATM.

I already thought about suggesting SCTP for doing reliable transmission (As
I heard there is a mixed mode reliable/unreliable planned), but your
argument seems reasonable.

What about a reliability layer for L2TP control connections over ATM?

regards
- walter

> 
> >
> >
> > The fixed 1-second initial timeout is quite lousy in high call rate
> > systems, as you found out.
> 
> Paul Howard
> 
>

---

From: "Reinaldo Penno" <[email-masked]>
Date: Fri, 24 Aug 2001 12:16:27 -0700

Only one comment..

> From: W. Mark Townsley [mailto:[email-masked]]
> Sent: Friday, August 24, 2001 1:31 AM
> To: Jain, Vipin [SC9:T337:EXCH]
> Cc: 'Greg Burns'; [email-masked]
> Not really packets, TCP windowing operates at the byte level. 
> So, it may queue
> bytes received out of order.
> 
> Anyway, TCP is known to not behave well when there is a great 
> deal of packet
> reordering. I can dig up some reports on this if you like.
> 
> > 

I think this was a problem with the fast retransmit algorithm on TCP Reno
code and is mentioned on Richard Stevens' book. Selective Acks should
overcome this issue, since you do not have to send 3 acks to request a
retransmission anymore.

regards,

Reinaldo

---

From: "Vipin Jain" <[email-masked]>
Date: Tue, 28 Aug 2001 09:17:47 -0700

Mark,

Agree with you. Smart implementations can deal with various situations you
mentioned below. Rx window should be interpreted as Max RX Window. 

However I would really like to know how to interpret following from the rfc:
" Messages arriving out of order may be queued for in-order
delivery when the missing messages are received, or they
may be discarded requiring a retransmission by the peer"

I am not sure about the discussions that went when l2tp became an rfc. Can
you shed some light on the context when introducing "may be discarded
requiring retransmission"? Is it mentioning every out of order packet can be
dropped, or is it mentioning that they can be dropped when packets can't be
queued and/or when resource allocation problems are detected. Because if
there is a context in which this statement should be interpreted then I'd
say its missing.

This will help clarify my doubt as well.

thanks,
-- vipin

-----Original Message-----
From: W. Mark Townsley [mailto:[email-masked]]
Sent: Friday, August 24, 2001 1:31 AM
To: Jain, Vipin [SC9:T337:EXCH]
Cc: 'Greg Burns'; [email-masked]
Subject: Re: rfc2661: drop out of order (within rx-window) packets..

Comments on window-size and retransmission inline.

- Mark

> Vipin Jain wrote:
> 
> Greg,
> 
> thanks for your comments.. some quick comments inline.
> 
> > I just meant to point out the timer is not fixed.  I used to
> > let it go much
> > lower than 1 second because I often found LAC/LNS to be on a LAN.
> agree; timer not being fixed, like what TCP does, is one way out.
> 
> > >Anyways, decreasing the timeout, IMO, might be more stress
> > (processing
> > >wise) on the implementation without any significant
> > benefit.. because the
> > >fact still remains: sender will have to retransmit each and
> > every packet,
> > >upon expiry of a timer.
> >
> > Well, whatever packets are in the front half of the window.
> > Next timeout, it's
> > a quarter of the window.
> again agree; like what TCP does.. as mentioned in Annex A rfc2661
> 
> > >  Besides, there is a limit to which you can decrease this
> > timer because
> > > otherwise you run into a situation where you would end up sending
> > > duplicates too early.
> >
> > I used the adaptive algorithm from TCP-land.
> Well, TCP also queues receive window packets that are out of order..

Not really packets, TCP windowing operates at the byte level. So, it may
queue
bytes received out of order.

Anyway, TCP is known to not behave well when there is a great deal of packet
reordering. I can dig up some reports on this if you like.

> 
> > I know.  But I think both implementations are valid.  If you want
> > interstellar probe
> > level of reliability, you preallocate all resources up to the
> > theoretical max.
> > Now lots of L2TP boxes can't afford to do that, especially
> > with all the tunnels
> > some customers seem to want.  So I'd hate to leave open a
> > corner case where
> > you run out of queued control message buffers.
> That's precisely is my point as well: "corner cases" is what the packets
are
> meant to be dropped. Not always. Despite, if a node wishes to drop every
out
> of order packets, then it better advertise its rx window size as "1" not
> anything greater. This would be give an optimal network performance. 

I think you are focusing too much on one case, Vipin. An advertised window
of 1
essentially puts the control channel into a "stop and wait" mode. Thus, each
message sent must be delayed by the 2*RTT time between the LAC and LNS + ack
processing time at the peer since the last message was sent. Only in a
drastic
state of misordered packets or packet loss, or an implementation so limited
in
ability that it can only operate on packets in real time without any queuing
ability whatsoever would this ever be optimal. For the former, the
algorithms in
L2TP should dynamically adjust to this situation anyway. e.g. if a great
deal of
packet loss is seen, one should reduce the transmit window exponentially,
all
the way down to 1 if need be (unlike TCP, we cannot fall to less than a 1
packet
windowsize). The latter is a statement about the peers ability to queue and
process packets (and, to some much smaller degree, the number of packets
allowed
to be in transit or queued within the intervening network).

So, again, the current and advertised initial RX buffer in L2TP is a
statement
about the number of packets that are allowed to be in transit between the
FSMs
on two tunnel peers at a given moment. 
Those packets may be queued within the sending box (if its output interface
is
throttled), queued somewhere on the network, in transit, or sitting in an
interface or process level queue at the receiving node. If there are truly
no
queues whatsoever between the LAC and LNS, and your peer can only operate on
one
message at a time, then an RX window of 1 makes sense. However, this is a
fairly
pitiful (and hopefully unrealistic) state of implementation.

If you look at the algorithm carefully, one can surmise that an L2TP
implementation should advertise the RX window that is based on a perfect
situation. e.g. no packet loss, low load on the box, etc. (perhaps the AVP
should be renamed to "Maximum" RX Windowsize). If packets get dropped, the
sender should reduce the window size and retransmission rate accordingly.
This
packet loss may be due to load at the receiving end, or simply congestion on
the
intervening network. Like TCP, the method for increasing and decreasing the
window size and retransmission rates follow a slow start and congestion
avoidance mode... e.g. decreasing exponentially and increasing linerally or
exponentially depending on the mode.

> Most
> efficient implementation of adaptive transmit windowing can't catch up
with a
> peer dropping out of order in-window packets creating an artificial
> congestion.

Agreed. However, ultimately one must be just as resiliant in a high packet
loss
situation as in a highly misordered situation. Even if you implement a
reordering scheme locally, you get into problems where, while waiting on an
"out
of order" packet, your peer triggers an unecessary window of retransmissions
if
the timing is not just right. This is one of the problems with TCP. Thus,
chronic packet misordering, in general, is something that is not tolerated
well
on the internet. If you are seeing problems between an LAC and LNS, I
suspect
that your TCP performance is probably suffering as well (though it may not
be as
obvious to you as it is to your end-users!).

> > >re-transmits to try to close the window? And by luck if any
> > one of them
> > >gets dropped by the network peer will drop the rest of them
> > anyways. And
> > >retransmit them?
> >
> > See above.  Get a burst of 3000 messages within the timeout
> > period and drop the
> > first one and 1500 get retransmitted (close first, then retransmit).
> I see your point of view.
> 
> thanks,
> -- vipin