Subject: [tcpm] Proposing ACKNOW bit in TCP header

From: Andre Oppermann <[email-masked]>
Date: Wed, 15 Nov 2006 14:00:23 +0100

I propose to include an ACKNOW bit into the TCP header instructing the
recipient to immediately, without any delay whatsoever, acknowledge
the received segment.  The bit location is the 9th bit in the TCP flags.
This bit is to be included in TCP SYN segments to indicate the support
for it.  Only if the SYN-ACK response has the bit set as well the session
is allowed to make use of the ACKNOW feature.

Rationale:

During the slow start and congestion recovery phases the sender is
dependent on immediate feedback to quickly make optimal use of all
network resources again.  The presence of delayed ACKs add unnecessary
delays and introduces uncertainty to the recovery progress and the
discovery of the true network conditions.  On the other hand delayed
ACKs serve an important function on fast networks with high PPS rates
to improve efficiency and to coalesce the acknowledging of multiple
segments at once.  It is extremely difficult for the receiver to guess
itself when to switch from delayed ACKs to immediate ACKs as it has
only insufficient hints the current state the sender operates in.

The ACKNOW flag solves this in a simple and elegant way by allowing the
sender to explicitly request an immediate ACK to its segments when it is
appropriate for the congestion control recovery mechanism to do so.
As a second benefit a receiver on a ACKNOW flag enabled session may
increase the delayed ACK from every other packet (as currently required)
to a higher number, lets say every eighth.  The exact value is to be
determined based on the estimated delay*bandwidth product and the send/
receive socket buffer sizes.

The use of the ACKNOW bit allows significant faster recovery times in
the presence of difficult network conditions.

-- 
Andre

<[email-masked]>

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: "Sushant Rewaskar" <[email-masked]>
Date: Wed, 15 Nov 2006 10:16:25 -0500

Hi,
Adding a new interpretation to bits in TCP header would be one way of doing
it. I briefly thought about this problem and here is another way you can do
it without changing the header fields (which lots of people are dead
against).

Also, by 9th bit in the header flags did you mean the PUSH bit?, If yes,
what about the implications of overwriting the default meaning associated
with it?

Anyway, here is an alternate solution. We are assuming that the sender know
when it "needs" the "quick ack" mode and tells the receiver accordingly. The
rationale was that with delayed acks the window does not open to twice the
current window but to just 1 and half time the current window. Anyway, my
point is that if sender knows that acks are delayed why not interpret them
differently. Everything you get an ack in the "special state" (when you need
immediate acks) increase your window by two and not one MSS. (the only
different is that in this method loss of one ack will be more harmful then
in the earlier method, but then you can take care of that as well, depend on
how much code and processing you want to add to the sender.)

Of course, I am just presenting an alternate method. The benefits of doing
something like this have still to be argued. (Hopefully with do that soon)

Take care,
Sushant Rewaskar
-----------------------------
UNC Chapel Hill
www.cs.unc.edu/~rewaskar 
 
-----Original Message-----
From: Andre Oppermann [mailto:[email-masked]] 
Sent: Wednesday, November 15, 2006 8:00 AM
To: [email-masked]
Subject: [tcpm] Proposing ACKNOW bit in TCP header

Rationale:

-- 
Andre

<[email-masked]>

---

From: Andre Oppermann <[email-masked]>
Date: Wed, 15 Nov 2006 16:37:32 +0100

Sushant,

I counted the bits from the other side, it's the unused bit next to CWR.
Extending the semantics of the PUSH bit would be possible but be quite
hackish IMHO.

Your alternative is not an option.  The sender has no knowledge if delayed
ACK is enabled on the receiver at all.  Upping the window in double steps
would be too aggressive and violate the TCP spec and spirit.  Neither is
it a solution to the congestion recovery case which is very important as
well.

Occupying bits in the TCP header wasn't too difficult for ECN and ECN
has only enjoyed moderate uptake.  ACKNOW is much simpler and has much
more immediate and tangible benefits than ECN.

-- 
Andre

---

From: "Sushant Rewaskar" <[email-masked]>
Date: Wed, 15 Nov 2006 11:13:42 -0500

Hi Andre,
You points are valid. As I said I had not thought it through, I can change
my solution to take care of these objections (for e.g. to overcome the
sender not knowing when the receiver is using delayed ack we simply do not
change the window by a fixed amount but by the number of bytes acked by that
ack, this way we also preserve the basic principle of TCP and are not
aggressive). But then it may go on for a while and in the end my solution
may be more of a hack then an elegant solution. I will probably flesh out
more details before I present my new and improved solution. 

I also did not get what you meant by "it is not a solution to congestion
recovery case", are you talking about fast retransmit and recovery?

Do you have a draft or some more details about what you are proposing? 

Take care,
Sushant Rewaskar

-----Original Message-----
From: Andre Oppermann [mailto:[email-masked]] 
Sent: Wednesday, November 15, 2006 10:38 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [tcpm] Proposing ACKNOW bit in TCP header

Sushant,

-- 
Andre

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 15 Nov 2006 11:29:43 -0500

On Wed, Nov 15, 2006 at 11:13:42AM -0500, Sushant Rewaskar wrote:
> I can change
> aggressive).

This sounds like RFC 3465.

-- 
Wesley M. Eddy
Verizon Federal Network Systems

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 Nov 2006 08:46:42 -0800

Hi, Andre,

Andre Oppermann wrote:
> the received segment. 

This sounds like a fairly substantial change to TCP semantics. TCP has
only two kinds of headers that require a direct reply - SYN and FIN.

It seems like all other interactions are driven by only data packets,
and require the preceding data to have been correctly received to
generate a positive ACK. I.e., short of start/end of a connection, TCP
is more data driven than header driven. This would require a substantial
change to that semantic, which seems like it would require a substantial
revision.

Forcing the other end to react is also a DOS opportunity.

Can anyone speak to whether SCTP handles this better, since this sounds
like a signalling issue?

> Rationale:
>
>...It is extremely difficult for the receiver to guess

Can you clarify this? I thought delayed (compressed 2:1) ACKs were
required, period. It's only on the send side that delays are avoided,
e.g., depending on NAGLE.

> ...As a second benefit a receiver on a ACKNOW flag enabled session may

This would increase burstiness in the network, which would affect
traffic sharing paths through non-RED routers at least. Also, unless ABC
is supported (3465), this would affect the fairness of TCP's ACK clocking.

> the presence of difficult network conditions.

I don't think this should be a goal of a TCP modification, UNLESS the
modification can be shown to do no harm in other, equally difficult
network conditions.

Whether this mod passes that test is not yet clear.

Joe

---

From: Andre Oppermann <[email-masked]>
Date: Wed, 15 Nov 2006 18:46:14 +0100

> only two kinds of headers that require a direct reply - SYN and FIN.

I beg to differ, see below.

> change to that semantic, which seems like it would require a substantial

I think you're putting a bit too much into ACKNOW.  The only thing that
happens with ACKNOW flag set is the same as what happens on every other
packet anyway, an ACK segment is generated by the receiver.  The very
same rules as ever apply to the values in it.

The only meaning of ACKNOW is *not* to delay the ACK that would eventually
happen anyway.

Perhaps you were thinking that ACKNOW must ACK the current segment
but that is not the case.  The normal rules of ACKing only what has
been received in sequence and yaddayaddayadda applies (SACK...).

> Forcing the other end to react is also a DOS opportunity.

How so?  The other end would react anyway if it received this and the
following segment.

> e.g., depending on NAGLE.

Nope.  RFC2881 Section 4.2 lists them as SHOULD.  It is perfectly legal
for an implementation to ACK every packet.  On the other hand it is not
legal (and not efficient without ACKNOW or some reliable heuristic) to
ACK less than every other segment.  With ACKNOW it can easily be made
legal and also cuts down on the network pps load quite a bit.

> is supported (3465), this would affect the fairness of TCP's ACK clocking.

Quite the opposite is the effect.  With every ACK you have a much smoother
curve than to do larger jumps where two segments are ACKed at the same
time and may cause even larger jump bursts.

> modification can be shown to do no harm in other, equally difficult

It doesn't.  ACKNOW is a hybrid between a (perfectly legal) implementation
ACKing every segment and one ACKing only every other.  ACKNOW gives the
advantages, benefits and efficiency of both worlds without the downsides
of each (slower recovery times for delayed ACK vs. higher packet rate for
every seg ACK).

-- 
Andre

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 Nov 2006 10:00:07 -0800

> been received in sequence and yaddayaddayadda applies (SACK...).

Yes, I was. So this means "ACK this immediately" where "immediately"
means only when all the data up to this point has been received?
Otherwise, how do you know that the ACK you get back had anything to do
with the ACKNOW?

> How so?  The other end would react anyway if it received this and the

It is only in the case I assumed above.

> legal and also cuts down on the network pps load quite a bit.

Without ABC, it'll slow the CWND algorithms.

> time and may cause even larger jump bursts.

You suggested an ACK every 8th (e.g.); that means that the ACK would
cause the window to jump 8 segments, which means the sender would burst
them out.

> of each (slower recovery times for delayed ACK vs. higher packet rate for

It seems like it requires ABC or you have unpredictable effects on
congestion window growth. If you add this to ABC, do you still get a
benefit?

Joe

---

From: David Borman <[email-masked]>
Date: Wed, 15 Nov 2006 12:35:05 -0600

Andre,

On the surface this might seem like a clean and simple idea, but it  
really isn't.  There are other ramifications of such a change, and  
the statement of what is really desired needs to be explored more  
deeply.

As I read your description for ACKNOW, it's a way for the sending TCP  
to say "I'm not sending any more data, so send me an ACK now!"  I  
really don't see the need for a separate ACKNOW bit, this same  
functionality could be implemented using the PUSH bit (which says  
"I'm not sending any more data, so make all the data I've sent so far  
available to the application".)  Hence, the ACKNOW functionality as  
you describe could be implemented just by using the PUSH bit.  But  
that doesn't address the issue of whether or not this is a good thing  
to do.

Returning ACKs are not just used to clock out new data and open the  
congestion window, they are also used for calculating RTT, which is  
used for deciding when to retransmit unacknowledged segments.  If  
your RTT is too short, you are in danger of retransmitting too soon.   
So your RTT isn't just how long it takes packets to get to the other  
side, it also needs to account for any latency, including the time it  
takes for delayed ACKs to be sent.  Every ACKNOW packet is an  
exception to the normal delayed ACK processing, and will produce a  
RTT measurement that does not account for latency due to delayed ACK  
processing.

Decreasing the length of delayed ACKs (as suggested, to every 8th  
packet) could cause harm to the clocking out of new data, based on  
the ACKs.  In steady state, I could see this leading to even burstier  
traffic, unless the sender is regularly setting the ACKNOW bit, in  
which case we now have the sender deciding the delayed ACK strategy  
for the receiver...  that seems like a potential source of problems.

In general, though, I think that a better approach is to develop a  
TCP sending strategy that takes into account the presence of delayed  
ACKS, rather than trying to develop a strategy the circumvents the  
presence of delayed ACKs.  For example, just increasing the initial  
TCP window (RFC 3390) to two segments can eliminate the latency  
induced by delayed ACKs at connection start-up.  Adding an ACKNOW bit  
is a solution, not a problem.  Discussion at this point should be  
focused on what the problem is that needs addressing, rather than  
whether a particular solution to that problem is a good solution.

			-David Borman

---

From: Andre Oppermann <[email-masked]>
Date: Wed, 15 Nov 2006 19:40:27 +0100

> Otherwise, how do you know that the ACK you get back had anything to do

I don't.  And I don't have to.  All I'm requesting with the ACKNOW flag
is to produce an response *now* instead of potentially waiting for 200-
1000ms until the delayed ACK timer at the receivers end expires.

> Without ABC, it'll slow the CWND algorithms.

You mean the congestion avoidance algorithm as defined in RFC2581 3.1?
Yes, it would slow it down.  OTOH ACKNOW is under the control of the
sender and until a steady state plateau is reached it may set ACKNOW
on every other segment after the initial slow start phase where it was
set on every segment.

> cause the window to jump 8 segments, which means the sender would burst

Which is not much of a problem.  Even RFC2414 allows up to 4 segments to
burst into the network in the initial send of a just established connection.
Things like TSO (TCP segmentation offload) get very happy with this.  As
long as it is not every 1/100th segment networks won't break down.  Although
I agree that this topic should be clearly discussed and specified in any
RFC that may come out of the ACKNOW proposal.  You have to keep in mind
though that the centerpiece of ACKNOW is not to allow further spaced ACKs
but to selectively override delayed ACK which causes much pain and inefficiency
for fast recovery.

> congestion window growth. If you add this to ABC, do you still get a

No, it doesn't require ABC.  Yes, with ABC the benefits stay the same.
See the formula (1) in RFC2581 Section 3.1.  (Leaving the ACK stretching,
which is a separate and way less important topic, out of the picture.)

-- 
Andre

---

From: Andre Oppermann <[email-masked]>
Date: Wed, 15 Nov 2006 19:59:25 +0100

David Borman wrote:
> really isn't.  There are other ramifications of such a change, and the 
> statement of what is really desired needs to be explored more deeply.
> 
> As I read your description for ACKNOW, it's a way for the sending TCP to 
> say "I'm not sending any more data, so send me an ACK now!"  I really 
> don't see the need for a separate ACKNOW bit, this same functionality 
> could be implemented using the PUSH bit (which says "I'm not sending any 
> more data, so make all the data I've sent so far available to the 
> application".)  Hence, the ACKNOW functionality as you describe could be 
> implemented just by using the PUSH bit.  But that doesn't address the 
> issue of whether or not this is a good thing to do.

Nope.  I don't particularly care about a quick ACK when I'm finished
with sending.  I care about quickly reaching steady state from either
connection start (slow start) or a loss recovery when I've got a lot
more yet to send.

> congestion window, they are also used for calculating RTT, which is used 
> for deciding when to retransmit unacknowledged segments.  If your RTT is 
> too short, you are in danger of retransmitting too soon.  So your RTT 
> isn't just how long it takes packets to get to the other side, it also 
> needs to account for any latency, including the time it takes for 
> delayed ACKs to be sent.  Every ACKNOW packet is an exception to the 
> normal delayed ACK processing, and will produce a RTT measurement that 
> does not account for latency due to delayed ACK processing.

Your RTT can't be too short.  It can, and too often is, too long.
The definition of RTT is very simple and is the time it takes from
leaving the senders socket buffer to arrival in the receivers socket
buffer.  No artificial ACK delays included.  It's a fact that ACKs
get delayed by the delayed ACK mechanism and that really screws up
fast recovery because delACK can't be turned off reliably.  To optimally
size my send and receive socket buffers I want to know the real RTT
and not some artificial number with +0-500ms variance.

> packet) could cause harm to the clocking out of new data, based on the 
> ACKs.  In steady state, I could see this leading to even burstier 
> traffic, unless the sender is regularly setting the ACKNOW bit, in which 
> case we now have the sender deciding the delayed ACK strategy for the 
> receiver...  that seems like a potential source of problems.

Well, the ACKs clock the sender so it should have some say in there.
It may allow for smarter congestion control algorithms.  Like I
said in other replies the ACK stretching is not the core rationale
but a possibly useful side effect given the right care.

> In general, though, I think that a better approach is to develop a TCP 
> sending strategy that takes into account the presence of delayed ACKS, 
> rather than trying to develop a strategy the circumvents the presence of 
> delayed ACKs.  For example, just increasing the initial TCP window (RFC 
> 3390) to two segments can eliminate the latency induced by delayed ACKs 
> at connection start-up.  Adding an ACKNOW bit is a solution, not a 
> problem.  Discussion at this point should be focused on what the problem 
> is that needs addressing, rather than whether a particular solution to 
> that problem is a good solution.

TCPM history is rich with such attempts to algorithmic blind guessing.
So far none seems to have worked out really well or it would be widely
implemented already.

-- 
Andre

>> --Andre

---

From: David Borman <[email-masked]>
Date: Wed, 15 Nov 2006 13:31:24 -0600

> David Borman wrote:
...
>> As I read your description for ACKNOW, it's a way for the sending  
>> TCP to say "I'm not sending any more data, so send me an ACK  
>> now!"  I really don't see the need for a separate ACKNOW bit, this  
>> same functionality could be implemented using the PUSH bit (which  
>> says "I'm not sending any more data, so make all the data I've  
>> sent so far available to the application".)  Hence, the ACKNOW  
>> functionality as you describe could be implemented just by using  
>> the PUSH bit.  But that doesn't address the issue of whether or  
>> not this is a good thing to do.
>

Hey, I didn't say it was a good or well thought out example. :-)  I  
wasn't thinking of the PUSH bit as only being set when there is no  
more data to send, it could also set it when you won't send anymore  
data, even if you do have more data to send.  The point is, you have  
more data to send, you aren't going to send it until you get an ACK  
from the other side, so you want a quick ACK and the PUSH bit could  
be used to indicate that instead of adding a new ACKNOW bit.  The  
fact the ACK would also come back quicker when you are finished  
sending doesn't matter.  And I'm not saying that I'd support adding  
these semantics to the PUSH bit any more than I'd support adding an  
ACKNOW bit.

>
>> Returning ACKs are not just used to clock out new data and open  
>> the congestion window, they are also used for calculating RTT,  
>> which is used for deciding when to retransmit unacknowledged  
>> segments.  If your RTT is too short, you are in danger of  
>> retransmitting too soon.  So your RTT isn't just how long it takes  
>> packets to get to the other side, it also needs to account for any  
>> latency, including the time it takes for delayed ACKs to be sent.   
>> Every ACKNOW packet is an exception to the normal delayed ACK  
>> processing, and will produce a RTT measurement that does not  
>> account for latency due to delayed ACK processing.
>

Umm... no.  RTT = Round Trip Time.  It's not one way, it's how long  
it takes to send data and to get back and ACK for that data.  And  
that includes any latency, including delayed ACKs.  It's the  
returning ACK that prevents retransmissions, so decisions on when to  
retransmit need to take into account the full RTT, including delay in  
the ACKs, or you may have unnecessary retransmits due to delayed ACKs.

...
>> TCP sending strategy that takes into account the presence of  
>> delayed ACKS, rather than trying to develop a strategy the  
>> circumvents the presence of delayed ACKs.  For example, just  
>> increasing the initial TCP window (RFC 3390) to two segments can  
>> eliminate the latency induced by delayed ACKs at connection start- 
>> up.  Adding an ACKNOW bit is a solution, not a problem.   
>> Discussion at this point should be focused on what the problem is  
>> that needs addressing, rather than whether a particular solution  
>> to that problem is a good solution.
>

The perhaps the focus should be on the algorithmic delayed ACK  
strategy, to make it deterministic.

			-David Borman

---

From: Andre Oppermann <[email-masked]>
Date: Wed, 15 Nov 2006 21:22:05 +0100

David Borman wrote:
> wasn't thinking of the PUSH bit as only being set when there is no more 
> data to send, it could also set it when you won't send anymore data, 
> even if you do have more data to send.  The point is, you have more data 
> to send, you aren't going to send it until you get an ACK from the other 
> side, so you want a quick ACK and the PUSH bit could be used to indicate 
> that instead of adding a new ACKNOW bit.  The fact the ACK would also 
> come back quicker when you are finished sending doesn't matter.  And I'm 
> not saying that I'd support adding these semantics to the PUSH bit any 
> more than I'd support adding an ACKNOW bit.

Redefining the PUSH bit this would seem risky and really dirty as the
meaning is really different.  And there would be no way for the two
endpoints to distinguish if the opponent supports the ACKNOW feature
or not.

> Umm... no.  RTT = Round Trip Time.  It's not one way, it's how long it 
> takes to send data and to get back and ACK for that data.  And that 
> includes any latency, including delayed ACKs.  It's the returning ACK 
> that prevents retransmissions, so decisions on when to retransmit need 
> to take into account the full RTT, including delay in the ACKs, or you 
> may have unnecessary retransmits due to delayed ACKs.

I stand corrected.  I was thinking of what you said regarding full round
trip but didn't write it.  ;-)

The delayed ACK tries to squash a two of flies at once with varying
success:

  a) ACK only every other segment to conserve bandwidth/pps

  b) to give the chance to piggyback the ACK on a data segment back to
     the sender

Ideally the delayed ACK timer calibrates itself to once RTT + fudge
like the retransmission timer instead of being a fixed value.  That
way it can generate much more timely feedback to the sender.  For b)
the sender could implement a fairly simply heuristic to determine
if this applies at all and a ping-pong protocol is active.
Unfortunately this can only come to play after a number of segments
have been exchanged and the measurements have converged and settled
thus we lose out on the early part of the connection where it matters
most.  ACKNOW would solve that.

> The perhaps the focus should be on the algorithmic delayed ACK strategy, 
> to make it deterministic.

Making a black box deterministic is a very fine art. ;-)

-- 
Andre

---

From: Ethan Blanton <[email-masked]>
Date: Wed, 15 Nov 2006 16:39:03 -0500

> 1000ms until the delayed ACK timer at the receivers end expires.

I'm not convinced we're all talking about the same problem here.  This
kind of delay will *only* happen to the last packet in a flight,
because otherwise the next packet to arrive will clock out the ACK.
Additionally, 500ms is specified as the *maximum* delay that a
receiver can impose.  (Both of these are in RFC2581, section 4.2.)

In addition, segments being ACKed during loss recovery SHOULD be ACKed
immediately (also RFC2581, section 4.2).

I think you're solving a completely imaginary problem with ACKNOW.
Please show us a real scenario (which is not handled by ABC, RFC3465)
where delayed ACKs pose a real performance problem.  All of the
scenarios I have seen cited are misunderstandings, and don't happen in
the real world.

Ethan

-- 
The laws that forbid the carrying of arms are laws [that have no remedy
for evils].  They disarm only those who are neither inclined nor
determined to commit crimes.
		-- Cesare Beccaria, "On Crimes and Punishments", 1764
_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Mark Allman <[email-masked]>
Date: Wed, 15 Nov 2006 23:44:52 -0500

I just took a spin through today's thread and am not going to do a
point-by-point reply to everything.  But, I have thought about these
issues a little bit over the years and have a few comments (these are
not all in response to Andre's proposal or comments, but to the thread
in general):

  * If the goal is to do slow start more quickly by coaxing the receiver
    into sending more ACKs then you should compare this with ABC.  If
    you go back and chase the references in the ABC document you will
    find an old CCR paper that I wrote that plays with something called
    "delayed ACKs after slow start" and compares that with byte counting
    (the paper uses simulations and the receiver knew when the sender
    was in slow start through this wonderful mechanism of a global
    variable---just a test of feasibility not of mechanism).  Indeed, as
    noted in the thread, getting more ACKs during slow start does make
    things a bit smoother than the slightly more bursty byte counting
    approach.  However, the conclusion is that this difference is fairly
    minor and so using the sender-only approach seems reasonable.  

    Of course, if you have some evidence that shows different results
    and that the burstiness of something like ABC matters then that
    would be very useful to see (even if not for ACKNOW, but as a
    comment on whether ABC should be standards-tracked at some point).

  * Contrary to the claim in the thread, ABC is not over-aggressive and
    does in fact follow the spirit of congestion control.  As noted,
    delayed ACKs are not strictly required and ABC is never more
    aggressive than an ACK counting sender would be in the face of a
    receiver that ACKs every packet.

  * The other big claim about ACKNOW seems to be that it aids loss
    recovery.  But, I am not sure how.  As Ethan noted, RFC 2581 says
    that receivers SHOULD ACK every segment when there is a gap in the
    sequence space (i.e., during loss recovery).  To expound on this a
    little... The problem is that the receiver is in a great position to
    understand when to change to immediately ACK to aid loss recovery
    and the sender is in a lousy position to drive the process with
    ACKNOW.  When the sender figures out that some segment X has been
    lost and so initiates loss recovery an entire window of data behind
    X has already been sent.  And, it is precisely this window of data
    that will generate the duplicate ACKs (and SACKs) to drive loss
    recovery.  So, by the time the sender understands there is loss the
    segments that would be best to use with ACKNOW are already in
    transit.  However, the receiver knows there is something wrong as
    soon as the first duplicate ACK is sent and so can switch gears
    easily to immediate ACKing and have a real impact on the process.

  * The broader point here is that I think it is a mistake to think that
    the sender always knows best.  We do not need one "best" vantage
    point, IMO.

  * ACKing every packet is legal.  RFC 2581 says that stacks SHOULD ACK
    every other packet.  I take this to mean that you ought to have a
    pretty good reason to do something different.  So, while it is
    technically OK to ACK every segment I think it is a discouraged
    practice, in general.

  * It may be useful to increase the ACK interval beyond 2.  There is
    some evidence that shows this to be useful to performance in some
    cases and there is a general notion that the reverse path could
    stand to be congestion controlled itself.  There is also clearly
    evidence that extending the ACK interval makes things more brittle
    (e.g., easier to lose a window of ACKs and be stuck doing full
    restart).  Maybe it is a good idea to make this dynamic.  I don't
    know.  But, I know that I'd want to see more than hand wavy claims
    about easy heuristics before I believe it.  Further, I have a
    feeling that if one really wanted to do this it may well take
    exchanging more than one bit of information.  Maybe not.  I have no
    idea.  Someone would need to wring this out.

  * TCP does not care about RTTs.  TCP uses RTTs to figure out an
    *RTO*.  That is, TCP just wants to know how long to wait before
    it should assume loss and retransmit.  Clearly, that amount of time
    should include everything from the propagation delay to the queueing
    time to the amount of time the receiver twiddles its thumbs waiting
    to ACK a packet (and, everything in between).

    That said, delayed ACKs do not have a huge impact on the RTO of a
    steady-state TCP connection because delayed ACKs are basically not
    delayed all that long.  That is, when sending bunches of packets the
    time an ACK is delayed is just until the next packet arrives.  (And,
    half the packets are delayed not at all.)  Let's not forget that
    delayed ACKs are usually clocked out by packets not timers.

    So, I am not sure that I buy Dave Borman's notion that ACKNOW
    somehow sets us up to measure fundamentally different RTTs that are
    then thrown into the estimator.  My *guess* is this is in general a
    small effect.  (Although, I am sure an example where there was a big
    effect isn't that hard to cook.)

  * It would be nice if the easy heuristics and algorithms were tried
    and the "tangible benefits" were shown.  We've all heard (and made!)
    those sorts of hollow claims before.

I don't want to say that this ACKNOW bit is a totally dumb idea.  Maybe
it isn't.  But, from the mail thus far I don't think it sounds like it
is really accomplishing much.  But, I am game to be proven wrong with
some evidence.  The devil is in the details.

allman

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 16 Nov 2006 09:35:41 +0100

Hi Joe,

comment in-line.

Best regards
Michael

> Can anyone speak to whether SCTP handles this better, since this  
SCTP should SACK every other packet containing DATA chunks. However,
if the receiver detects gaps/duplicates, he must SACK every packet  
containing
DATA chunks.

The sender can only derive from RTT measurements if the receiver delays
SACK or not. In networks were small RTTs are necessary (also with  
reduced
RTO.min/RTO.max) it might be better to disable delaying SACKs or to
reduce the SACK delay time. However, there is currently no way for  
the sender
to enforce that behaviour on the receiving side.

It might be a nice thing because it would decouple the sender and  
receiver
from the RTO.min > SACK.delay constraint. Keep in mind that SCTP has two
different ways of measuring the RTT:
- based on DATA/SACK which will include the SACK.delay
- based on HEARTBEAT/HEARTBEAT-ACK which will not include the  
ASCK.delay.

---

From: Andre Oppermann <[email-masked]>
Date: Fri, 17 Nov 2006 15:13:27 +0100

> some evidence.  The devil is in the details.

I will produce conclusive evidence (or counter evidence) and report
back here in the next weeks.  ;-)

-- 
Andre

---

From: "Jeffrey Hsu" <[email-masked]>
Date: Fri, 17 Nov 2006 10:13:44 -0800

For those of you who want to experiment on a production system with ABC,
DragonFlyBSD, an advanced production-quality version of FreeBSD,
has ABC implemented.  You can turn it on or off with the net.inet.tcp.abc
sysctl variable.  SACK, SCTP, Limited Transmit, Eifel Detection,
Early Retransmit, and many other options are also available and
tunable with run-time sysctl settings.

							Jeffrey

---

From: Ted Faber <[email-masked]>
Date: Mon, 20 Nov 2006 13:18:53 -0800

> Mark Allman wrote:
[many things]
> back here in the next weeks.  ;-)

If you continue to believe ACKNOW is worth talking about, an internet
draft that lays out the case and describes the details would be a great
starting point.

Let me know, on- or off-list, if there's something I can to to make that
easier for you.

-- 
Ted Faber
http://www.isi.edu/~faber           PGP: http://www.isi.edu/~faber/pubkeys.asc
Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG
_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Pekka Savola <[email-masked]>
Date: Tue, 21 Nov 2006 09:49:14 +0200 (EET)

On Fri, 17 Nov 2006, Jeffrey Hsu wrote:
> has ABC implemented.

FWIW, Linux has had ABC for quite some time now as well.  First it was 
enabled by default, but no it's disabled now due to the performance 
problems in lots of applications which seem to rely on earlier 
behaviour.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Mark Allman <[email-masked]>
Date: Tue, 21 Nov 2006 06:47:24 -0500

> problems in lots of applications which seem to rely on earlier

Can you say more about this last part?  I assume by "earlier behavior"
you mean slower cwnd growth?  Or, strictly ACK counting?  Or, what?  I
guess I don't quite follow why applications are somehow caring about
this or expecting some behavior from TCP in this regard at all.

allman

---

From: Pekka Savola <[email-masked]>
Date: Tue, 21 Nov 2006 14:37:52 +0200 (EET)

> this or expecting some behavior from TCP in this regard at all.

Unfortunately, of late I don't think developers have been very 
interested in participating in the IETF, so it's difficult to know for 
sure.

These messages might shed some light to the subject.

Turning it off:
http://www.mail-archive.com/[email-masked]/msg21294.html

Why it's not applicable:
http://www.mail-archive.com/[email-masked]/msg21257.html

Some further info:
http://www.mail-archive.com/[email-masked]/msg21204.html
http://www.mail-archive.com/[email-masked]/msg21220.html
http://www.mail-archive.com/[email-masked]/msg21256.html

The start of the thread, and the second thread:
http://marc.theaimsgroup.com/?t=115693258400001&r=1&w=2
http://marc.theaimsgroup.com/?t=115738569600001&r=1&w=2

HTH..

---

From: Mark Allman <[email-masked]>
Date: Tue, 21 Nov 2006 07:49:59 -0500

> interested in participating in the IETF, so it's difficult to know for

Yes.  This is unfortunate for a number of reasons.

> These messages might shed some light to the subject.

I read them.  I guess I am still not sure what the concern is.  However,
I will readily admit that I didn't read the entire threads and sometimes
getting the big picture points from the middle of a thread is tough.  I
guess if we ever want to push ABC onto the standards track there ought
to be an action item to chat with these folks.

Thanks,
allman

---

From: Lars Eggert <[email-masked]>
Date: Tue, 21 Nov 2006 15:14:03 +0200

> interested in participating in the IETF, so it's difficult to know for

For Linux kernel developers, which is a community I'm not tuned into,  
can you venture a guess as to why this is (lack of travel funds,  
disinterest with the IETF, etc.)?

I agree with Mark that it would be very useful to have stack  
developers be involved in the process. We had been thinking of ways  
in which we can encourage such participation.

Lars

---

From: Pasi Sarolahti <[email-masked]>
Date: Tue, 21 Nov 2006 16:51:56 +0200

> this or expecting some behavior from TCP in this regard at all.

Linux implements the congestion control based on packet counts, not
based on byte counts. So after sending four 1-byte packets Linux sender
has used up the initial congestion window (assuming iwnd = 4 *packets*).
I haven't checked out the code for a long while, but just as a guess,
maybe this has caused some confusion with Linux ABC implementation.

- Pasi

---

From: Lloyd Wood <[email-masked]>
Date: Tue, 21 Nov 2006 15:41:04 +0000

>problems in lots of applications which seem to rely on earlier 

Do you have a pointer to information on the problems with applications?

I'd been presuming that ABC is universally a good thing, even though its use blocks the count-the-acks-not-the-bytes sender hole that allows simple ack-splitting intercept-one-ack-send-two 'TCP acceleration', which isn't outlined in RFC3135, but is implemented in Cisco IOS:
http://www.cisco.com/en/US/products/ps6441/products_feature_guide09186a0080683d9d.html

L.

---

From: John Heffner <[email-masked]>
Date: Tue, 21 Nov 2006 11:10:44 -0500

> to be an action item to chat with these folks.

The crux of the issue is that linux tracks cwnd in packets rather than 
bytes.  The failure mode occurs when you have some very latency-sensitve 
application does small writes with Nagle disabled (there are a number of 
these out there, and though I'm not sure exactly what they are, people 
scream loudly when they break).  The initial window allows a few 
segments, so you send out a few segments.  These get delayed-acked, and 
bytes_acked gets increased.  However, cwnd does not increase, because 
you can't until bytes_acked equals cwnd*mss.  So there is a 
de-amplification effect on the cwnd increase rate of (average 
segsize)/MSS.  The small segments get coalesced into larger segments 
while cwnd is blocking them, creating a long latency in a 
latency-sensitive application, defeating the whole purpose of disabling 
Nagle to begin with.

This is not an issue with the ABC spec, as it's defined in terms of cwnd 
tracking in bytes.  It's an unfortunate interaction between the two models.

   -John

---

From: John Heffner <[email-masked]>
Date: Tue, 21 Nov 2006 11:12:05 -0500

Lars Eggert wrote:
> I agree with Mark that it would be very useful to have stack developers 
> be involved in the process. We had been thinking of ways in which we can 
> encourage such participation.

I believe lack of funds is a big issue for some.  I would also guess 
that of the small number of "core" developers, most are extremely busy 
actually working on the kernel, and just can't afford or justify the 
large amount of time that active IETF participation requires.

There are a few of us out there actively working in both areas. ;)

   -John

---

From: Pekka Savola <[email-masked]>
Date: Wed, 22 Nov 2006 16:59:19 +0200 (EET)

I changed the subject line..

On Tue, 21 Nov 2006, Lars Eggert wrote:
> For Linux kernel developers, which is a community I'm not tuned into, can you 
> venture a guess as to why this is (lack of travel funds, disinterest with the 
> IETF, etc.)?

The issue is not TCP(m)-specific, but may be magnified as TCP has been 
around for a long time and is a protocol many hold very strong views 
about, so any maintenance easily ratholes in a layer higher than L7.

...

I suspect bad ROI -- the IETF process takes so long time and eats up 
so much energy that most rather spend it spend on the implementation 
work.

Further, currently the IETF is mostly useful as a body for new 
specifications.  The IETF hasn't paid much attention to protocol 
maintenance, so if you look at old protocols such as TCP, UDP or ICMP, 
the IETF has long since ceased being relevant as a forum for 
up-to-date behavioural descriptons.  If a developer needs to make a 
modification, they'll just do it -- no need to try to waste time with 
the IETF.

Fixing this would likely require finding ways to encourage 
low-overhead participation and evaluating how much running code has 
diverged from the implementations and trying to close that gap.

---

From: Stephen Hemminger <[email-masked]>
Date: Wed, 22 Nov 2006 10:39:22 -0800

On Wed, 22 Nov 2006 16:59:19 +0200 (EET)

> about, so any maintenance easily ratholes in a layer higher than L7.

To help you understand. Linux development is not centralized, there
are many community members. Several of the network developers,
watch this list (and the e2e list).
Any discussion with direct impact on Linux gets evaluated.

The Linux community works mostly through email with occasional face to face
meetings. The meetings are informal and dedicated to discussing issues
that are not getting enough traction. There has been an annual meeting
of the Linux network developers (http://vger.kernel.org/netconf2006.html)
as well as discussions at the kernel summit (http://www.usenix.org/events/kernel06/).
 
> so much energy that most rather spend it spend on the implementation 

If you look at who contributes to Linux networking; there are contributions
from distributions (Redhat, Novell, Ubuntu), non-profits (OSDL), and
research (NCSU, Hamilton, etc).  Each contributor has there own needs
and spending time on IETF issues is not a priority for most.
For many, getting funding and time to go to IETF meetings has
been impractical. If there are issues and agenda items that warrant
discussion with IETF, I am sure we can get several people to go to
one of the meetings.

> modification, they'll just do it -- no need to try to waste time with 

Any practical commercial implementation is a hybrid of specifications
from standards organizations, and heuristics based on real world feedback
and device quirks.

> diverged from the implementations and trying to close that gap.

There is also, the possibility of more focused groups (security?),
or getting cross vendor participation. Several vendors
use Linux in their products and participate on both IETF and Linux
lists.

-- 
Stephen Hemminger <[email-masked]>

---

From: "Duke, Martin" <[email-masked]>
Date: Wed, 22 Nov 2006 10:56:42 -0800

If IETF would like to coordinate more with Linux, is the right move to
send IETF reps to the next Linux meeting, to tell them what we're
working on and how it might be useful to them?

Martin

> From: Stephen Hemminger [mailto:[email-masked]] 
> Sent: Wednesday, November 22, 2006 10:39 AM
> To: Pekka Savola
> Cc: Lars Eggert; Jeffrey Hsu; [email-masked]; Mark Allman
> Subject: Re: [tcpm] encouraging wider participation
> 
> On Wed, 22 Nov 2006 16:59:19 +0200 (EET) Pekka Savola 
> <[email-masked]> wrote:
> 
> > >> interested in participating in the IETF, so it's 
> difficult to know 
> > >> for sure.
> > >
> > > For Linux kernel developers, which is a community I'm not tuned 
> > > into, can you venture a guess as to why this is (lack of travel 
> > > funds, disinterest with the IETF, etc.)?
> > 
> To help you understand. Linux development is not centralized, 
> there are many community members. Several of the network 
> developers, watch this list (and the e2e list).
> The Linux community works mostly through email with 
> occasional face to face meetings. The meetings are informal 
> and dedicated to discussing issues that are not getting 
> enough traction. There has been an annual meeting of the 
> Linux network developers (http://vger.kernel.org/netconf2006.html)
> as well as discussions at the kernel summit 
> (http://www.usenix.org/events/kernel06/).
>  
> If you look at who contributes to Linux networking; there are 
> contributions from distributions (Redhat, Novell, Ubuntu), 
> non-profits (OSDL), and research (NCSU, Hamilton, etc).  Each 
> contributor has there own needs and spending time on IETF 
> issues is not a priority for most.
> been impractical. If there are issues and agenda items that 
> warrant discussion with IETF, I am sure we can get several 
> people to go to one of the meetings.
> 
> Any practical commercial implementation is a hybrid of 
> specifications from standards organizations, and heuristics 
> based on real world feedback and device quirks.
> 
>  
> There is also, the possibility of more focused groups 
> (security?), or getting cross vendor participation. Several 
> vendors use Linux in their products and participate on both 
> IETF and Linux lists.
> 
> 
> --

---

From: Stephen Hemminger <[email-masked]>
Date: Wed, 22 Nov 2006 10:59:43 -0800

On Tue, 21 Nov 2006 15:41:04 +0000

> http://www.cisco.com/en/US/products/ps6441/products_feature_guide09186a0080683d9d.html

The first report was from users of the debugger in Eclipse, Java was doing several write's
before reading the response.  With ABC, the debugger crawled (even on local host).
Sun did fix the debugger quickly once it was identified.

The later reports were from users who had little or no control
over the application.  Application writers it seems are not network aware 
and have little understanding of underlying stream properties. 
That was when by consensus, Linux turned off ABC in 2.6.18; it is still available, 
just defaults to off.

As has already been mentioned, Linux counts packets not bytes, so the interaction
maybe related to that. Linux implements RFC2861 so it does not increase the congestion
window unless the application is limited by the congestion window.  This makes Linux
mostly immune to the ack-splitting attacks that motivated ABC in the first place.
There have been lots of annoying complexities introduced by the addition of TSO
to the mix, but these are resolved in more recent kernels.

---

From: Fernando Gont <[email-masked]>
Date: Mon, 27 Nov 2006 02:31:52 -0300

>working on and how it might be useful to them?

I'd argue that, if anything, somebody from each major open source OS 
should be sent to the IETF meetings. That way, they could share their 
implementation/real-world experience in the meetings.

Kindest regards,

--
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Fernando Gont <[email-masked]>
Date: Mon, 27 Nov 2006 02:31:52 -0300

Kindest regards,

---

From: Fernando Gont <[email-masked]>
Date: Mon, 27 Nov 2006 09:59:51 -0300

Kindest regards,

---

From: David Borman <[email-masked]>
Date: Mon, 27 Nov 2006 09:55:02 -0600

On Nov 27, 2006, at 6:59 AM, Fernando Gont wrote:

> Folks,
>
> Section 4.2 of tcpsecure describes a corner case that is supposedly  
> un-handled.
>
> My understanding is that the mitigation does *not* harm any feature  
> of TCP. Can anybody think of a reason for which RFC 793 states that  
> a reset should be sent in response to in-window SYNs, rather than  
> responding with an ACK (as suggested by TCP secure)?

A SYN is a request to create a new connection, and the other side  
should not be sending one within the window of an existing  
connection.  So, if the other side does send a new SYN in the window,  
it is an indication that the current connection no longer exists on  
the remote side, hence the current connection is dropped, and an RST  
is generated to indicate to the other side that we've dropped the  
connection.

If we are legitimately in this situation, and we just send a current  
ACK instead of a RST, then as long as our current ACK doesn't ack the  
new SYN, the arrival of our ACK will cause the other side (in SYN- 
SENT state) to generate an RST which will blow away our side of the  
old connection.  Because we're dealing with an in-window SYN, it is  
impossible for us to accidentally ACK this SYN, since we only advance  
the ACK for in-sequence data.

>
> (maybe simply the fact that an out-of-window SYN could be an old- 
> duplicate, while an in-window SYN certainly indicates an abnormal  
> behaviour (i.e., error condition)?)

Yes, an in-window SYN raises the bar on how serious a problem this  
is, where as, as you say, an out-of-window SYN can be just an old  
duplicate, or a new connection request when lost packets haven't  
allowed the current connection to fully close (e.g., CLOSING or TIME- 
WAIT)

			-David Borman

---

From: Ted Faber <[email-masked]>
Date: Mon, 27 Nov 2006 13:27:24 -0800

> implementation/real-world experience in the meetings.

Make it so.

(See the problem?)

---

From: Fernando Gont <[email-masked]>
Date: Thu, 30 Nov 2006 09:10:08 -0300

Dave,

Thanks so much for your answer. Comments inline....

>allowed the current connection to fully close (e.g., CLOSING or TIME- WAIT)

Related to this:
Any idea about the reason for sending an ACK (instead of a RST) when 
SEG.SEQ is in-window but SEG.ACK > SND.NXT?

(just me going through the spec...)

Thanks!

---

From: "Dov Hirshfeld" <[email-masked]>
Date: Thu, 30 Nov 2006 08:18:16 -0800

I think that the reason is that this ACK might be an ACK for old packet
which was delayed in the network.
Now, if the transmitter RTO has expired,  we have to assume that it
moved to slow-start and dragged its SND.NXT back.

Therefore, if   SND.MAX >= SEG.ACK > SND.NXT it is still a legitimate
packets which should not cause a reset.

-Dov hirshfeld