Subject: RE: [Tsvwg] SCTP Timestamps

From: "Duke, Martin" <[email-masked]>
Date: Wed, 16 Jul 2003 14:02:14 -0700

> From: Armando L. Caro Jr. [mailto:[email-masked]]
> 
> Like I said, using HBs is good enough for gathering more RTT 
> measurements
> on alternate paths, but _may_ not be good enough for Eifel 
> detection. I
> think that using HBs would be almost as good as the timestamp 
> chunk if we
> could figure out how to solve the HB-ACKing every packet problem. I do
> agree that an optional timestamp chunk is the best way, because the
> timestamp is guaranteed to be bundled. However, if we can get 
> most of the
> gains with just the HBs, that would be a big plus. It's 
> always nice when
> only one end has to be modified to get the improvment.
> 
> ~armando

We seem to be trying to solve multiple problems with one fix, although their needs diverge somewhat.

1) More frequent RTT measurements (more often than 1/cwnd): obviously does not require every packet to have a HB, as Caitlin points out.

2) Allow RTT measurements on retransmissions: HB only required on retransmissions.

3) Detect spurious timeouts by specifying if original or retransmission was ACKed:  There's no need to be perfect here.  First, on multihomed connections the ACK will normally come along the same path as the DATA, allowing some differentiation.  Second, only initial transmissions require HB; if a HB-ACK comes back for it, you can restore the initial cwnd.  If you don't like a HB-ACK after every packet, there's a tradeoff: more packet density or less assurance of Eifel detection?  It would be an interesting study to see the dynamics of this tradeoff in different loss or delay environments.

I think HBs are superior to a new TS option for several reasons:

A) A cooperative peer will wait long enough to bundle its HB-ACKs with SACK.  An uncooperative peer will flood the network with separate HB-ACKs.  With timestamps, a cooperative peer will behave as with HBs, but an uncooperative peer will do nothing.

B) If the peer turns out to flood the network, the sender can even adjust the frequency of HBs to compensate.

C) 100% reliable detection of spurious timeouts is unnecessary.

D) Changes to the stack (limited to one side) are quite minimal: eliminate the HB timer for timestamp-related heartbeats, store the mapping of TSNs to timestamps, possibly do some dynamic rate control, possibly add 500ms to RTT measurements to take delayed ACKs into account.  And they're entirely borne by the measuring endpoint.  The peer has no choice but to comply if they are conforming to the specification.

That said, messing with SACK to avoid duplication with HB-ACK is a bad idea in my opinion.  It requires changes in the peer implementation and may open up a whole range of unforeseen consequences.

Martin

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Wed, 16 Jul 2003 18:06:43 -0400 (EDT)

On Wed, 16 Jul 2003, Duke, Martin wrote:

> 1) More frequent RTT measurements (more often than 1/cwnd): obviously
> does not require every packet to have a HB, as Caitlin points out.

This is not a problem in my opinion. More frequent RTT measurements has
not been shown to be necessary or even useful.

> 2) Allow RTT measurements on retransmissions: HB only required on

As I mentioned in a reply to Ivan... Unless HBs are bundled with SACKs,
this technique doesn't work without heuristics. To make sure the peer
bundles HBs with SACKs, you might as well use Timestamp chunks.

> 3) Detect spurious timeouts by specifying if original or
> retransmission was ACKed:  There's no need to be perfect here.

Nobody is talking about being perfect, but we would like to have _some_
accuracy.

> First, on multihomed connections the ACK will normally come along the
> same path as the DATA, allowing some differentiation.

No, you don't know that. SACKs are sent to the IP address from which data
was sent, which means that the receiver may use a different return path.
Imagine the following:

       +- A1                     B1 -+
Host A |                             | Host B
       +- A2                     B2 -+

Data may be sent from A1 to B1, but sacks may be sent B2 to A1. That
follows the specs rules, but doesn't have the same return path.

> Second, only initial transmissions require HB; if a HB-ACK comes back
> for it, you can restore the initial cwnd.

Maybe it doesn't matter, but what if the first  retransmission spuriously
times out.

> If you don't like a HB-ACK after every packet, there's a tradeoff:
> more packet density or less assurance of Eifel detection?

Well, sending HBs with every other packet brings down the detection to at
most 50%. Not very good. What is good enough? 90%? So send HBs with 9 of
10 packets. I don't think I want to go down that road.

> A) A cooperative peer will wait long enough to bundle its HB-ACKs with
> SACK. An uncooperative peer will flood the network with separate
> HB-ACKs.  With timestamps, a cooperative peer will behave as with HBs,
> but an uncooperative peer will do nothing.

How can a peer be cooperative without negotiation? And if there is
negotiation, then why not use timestamps? (I assume we all agree that if
negotiation is used, then timestamps are the best way to go.)

> B) If the peer turns out to flood the network, the sender can even
> adjust the frequency of HBs to compensate.

How does the sender know when the peer is flooding the network?

> C) 100% reliable detection of spurious timeouts is unnecessary.

Maybe not, but then what is necessary? The purpose of timestamps is to
move away from heuristics for robustness. But the HB idea seems to add
more heuristics.

> D) Changes to the stack (limited to one side) are quite minimal:
> eliminate the HB timer for timestamp-related heartbeats, store the
> mapping of TSNs to timestamps, possibly do some dynamic rate control,
> possibly add 500ms to RTT measurements to take delayed ACKs into
> account.  And they're entirely borne by the measuring endpoint.  The
> peer has no choice but to comply if they are conforming to the
> specification.

All these changes are fine, but I don't feel that it adds robustness to
Eifel detection. The only way I can see to add robustness is to bundle
timestamp echos with sacks... doesn't matter if they are HB-ACKs or
Timestamp chunks.

> That said, messing with SACK to avoid duplication with HB-ACK is a bad
> idea in my opinion.  It requires changes in the peer implementation
> and may open up a whole range of unforeseen consequences.

Hmmm... I don't follow this point. Are you arguing against your previous
arguments? HB-ACK seems to be morphing into SACK with a timestamp. Is it
not?

~armando

0--                                                  --0
| Armando L. Caro Jr.      |  Protocol Engineering Lab |
| www.cis.udel.edu/~acaro  |    University of Delaware |
0--                                                  --0

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 17 Jul 2003 13:56:25 -0500

I haven't gone thru the all the mails in this thread. My apology if my
comments below overlap with others.

"Armando L. Caro Jr." wrote:
...
> negotiation is used, then timestamps are the best way to go.)

In the current SCTP, normally an endpoint will not receive a HB with an
outbound SACK pending. So, we could simply add the following rule to get
the behavior you want without negotiation:

 When an endpoint receives a HB while having received data pending 
 for ack, the endpoint should delay sending its HB-ACK till the 
 SACK is sent and the HB-ACK should be bundled with SACK.

Would this solve the problem?

> move away from heuristics for robustness. But the HB idea seems to add

I've heard that 50% detection is not good enough and 100% is unnecessary
(and in practice unachievable), and 90% may be about right... This
sounds all too speculative to me. Are there any real world data from the
field or simulation that shows what's the performance gain vs. various
percentage of detection? Or these are just theoretical speculations? 

Also, there seem to be other efforts to address spurious timeouts (e.g.,
D-SACK and two other methods mentioned in this thread). Is there a
comparison study on them? Or, are we going to build them all into TCP
and SCTP? :-)

regards,
-Qiaobing

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 17 Jul 2003 14:05:55 -0600

Qiaobing,

> Would this solve the problem?

Would the above not result in delaying ALL HEARTBEAT-ACK chunks?  When
you delay that HEARTBEAT-ACK chunk, cannot an RTO can occur because the
HEARTBEAT-ACK is lost?  Currently the HEARTBEAT-ACK is sent to the
source address in the HEARTBEAT, and the RFC says to send any bundled
DATA to that address too.  Will this not result in sending DATA to the
wrong address (not the primary) most of the time?  Will not HEARTBEAT-ACKs
sent testing idle or failed destinations also have to wait when data
acknowledgement is pending for heartbeat acknowledgement?  There is no
way for the receive to know the difference between a normal HEARTBEAT
and a timestamp HEARTBEAT, is there?  The difference lies in the opaque
data.

Please consider using a new chunk to avoid breaking the current heartbeat
mechanism.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 17 Jul 2003 19:49:27 -0700

> comparison study on them? Or, are we going to build them all into TCP

I also want to know why can't (since nobody in this thread even
mentioned it) the duplicate TSN be used to detect spurious
retransmission.  It is already specified in the standard and does
not require any new chunk be defined.  The draw back is not 100%
reliable and it is "slower" to detect the problem.  Is there a
fundamental issue with it such that people in this discussion want
a new chunk to be defined?

---

						K. Poon.
						[email-masked]

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 18 Jul 2003 00:03:31 -0400 (EDT)

> a new chunk to be defined?

From what I understand, timestamps can provide a more robust method of
detection.

~armando

0--                                                  --0

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 18 Jul 2003 01:52:32 -0600

Armando,

What is not robust about the receiver indicating that it received
something multiple times?  The receiver is in the best position to
determine when a spurious retransmission has occured at the sender
(because it receives multiple copies).

I think that is more robust than timestamps that may or may not be
included, might get lost, misbundled, associated with the wrong ack,
etc.

With duplicates, the receiver says "I already received this TSN".
I agree with Kacheong here, but I disagree that it is slower.  In
the RFC receiving duplicates is reason for immediate SACK (whereas
timestamps would not).  Therefore, the dup TSNs are, I believe,
faster, in that the duplicate TSN report is immediate under most
situations.  (See 6.2 RFC 2960).

Furthermore, the duplicate TSN mechanism is already exists, and is
already used to determine that a spurious restransmission has
occurred, possibly due to a lost SACK return path (the most obvious
cause of persistent spurious retransmissions).

Can someone back up the statement of timestamps being "more robust"?

The statement doesn't seem to withstand any scrutiny...

--brian

---

From: Sourabh Ladha <[email-masked]>
Date: Fri, 18 Jul 2003 05:00:49 -0400 (EDT)

> a new chunk to be defined?

From some of the emails I have seen in this thread, it seems folks have
the wrong impression that timestamps are being proposed for "Eifel
purpose".

I would like to clarify and REPEAT myself again that we were NOT
discussing timestamps for Eifel purposes. Eifel happens to be
ONE of the possible mechanisms which can use timestamps.

-Sourabh

---

From: Stephan Baucke <[email-masked]>
Date: Fri, 18 Jul 2003 11:01:58 +0200

> a new chunk to be defined?

You have already mentioned one of the reasons: Detection of spurious 
retransmissions using DSACK only works "after the fact", whereas 
something like Eifel detects it early enough to avoid a go-back-n-like 
behavior after a spurious timeout (also see RFC 3522). Also, DSACKs do 
not address the other issues mentioned in this thread, such as the "path 
ambiguity" and RTTM. Note that DSACKs are still useful and may be 
combined with Eifel; I think there is a compelling case for introducing 
optional timestamps even if DSACKs are already utilized.

Stephan.

---

From: Stephan Baucke <[email-masked]>
Date: Fri, 18 Jul 2003 11:21:26 +0200

> included, might get lost, misbundled, associated with the wrong ack,

Well, this is what I was hoping to get out of this disucssion: A good 
way to do timestamps without those problems :-)

> situations.  (See 6.2 RFC 2960).

The problem is that with DSACKs the sender is usually only notified 
after he has already spuriously retransmitted all packets he erroneously 
considered lost. With timestamps, the sender can tell that a spurious 
retransmission happened from the first new ACK. This way unnecessary 
go-back-n retransmissions can be avoided in many cases. Please see RFC 
3522 for more.

> Can someone back up the statement of timestamps being "more robust"?

For one, the DSACK method is more sensitive to ACK losses.

Stephan.

---

From: Sourabh Ladha <[email-masked]>
Date: Fri, 18 Jul 2003 05:26:04 -0400 (EDT)

> cause of persistent spurious retransmissions).

hmm..What about packet reordering, packet duplication, delay spikes..we
seem to be missing some basic causes of spurious retransmissions.

> Can someone back up the statement of timestamps being "more robust"?

First, we seem to be using the terms Eifel and timestamps in the same
context. This is a fundamental flaw. Are you asking the motivation behind
timestamps or are you asking how Eifel using timestamps is more robust
than the dup-tsn detection.

I would guess the second..from the top of my head, I can think of a
few advantages of Eifel over dup-tsn detection.

* the dup-tsn notification comes much late in the game. You cannot avoid a
GoBackN (which is one of the motivations anyway for having a spurious
retransmission detection scheme) if you had a spurious timeout. (I believe
RFC3522 will have more details on these). Eifel will be able to detect
possibly on the arrival of the first acceptable ACK if the retransmission
was spurious.

* the dup-tsn detection is sensitive to ACK loss: if you lost the ACK
carrying the duptsn you possibly fail the detection.

* the duptsn notification is less robust to packet duplication (in
particular for SCTP). I remember Reiner and I had noted this in some of
the previous emails on a different thread: If the SCTP association is
multihomed and the alternate path is duplicating packets then with the
dup-tsn detection scheme coupled with a response algorithm may end up
making an SCTP non-responsive to congestion.

-Sourabh

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 18 Jul 2003 04:39:32 -0600

Sourabh,

> seem to be missing some basic causes of spurious retransmissions.

I didn't mean that it couldn't be used for those, it just isn't at the
moment.  2960 mentions the more obvious, most injurious to SCTP (blindly
sending SACKs to a dead destination address that never receives DATA and
never suffers a timeout except at slower HEARTBEAT intervals.)

> possibly on the arrival of the first acceptable ACK if the retransmission

Ok. I see that.  But I would say that it is "possible" that timestamps will
detect faster, and not necessarily always or even often and certainly not
obviously the case.

> carrying the duptsn you possibly fail the detection.

Lost ACKs can cause Eifel to fail detection on timestamps too.

> making an SCTP non-responsive to congestion.

Duplicating network paths are easy to detect: a TSN cannot have more
dup-TSns that it was retransmitted.

--brian

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 18 Jul 2003 19:47:59 -0700

I aggregate several replies into one mail to save bandwidth...

> ONE of the possible mechanisms which can use timestamps.

I guess what people have said so far is that the other purposes
are either not very beneficial or do not necessarily require the
use of timestamp.  That's why I try to concentrate on the one
issue which people think "definitely" require the use of timestamp.

Stephan Baucke wrote:

 > behavior after a spurious timeout (also see RFC 3522).

How about Pasi's F-RTO?  I have read Sourabh's questions about
F-RTO.  But I think F-RTO can be modified, possible making use
of duplicate TSN, for SCTP to address those questions.

 > Also, DSACKs do

I guess many people have pointed out that RTTM is not very
crucial and there are implementation techniques which can make
more RTT measurements without using timestamp.

For the issue of path ambiguity, I guess we need to first
understand why it happens.  And can it be avoided?  One case
mentioned is that a retransmission takes place in an alternate
path causing problem in cwnd growth.  We should first ask if
retransmitting using an alternate path is always a good thing.
There is a reason why RFC 2960 does not require (MUST) an
implementation to always retransmit using alternate path.  I
guess we should first decide when an alternate path be used for
retransmission to minimize the path ambiguity issue.

Another path ambiguity issue mentioned is changeover by the app.
A "simple" way to avoid that is for the implementation to withhold
sending packets before all outstanding packets to the original
primary destination address have been ack'ed.  This has the side
effect of discouraging an app to change often.  Granted, this "simple"
solution prevents a lot of "experimentation" on, for example,
load balancing and has obvious performance implication.  But I think
at this point, there is no solid proposal in making using of alternate 
paths other than retransmission.  So I believe it is OK for now.

I guess we should try to avoid the problems first before trying
to find a solution, timestamp in this case, to the problems.  And
it seems to me that we can actually minimize the chances of the
problems mentioned so far.  I don't see a compelling reason to
introduce a new chunk for timestamp.

---

						K. Poon.
						[email-masked]