Subject: [Tsvwg] SCTP Timestamps

From: Stephan Baucke <[email-masked]>
Date: Tue, 15 Jul 2003 11:29:50 +0200

Hi all,

we have been thinking about timestamps a la RFC 1323 for SCTP. We see a 
number of compelling uses:

- Allowing for better and more frequent RTT measurements

- Resolving the "path ambiguity" in a multihomed association: If an ACK 
arrives for a chunk that has been retransmitted one or more times on 
alternating paths, it is not possible to safely determine which 
destination the ACK should be credited to (this is briefly mentioned in 
section 8.2 of RFC 2960). This may result in resetting the wrong error 
counter (delaying the detection of an inactive path), or in unfair 
behavior if the congestion window of the wrong path is adapted as a 
result of the ACK.

- Resolving the retransmission ambiguity, i.e. enabling the sender to 
determine whether an ACK refers to the original transmission or a 
retransmission of a chunk. This is a necessary precondition for 
implementing the Eifel algorithm for SCTP (which we are currently 
looking at).

It seems to me there is a pretty strong case for having timestamps in 
SCTP. Unfortunately there are no provisions for a timestamp option 
similar to TCP's in the SCTP common header. There has been a proposal 
(first brought up by Armando, I believe) to introduce a timestamp chunk. 
This is fine for doing RTT measurements, but for the latter two cases 
mentioned above, it would require that the timestamp chunk is always 
bundled with the data chunk it is referring to.

My question is how such a "mandatory bundling" is viewed in the 
community. In some ways it seems to be a break with existing SCTP 
"philosophy", since so far all chunk types are "self contained", and 
there is no known case of mandatory bundling. Is it desirable to use new 
chunk types as a replacement for TCP-like per-packet options? There is 
also the very practical problem of fragmentation. If we want to avoid 
possible refragmentation of data chunks, we would have to at least 
reserve room for the timestamp chunk in each packet (this could be 
negotiated at connection setup). What happens if we want to introduce 
further per-packet information in the future?

Stephan.

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 15 Jul 2003 12:23:32 +0200

Hi Stephan,

I'm not sure if you want per packet info or per DATA chunk info?

In case of per packet info you could define a new chunk and have it
in the packets.

In case of per DATA chunk info there is the possibility of defining a
new EXT_DATA chunk which is like a DATA chunk but also holds the 
additional
information.

Both ways require an extension of the protocol.

Best regards
Michael

On Tuesday, Jul 15, 2003, at 11:29 Europe/Berlin, Stephan Baucke wrote:

> Hi all,
>
> we have been thinking about timestamps a la RFC 1323 for SCTP. We see 
> a number of compelling uses:
>
> - Resolving the "path ambiguity" in a multihomed association: If an 
> ACK arrives for a chunk that has been retransmitted one or more times 
> on alternating paths, it is not possible to safely determine which 
> destination the ACK should be credited to (this is briefly mentioned 
> in section 8.2 of RFC 2960). This may result in resetting the wrong 
> error counter (delaying the detection of an inactive path), or in 
> unfair behavior if the congestion window of the wrong path is adapted 
> as a result of the ACK.
>
> (first brought up by Armando, I believe) to introduce a timestamp 
> chunk. This is fine for doing RTT measurements, but for the latter two 
> cases mentioned above, it would require that the timestamp chunk is 
> always bundled with the data chunk it is referring to.
>
> there is no known case of mandatory bundling. Is it desirable to use 
> new chunk types as a replacement for TCP-like per-packet options? 
> There is also the very practical problem of fragmentation. If we want 
> to avoid possible refragmentation of data chunks, we would have to at 
> least reserve room for the timestamp chunk in each packet (this could 
> be negotiated at connection setup). What happens if we want to 
> introduce further per-packet information in the future?
>
> Stephan.
>
>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 15 Jul 2003 05:37:50 -0600

Stephan,

"Better and more frequent RTT measurements"?

Better how?  More frequent?  Not significantly so, except
perhaps during special (read: rare) anomalies I would think.

Clear the wrong error counter?  RFC 2960 says:

  If this ambiguity is undesirable, the transmitter may choose
  not to clear the error counter if the last chunk was a
  retransmission.

Where is the congestion window of the wrong path adapted as a
result of an ACK?

I think this could only be as a result of some really strange
situation.  Besides, not crediting a destination cwnd for
retransmitted packets would be more conservative than
crediting, and is therefore, permitted.  The same applies, if
you don't like possibly crediting the wrong destination for
the retransmitted packet, don't credit any destination for a
retransmitted packet.  It shouldn't really matter, some
destination got dinged really bad for slipping a RTO in the
first place.  This exponential back off far overwhelms any
meager linear growth from a few retransmitted packets.

"Unfair behavior"?

Fairness between paths is not of consideration.  Congestion
fairness between TCP connections and SCTP associations is.
Only the primary gets credited when all is well and the other
paths suck slow start, even if heartbeats show superb RTT.
How fair is that?

What do you mean "refragmentation [sic] of data chunks"?

Once fragmentation of data is made into data chunks, they
cannot be re-fragmented.  See RFC 2960 Section 6.9:

  Note: Once a message is fragmented it cannot be re-fragmented.
  Instead if the PMTU has been reached, then IP fragmentation must be
  used. ...

Perhaps Eifel an TSN wrap on Googleplex-bytes-per-second paths
might be reasons for timestamps, but the above don't sound
like justifications.

If you want to take a really accurate RTT measurement in SCTP,
send a heartbeat and place whatever timestamps and identifiers
you want in the heartbeat data.

I find sending a heartbeat just after Path.Max.Retrans has
been exceeded to be quite useful for quickly restoring paths
to which you were simply too cruel, and particularly when
Path.Max.Retrans is set to 0.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Stephan Baucke <[email-masked]>
Date: Tue, 15 Jul 2003 15:12:20 +0200

> perhaps during special (read: rare) anomalies I would think.

Timestamps would enable us to do RTTM (RTT measurement on every packet), 
as described in RFC 1323. Together with an appropriate timer 
calculation, this has the potential for an RTO that more accurately 
follows the RTT. Also, since timestamps resolve the retransmission 
ambiguity, we don't have to use Karn's algorithm. Karn's algorithm 
together with SCTP's retransmission policy has been shown to cause an 
overly conservative RTO in certain multihomed scenarios (see 
http://www.cis.udel.edu/~acaro/papers/2003.milcom.acaro.pdf)

>   not to clear the error counter if the last chunk was a

This helps, but can result in a premature failure detection on the 
secondary path (which is usually only used for retransmissions).

> Where is the congestion window of the wrong path adapted as a

If an ACK is credited to a wrong path, the CWND on that path will be 
falsely increased where it should be decreased.

> situation.

Consider the following example:
1) Chunk A is sent over the primary path
2) A spurious timeout occurs on the primary path
3) Chunk A is retransmitted over the secondary path
4) The packet is lost on the secondary path
5) Finally, the SACK arrives for the original transmission over the 
pirmary path

According to RFC 2960, the SACK will be credited to the secondary path, 
since that's where the last retransmission took place. So, we will 
increase its congestion window where we should really halve it or set it 
to <=2*MTU, since a packet loss occured on that path.

 >  Besides, not crediting a destination cwnd for
> retransmitted packet.

We would still fail to reduce the cwnd on one path.

> fairness between TCP connections and SCTP associations is.

Of course I was referring to the latter case.

>   Instead if the PMTU has been reached, then IP fragmentation must be

Yes. That's why we would run into problems if we were to use new chunk 
types as per-packet options without reserving space for them in every 
packet, regardless whether the option is actually used in all packets or 
only a part of them.

> you want in the heartbeat data.

Heartbeats might be too infrequent for some scenarios.

Stephan.

---

From: [email-masked]
Date: Tue, 15 Jul 2003 09:16:31 -0400 (EDT)

Brian,

> perhaps during special (read: rare) anomalies I would think.

I hope that you know the answer to "More Frequent?". As far as better is
concerned, I think this has been argued before on the list as well as in
research papers. You can probably look for RTTM in RFC1323 and Phil Karn's
1987 paper which brings out the Retransmission-Ambuiguity problem and on
the other side Allman's 1999 SIGCOMM paper. I wonder if you are also
questioning timestamps in TCP ?

>   If this ambiguity is undesirable, the transmitter may choose
            ^^^^^^^^^
>   not to clear the error counter if the last chunk was a

I thought the point was that with timestamps you can avoid this ambuiguity.

> paths suck slow start, even if heartbeats show superb RTT.

I do not think Stephan's meant or even mentioned "fairness between paths".
I believe he meant that the "Retransmission-Ambuiguity" may cause the
wrong destination to be credited for cwnd. A simplistic situation is: If
you have a timeout and retransmit on the alternate, then you do not know
on the reciept of an ACK if it corresponds to the retransmission or the
original (which may have got delayed). Try adapting this to cwnd crediting
as well and you will see what Stephan referred to. I believe you can find
some action related to this at 
http://www.cis.udel.edu/~iyengar/publications/
(papers: [1] Making SCTP More Robust to Changeover [2] SCTP Congestion
Window Overgrowth During Changeover)

>   Instead if the PMTU has been reached, then IP fragmentation must be

I agree. Maybe the issue was that for mandatory bundling of timestamp
chunks one will have to take into consideration the association-PMTU as
well as the timestamp-chunk size before fragmenting.

 -Sourabh

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Tue, 15 Jul 2003 12:52:55 -0400 (EDT)

On Tue, 15 Jul 2003, Stephan Baucke wrote:

> there is no known case of mandatory bundling.

First, I don't see a need to make it mandatory. The sender can choose to
bundle timestamps with all packets containing data and/or sack chunks. If
the sender chooses not to include timestamps, then the sender must follow
the conservative behavior anytime an ambiguous scenario arises.

Second, I think the ECNE and CWR chunks are examples of chunk types that,
if used, need to be in every packet. The sender simply needs to reserve
space for these chunks.

~armando

0--                                                  --0
| Armando L. Caro Jr.      |  Protocol Engineering Lab |
| www.cis.udel.edu/~acaro  |    University of Delaware |
0--                                                  --0

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Tue, 15 Jul 2003 13:30:46 -0400 (EDT)

On Tue, 15 Jul 2003, Stephan Baucke wrote:

> follows the RTT.

"Together with an appropriate timer calculation" is an important clause.
As mentioned in RFC2960, Mark Allman and Vern Paxson show that measuring
RTTs more frequently than once per RTT does not yield any significant
benefit. Also, if measurements are made more often, RTO.Alpha and RTO.Beta
should be adjusted, but it is still an open question how to adjust the RTO
calculation.

Allman, M. and Paxson, V., "On Estimating End-to-End Network Path
Properties", Proc. SIGCOMM'99, 1999.

But with multihoming, timestamps can be useful for alternate paths as you
allude to below.

> Also, since timestamps resolve the retransmission

Also, see
http://www.cis.udel.edu/~acaro/papers/2003.icon.acaro.pdf

or the full tech report
http://www.cis.udel.edu/~acaro/papers/2003.TR2003-05.acaro.pdf

> secondary path (which is usually only used for retransmissions).

Yes. Also the point is, if we can eliminate the ambiguity to improve
performance, then we should. Should we not??

> falsely increased where it should be decreased.

Timestamps may help alleviate this problem, but I think the SFR-CACC
algorithm is a better solution. The SFR-CACC algorithm requires
changes only at the sender side without extra bits on the wire.

See:
http://www.cis.udel.edu/~acaro/papers/2003.spects.iyengar.pdf
http://www.cis.udel.edu/~acaro/papers/2002.sci.iyengar.pdf

Jana Iyengar has also made more progress with the algorithm to prevent the
same issue from occuring with cycling changeovers. Plus, the improved
algorithm is actually simpler. Unfortunately, he doesn't have any
documents yet which explain the modified algorithm.

> to <=2*MTU, since a packet loss occured on that path.

Your scenario is very similar to the example scenario which Jana uses,
except his scenario involves a changeover which causes spurious fast
retransmits.

> We would still fail to reduce the cwnd on one path.

Yes, that's right. The CACC problem causes spurious retransmits on the
alternate/new path and incorrect cwnd growth on the alternate/new path
...even if the retransmissions on the alternate/new path are lost!

> Of course I was referring to the latter case.

Obviously, incorrectly increasing the cwnd as described above is not fair
to TCP, TCP-friendly, Internet-friendly, or whatever you want to call it.

> packet, regardless whether the option is actually used in all packets or

I am not sure I understand the problem here. I know there is some
inefficiency with reserving space for control chunks, but since timestamps
will be essentially in every packet... what's the problem?

> Heartbeats might be too infrequent for some scenarios.

Yes. You _could_ decrease the HB.interval, but then the sender would
introduce more "useless" packets on the wire... which I don't particularly
like.

~armando

0--                                                  --0

---

From: Sourabh Ladha <[email-masked]>
Date: Tue, 15 Jul 2003 13:58:24 -0400 (EDT)

> the conservative behavior anytime an ambiguous scenario arises.

Of course if adopted, the timestamp chunk parameter will have to be
defined to be negotiated in the INIT/INIT-ACK. By mandatory the reference
was that if both endpoints agree on timestamp then the timestamp chunk
MUST be bundled with all packets containing DATA/SACK chunk(s). Your
clarification below addresses that concern, I think.

-Sourabh

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Tue, 15 Jul 2003 15:22:27 -0400 (EDT)

In a private discussion with Sourabh, he pointed out to me that the CACC
algorithm will not solve the problem presented below by Stephan. Although
the side effects of the scenario are similar, the CACC relies on Gap Ack
information to solve the problem. My mistake... So I take back what I say,
and agree that timestamps are a good solution.

Sorry for the confusion.

~armando

0--                                                  --0

---

From: Qiaobing Xie <[email-masked]>
Date: Tue, 15 Jul 2003 17:42:42 -0500

Hi,

Pls see me comments below...

> clarification below addresses that concern, I think.

Whenever I see a "MUST" used (or being suggested to be used) in the
spec, I'd like to understand what would be the error handling strategy
if this "MUST" is violated. Since you are suggesting a "MUST" clause
here, would you immediately severe the communication with a peer that
had agreed to use timestamp feature but you subsequently receive a
packet from it without the timestamp chunk bundled? If your answer is
"yes", then I'd guess that you somehow must view this missing timestamp
as an unrecoverable protocol error which can do potential harm to your
endpoint or the network if not stopping the communication right away.
But this is not obvious to me, so please explain. If your answer is
"no", then I'd question why you want to use a "MUST" here.

Another question: since the timestamp information (however it would be
defined for your purposes) seems to only have local significance to the
data sender. If so, this info can be kept opaque to the peer/receiver.
Since the SCTP heartbeat (it has been clarified that the HB can be sent
at different frequency or even "manually" triggered by the ULP whenever
one is needed) has the capability to carry and return an opaque data
blob, I don't see why you need to define a new timestamp chunk? 

> > if used, need to be in every packet. The sender simply needs to reserve

To reserve room for control chunks is part of the SCTP fragmentation
code present today in *every* implementation that supports
fragmentation. So this should not be a problem and should add no
complexity.

-Qiaobing

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 15 Jul 2003 18:37:08 -0500

On Tuesday, July 15, 2003, [email-masked] wrote:

>community. In some ways it seems to be a break with existing
>SCTP "philosophy", since so far all chunk types are "self
>contained", and there is no known case of mandatory bundling.
>Is it desirable to use new chunk types as a replacement for
>TCP-like per-packet options? There is also the very practical
>problem of fragmentation. If we want to avoid possible
>refragmentation of data chunks, we would have to at least
>reserve room for the timestamp chunk in each packet (this could

With RDMAP/DDP over SCTP requiring that such a chunk be included
in *every* packet would guarantee that it was included in none.

The effective PMTU is reported to the DDP layer, which is then
responsible for segmenting DDP messages to conform with the
PMTU. It is unlikely that anyone would configure their stack
to accept a reduction in the data carrying PMTU for a protocol
that involves bulk data transfer.

But there would be frequent times when such a chunk could 
piggy-back "for free" in an SCTP packet that was ready to
ship but not full. While not as useful as time information
included in every packet, it is more likely to be provided.

I would recomend allowing the sending stack to apply its
own criteria as to when to slip one in, and then simply
have the receiving side determine whatever it can from
the data actually provided. If complete info has to be provided,
fine, you haven't lost anything. If partial info can be useful,
allowing the sender to include it on a "standby" basis could
be a better solution.

---

From: Sourabh Ladha <[email-masked]>
Date: Wed, 16 Jul 2003 05:24:29 -0400 (EDT)

> "no", then I'd question why you want to use a "MUST" here.

Hi Qiaobing,

My answer would be "no"..I did not mean the spec MUST while writing the
email :(. However I am glad that you brought it up, since it would be
interesting, and maybe relevant for timestamps, to know what you all had
thought of ECNE and CWR bundling rules.
(The spec does not mention these 2 chunks in detail).

> blob, I don't see why you need to define a new timestamp chunk?

I remember earlier having thought of using HB's for timestamp purposes.
The problem is that while timestamp information is useful for the sender, it
does requires intelligence at the reciever. I.E. The reciever will have to
be aware  which timestamp to echo (e.x. In case a duplicate chunk, in case of
delayed SACKS, etc [please see rfc1323). And so the need for defining a
timestamp chunk.

-Sourabh

---

From: Stephan Baucke <[email-masked]>
Date: Wed, 16 Jul 2003 12:04:46 +0200

> if used, need to be in every packet. The sender simply needs to reserve

Good point, I didn't think of ECNE/CWR. Shouldn't this be in the 
implementer's guide? I mean, if an option is negotiated at startup time 
that may later take away room in packets containing data chunks, this 
must be taken into account when calculating the available room for data 
chunks (and when reporting the MULPDU to protocols such as DDP?).

---

From: Stephan Baucke <[email-masked]>
Date: Wed, 16 Jul 2003 12:11:34 +0200

> "Together with an appropriate timer calculation" is an important clause.

Yes :-)

> will be essentially in every packet... what's the problem?

Hm, ECNE/CWR chunks will not be in every packet. Has the issue of 
reserving room for those been addressed? I'm also thinking about more 
per-packet options that might be introduced in the future; should each 
of them be allowed to take away some more room?

---

From: Stephan Baucke <[email-masked]>
Date: Wed, 16 Jul 2003 12:16:48 +0200

> that involves bulk data transfer.

Caitlin,

I'm not that familiar with RDDP, but my understanding is that the LLP 
(in this case SCTP) reports the MULPDU to DDP. Besides a small loss in 
efficiency due to increased overhead, is there any problem with taking 
control chunks into account when calculating the MULPDU? How is this 
handled if TCP (where options also take away from the MULPDU) is used as 
the LLP?

Stephan.

---

From: Stephan Baucke <[email-masked]>
Date: Wed, 16 Jul 2003 13:52:20 +0200

> chunks (and when reporting the MULPDU to protocols such as DDP?).

As Sourabh just pointed out to me, the ECNE/CWR chunks in fact do *not* 
have to be bundled with any other chunk, since they contain a TSN that 
acts as a pointer to the packet where the CE bit was originally set. So 
this is not a precedent for "mandatory bundling" either.

Another idea: Couldn't we use the same method for timestamps, i.e. the 
timestamp chunk would contain the TSN of a chunk in the packet that this 
timestamp is referring to?

Stephan.

---

From: Pasi Sarolahti <[email-masked]>
Date: Wed, 16 Jul 2003 16:51:25 +0300 (EEST)

Hi,

I believe that timestamps are useful as an option for SCTP, but there 
might be cases where the user would not like to use them due to additional 
overhead (like with some telnet-kind of application protocol with small 
payloads). Just a quick note below inline, FWIW

> calculation, this has the potential for an RTO that more accurately 

Timestamps are not a requirement for this. If the sender stores the
transmission times of segments, it can measure RTT for every
non-retransmitted segment. This is a small overhead to the sender's
TCP/SCTP control data, but few bytes per segment should not be a problem
for modern computers. The retransmission ambiguity exists, but unless
retransmissions are very frequent, I wouldn't think it's too bad a
problem.

> http://www.cis.udel.edu/~acaro/papers/2003.milcom.acaro.pdf)

[Sorry, I can't read the paper right now, but will do it as soon as 
possible]

Was overly conservative RTO due to Karn's algorithm or due to only one RTT
sample per round-trip?

Thanks,
- Pasi

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Wed, 16 Jul 2003 11:40:08 -0400 (EDT)

> delayed SACKS, etc [please see rfc1323). And so the need for defining a

Two more points why HBs may not work well:

1. The sender would send HBs in almost every packet, which could make the
failure detection algorithm trickier.

2. If data transmission is bi-directional, then a packet which has SACK
and DATA chunks will need two HBs. On the other hand, a single Timestamp
chunk is needed since it has a "timestamp" and "timestamp echo" field.

~armando

0--                                                  --0

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Wed, 16 Jul 2003 11:42:18 -0400 (EDT)

> of them be allowed to take away some more room?

That's a tricky situation. It is really an implementation issue, but it
probably makes sense to reserve room for chunks that must be bundled.
Others which are self contained don't have to be bundled if they don't
fit.

~armando

0--                                                  --0

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Wed, 16 Jul 2003 11:50:36 -0400 (EDT)

> timestamp is referring to?

You could, but that would make the chunk significantly larger. The TSN
info would be needed for the timestamp and timestamp echo. Also, the TSN
info would have to be a range of timestamps. So the resulting timestamp
chunk would be something like:

 ------------------
| TIMESTAMP Header |
 ------------------
|    timestamp     |
 ------------------
|    Start TSN     |
 ------------------
|    End TSN       |
 ------------------
|  timestamp echo  |
 ------------------
|    Start TSN     |
 ------------------
|    End TSN       |
 ------------------

That's a total of 28 bytes! Or, you could break them up and have a
timestamp chunk and timestamp echo chunk. But then 4 bytes are wasted in
packets that bundle both, which may happen often with bi-directional data.

~armando

0--                                                  --0

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Wed, 16 Jul 2003 11:53:26 -0400 (EDT)

> Was overly conservative RTO due to Karn's algorithm or due to only one RTT

Karn's algorithm (specifically in multihomed SCTP associations).

~armando

0--                                                  --0

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 16 Jul 2003 11:18:59 -0500

>handled if TCP (where options also take away from the MULPDU) is used as 

Yes, the DDP layer can account for this overhead. But it would be to
the DDP applications advantage to refuse the option rather than to
reduce its effective bandwidth. DDP applications are generally
concerned with bulk data transfer. I believe that SCTP is far superior
to TCP for DDP. But there is already an issue that at first glance
the overhead of SCTP appears larger (it isn't when you understand that
it can be shared across multiple streams). But I would not want to make
those statistics appear worse by adding "just one more option".
I would certainly *not* enable such a chunk by default (because I
wouldn't want it to be part of my benchmark statistics).

At the tail end of a given exchange, however, there will always be
a message that needs to be sent without delay. For DDP it is the
tail segment of an Untagged Message. It would be highly probable
that those packets would have room for a "standby" Data Chunk with
the additional data you desired.

I suspect this would be true for *many* applications, and you may
find stacks willing to support opportunistic low-priority sending
of Data Chunks. The only thing unique is that the Data Chunk is
being filled in at placement time, not when it is placed in the queue.

My suggestion is to allow that type of strategy to supply the data
you want on an opportunistic basis, because there will be data
transfer applications that will resist yielding any of their
bandwidth when asked to do so on *every* packet. And it would
be very easy for them to insist on a stack that at least pretended
not to know what this option was, especially since they would have
no incentive to upgrade to such an option.

---

From: Sourabh Ladha <[email-masked]>
Date: Wed, 16 Jul 2003 14:18:03 -0400 (EDT)

Caitlin,

> no incentive to upgrade to such an option.

While what you say regarding overheads and bulk data transfer is indeed
true, much of the purpose of timestamps is lost if we can only send them
on an *oppurtunistic basis* (see some of the motivations listed in the
first email by Stephan). Personally, I think the best which can be done is
to disable timestamps if an app/stack feels that it can achieve better
throughput by doing so.

 -Sourabh

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 16 Jul 2003 15:13:36 -0500

On Wednesday, July 16, 2003, [email-masked] wrote:

>While what you say regarding overheads and bulk data transfer
>is indeed true, much of the purpose of timestamps is lost if we
>can only send them on an *oppurtunistic basis* (see some of the
>motivations listed in the first email by Stephan). Personally,
>I think the best which can be done is to disable timestamps if
>an app/stack feels that it can achieve better throughput by
>doing so.
>

There are some assumptions being made here that I am not following.

If I am following the thread, the purpose of timestamping is to
allow frequent ongoing measurement of the RTT along each path.
Further, knowing the current RTT for each path, it will be
possible to disambiguate certain Acks. This leads to better
decisions on when to shift paths and/or do a fast retransmit.

Is that correct?

If so, why is timestamp data required in every packet?

MPEG transport maintains a distributed clock over unreliable
jittered networks even in the face of frequent packet drops.
I am not following why a timestamp must *always* be present
to allow measurements to be made accurately.

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Wed, 16 Jul 2003 16:35:28 -0400 (EDT)

Caitlin,

There are actually 2 motivations for timestamps.

The original motivation was to be get rid of Karn's algorithm and be to
use retransmissions for RTT estimation. This purpose does not need
timestamps in every single packet, but the more the better.

The second motivation was to use timestamps for Eifel detection. A
spurious timeout retransmit can be detected if the retransmitted TSN gets
acked with a timestamp earlier than the timestamp bundled with the
retransmission. This purpose does require a timestamp in each packet (I
think).

~armando

0--                                                  --0

---

From: Sourabh Ladha <[email-masked]>
Date: Wed, 16 Jul 2003 16:57:19 -0400 (EDT)

> retransmission. This purpose does require a timestamp in each packet (I

Another motivation for having timestamp per packet or maybe just for
retransmissions (either way may require you to have room reserved for
timestamp chunk in every packet) was to allow correct crediting of cwnd
(i.e. in particular avoid congestion window overgrowth) by eliminating
"retransmission ambuiguity".

-Sourabh

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 16 Jul 2003 17:09:50 -0500

On Wednesday, July 16, 2003, [email-masked] wrote:

>The second motivation was to use timestamps for Eifel
>detection. A spurious timeout retransmit can be detected if the
>retransmitted TSN gets acked with a timestamp earlier than the
>timestamp bundled with the retransmission. This purpose does
>require a timestamp in each packet (I think).

It is still possible to infer a possible timestamp range for
when a packet was sent by interpolation from earlier and later
packets, particularly if the reception time of each packet is
captured for use by SCTP.

This is especially true for sustained transmits of large
messages that span multiple SCTP packets (which is precisely
when it would be valuable to omit the timing data). If we
assume that for the duration of a "sustained transmit" that
the sender will be paced by the congestion and buffer windows
then the timing of each packet becomes very predictable on
a statistical basis.

These aren't necessarily easy algorithms to work out, but they
can be *implemented* cheaply. Video on Demand projects had very
simple firmware algorithms to recover an MPEG Transport clock
even when transmitted over a heavily jittered ATM network. The
recovery algorithm could even be simple enough to be done in
hardware or 8-bit processors.

I am not certain that this is worthwhile, but the assumption
a packet without a timestamp was a complete mystery kind of
leapt out at me. MPEG Transport systems survive quite well
on periodic timestamps, and they have much more precise
timing requirements. Recovering an imputed clock with
sufficient resolution to differentiate between two possible
acks of the same TSN should be quite doable.

---

From: Pasi Sarolahti <[email-masked]>
Date: Thu, 17 Jul 2003 13:52:28 +0300 (EEST)

Hi,

> retransmission. This purpose does require a timestamp in each packet (I

Aside from this timestamp discussion, but since spurious RTOs were 
mentioned: We have proposed a scheme for detecting spurious RTOs without 
timestamps in draft-sarolahti-tsvwg-tcp-frto-04.txt. I solicited for 
comments on it in early June, but unfortunately got no answer. It would be 
nice to hear whether the SCTP community would find that useful.

The idea in short is that after an RTO the sender would try to send some
new data and check if there are acks of non-retransmitted segments
arriving after that, in which case the RTO is considered to be spurious.
This should work with SCTP as well as with TCP as far as I can see. I
believe Markku Kojo will have a short presentation about the draft in the
tsvwg meeting.

Thanks,
- Pasi

---

From: [email-masked]
Date: Thu, 17 Jul 2003 08:27:55 -0400 (EDT)

Pasi,

I had read the FRTO document some time back. On the SCTP front I was
wondering if the following are applicable:

* Multihoming: If the delay spike on the primary path is greater than (1
RTO(primary) + 2 RTT(alternate)), I would believe that FRTO will not be
able to detect a timeout spurious. (complying with the current spec that
retransmissions SHOULD go to alternate destination)

* Chunk Bundling: FRTO assumes that on a timeout the sender will
retransmit only one segment("I think chunk will be appropriate in SCTP
terminology"). This may not be true in SCTP. If the association is
multihomed and the PMTU's on the primary and alternate path differ by a
factor of two or more then, then I think the sender may be able to
retransmit more than one unacknowledged chunk on a timeout (bundled in a
single packet). This can also be true if the app is writing small chunks
such that on a timeout more than one of such chunks can be bundled in a
single packet and retransmitted. I remember having noticed a couple of
side effects of this which could lead FRTO to declare the timeout spurious
while it was not.

Comments?
(Others shoot in if I missed something)

A question I have is:
Section 3- step 3(b): When you refer to an ACK which acknowledges data
(above spurious threshhold or previously unacknowledged), do you refer to
acknowledging {via cum-ack point} or {via SACK blocks} or {either of these
two}?

-Sourabh

> -----Original Message-----
> From: Pasi Sarolahti [mailto:[email-masked]]
> Sent: Thursday, July 17, 2003 12:52 PM
> To: Armando L. Caro Jr.
> Cc: [email-masked]
> Subject: Re: [Tsvwg] SCTP Timestamps
>
>
> Hi,
>

---

From: Stephan Baucke <[email-masked]>
Date: Thu, 17 Jul 2003 14:31:07 +0200

> chunk would be something like:

You could probably get away with just one TSN, analog to the ECNW/CWR 
chunks. But I admit that this solution isn't ideal anyway for Eifel or 
the disambiguating of retransmissions; e.g. it gets difficult if the 
timestamp chunk is sent separately and gets lost or reordered in 
relation to the corresponding data packet.

Stephan.

---

From: Stephan Baucke <[email-masked]>
Date: Thu, 17 Jul 2003 14:41:10 +0200

> wouldn't want it to be part of my benchmark statistics).

I see your point. I think the best solution would be to simply let 
applications deactivate timestamps if desired. However, while the 
overhead may be significant for high performance bulk transfer, there 
are many other applications (e.g. in the signaling area) where it 
doesn't matter all that much.

> no incentive to upgrade to such an option.

Only sending the timestamp on an opportunistic basis would degrade the 
effectiveness of Eifel or algorithms to disambiguate the "path 
ambiguity". However, it may be good enough for those purposes to send 
timestamps with every packet that contains retransmitted chunks.

Stephan.

---

From: Stephan Baucke <[email-masked]>
Date: Thu, 17 Jul 2003 14:58:36 +0200

> captured for use by SCTP.

The algorithms for the disambiguation of retransmissions are meant to 
counter problems caused by anomalies such as sudden delay spikes, packet 
reordering or, in the case of SCTP, retransmissions alternating between 
paths with different RTTs and separate path estimators. I believe it is 
next to impossible to "guess" the timestamps with a good hitrate under 
those conditions.

Stephan.

---

From: Stephan Baucke <[email-masked]>
Date: Thu, 17 Jul 2003 15:07:43 +0200

Caitlin,

getting better knowledge of the RTT is only one of the aspects. Eifel 
basically works by marking packets and ACKs in a way that allows the 
sender to distiguish whether an ACK refers to the original transmission 
or a retransmission. For things like disambiguating the path the chunk 
being acknowledged was sent over, it needs to be able to identify which 
(re)transmission it was.

> to allow measurements to be made accurately.

It's not really about maintaining a common clock. There are other 
possible ways to achieve the goals mentioned above (e.g. marking packets 
with dedicated flags), but the timestamps have worked out nicely for 
TCP, and seem attractive to me because they can be used to solve several 
problems.

Stephan.

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 17 Jul 2003 09:17:36 -0500

>timestamps with every packet that contains retransmitted chunks.

The chunk size was already determined on the first transmit. There
won't be any more room on the retransmit.

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 17 Jul 2003 09:22:43 -0500

On Thursday, July 17, 2003, [email-masked] wrote:

>The algorithms for the disambiguation of retransmissions
>are meant to counter problems caused by anomalies such as
>sudden delay spikes, packet reordering or, in the case of
>SCTP, retransmissions alternating between paths with
>different RTTs and separate path estimators. I believe it
>is next to impossible to "guess" the timestamps with a
>good hitrate under those conditions.

You don't need to predict them. But you could easily detect
that they are out of the predictable range for a
retransmit-free stable path. That is, you could once a
history on the path was established.

For sporadic traffic this type of approach would indeed be
ineffective, but it is when you are transferring on a
sustained basis that you will be most concerned about not
inserting timestamps. Adding a small timestamp to sporadic
traffic will seldom be objectionable.

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 16 Jul 2003 17:04:09 -0500

It really IS an implementation issue and has already been addressed
since the very first implementation (and I'd guess each of the existing
impls does the same... just in its own different way). For example, in
the open source user-land ref impl as I remember it, there are two
queues; one for pending control chunks and the other for pending user
messages. Whenever an outgoing packet is being formed, the bundling and
fragmentation code will always first include all the pending control
chunks and then figure out how much room is still left for user data. In
the calculation, it always count in the data chunk header, SCTP header,
and IP header... It is tricky code... one of the focal points the early
debugging excitement :-)

-Qiaobing

---

From: Pasi Sarolahti <[email-masked]>
Date: Mon, 21 Jul 2003 20:24:12 +0300 (EEST)

Hi Sourabh,

Sorry for the delayed response. I have been behind a poor network access,
so my mail reactions are a bit slow currently :-) For the same reason I 
may not have read all the other mails in this thread.

> wondering if the following are applicable:

First, I have to be honest, we have mainly considered F-RTO as a TCP
algorithm and it was only recently when we started thinking that it should
apply on SCTP also. It seems that the special features of SCTP may need to
be discussed a bit more than the draft currently does.

Thanks for bringing up these good points, that definitely need to be
addressed. Nevertheless, I'm still optimistic that the principle of F-RTO
should work with SCTP, too: after an RTO, if an acknowledgement arrives
for a data chunk (or segment) that was sent before the RTO, the
RTO can be declared spurious.

> retransmissions SHOULD go to alternate destination)

So, the RTO retransmission goes to alternate path and ACK for the
retransmission comes back from the alternate path. Then, the F-RTO sender
transmits one or two chunks that were not transmitted earlier. These
segments should go to primary destination, since they are not
retransmissions, right? If they do, then everything goes as intended,
since the segments will be queued behind other segments that were sent to
the primary address.  In case of spurious timeout, next ACK or SACK should
acknowledge something that was not retransmitted. Maybe I missed something
here?

How would this go with timestamps? ACK (and tsecr) of the retransmission 
will arrive to the sender from the alternate destination before the ACK of 
the original segment, right? So Eifel would not detect a spurious RTO?

> side effects of this which could lead FRTO to declare the timeout spurious

According to the principle I gave above, only acknowledgements of data
chunks (or segment) that were not transmitted after RTO should be used
for declaring the RTO spurious. Chunk bundling is something I did not
consider earlier, so probably this is something that needs to be
explicitly stated when applying the algorithm for SCTP. I think we could
add some kind of "SCTP considerations" section that would address these
kind of issues, plus probably reword the algorithm description to be more
clear on these.

I would be interested to hear about the side effects you mention above in
more detail. False positives are something we definitely want to avoid.  

> acknowledging {via cum-ack point} or {via SACK blocks} or {either of these

either of the two. This is worth clarifying in the algorithm. Thanks for 
pointing it out.

- Pasi

---

From: Sourabh Ladha <[email-masked]>
Date: Mon, 21 Jul 2003 19:36:04 -0400 (EDT)

Hi Pasi,

Before sending a reply for FRTO-SCTP, I would like to get a bit of
clarification for Step 2 of the SACK-enhanced version of FRTO
(this applies for TCP as well):

"   2) The first acknowledgement after RTO arrives at the sender.

      a) if the cumulative ACK acknowledges all segments up to send_high
         stored in algorithm step 1, the TCP sender SHOULD revert to the
         conventional RTO recovery and it MUST set congestion window to
         no more than 2 * MSS. The sender does not enter step 3 of this
         algorithm.

      b) otherwise, the TCP sender MAY transmit two new segments. If the
         TCP sender does not transmit any previously unsent data, it
         MUST NOT enter step 3 of this algorithm, but revert to the
         conventional RTO recovery."

So I believe that even on the reciept of a dup-ack (with a SACK block) an
FRTO sender follows step 2(b)?

If so, In the event of loss (xxlost) coupled with a delay spike (/\):

A				    B
----------TSN101------>xxlost
----------TSN102--------------/\--->
----------TSN103--------------/\--->
  .
  .
timeout
----------TSN101------------------->
<-----SACK101 (sack block: 102)---(sack generated when TSN 102 reached B)
FRTO sends out TSN104 and TSN105 and enters step 3.
<-----SACK101 (sack block: 103)---(sack generated when TSN 103 reached B)

From your clarification of Step 3(b) below, FRTO will declare the timeout
spurious on the receipt of this SACK?

A false positive?

-Sourabh

---

From: Pasi Sarolahti <[email-masked]>
Date: Tue, 22 Jul 2003 13:53:41 +0300 (EEST)

Hi,

On Mon, 21 Jul 2003, Sourabh Ladha wrote:

[...]
> FRTO sender follows step 2(b)?

Yes, and it appears to be wrong, as you presented. The basic algorithm
would not do this and it seems that the SACK-based variant should also
wait for acknowledgement for the RTO-retransmitted segment before
proceeding.

> spurious on the receipt of this SACK?

Oops... You are right. It seems to be essential to get acknowledgement for
RTO-retransmitted segment before going to step 3. The question is, should
the sender wait (in step 2) until the ack arrives (or until another RTO
occurs), or should it cancel F-RTO immediately on dupack. The basic
algorithm in Section 2 does the latter, but I am wondering if the former
is safe enough alternative. It works better if reordering occurs at the
same time.

Ok, now if the connection is multihomed and delay spike is longer than RTO
(primary) + 2 RTT (alternative) there could be problems with this
scenario. I need to think about this more, but it seems that in multihomed
cases the use of F-RTO needs to limited a bit. It detects a delay spike,
but it may not always detect if the first segment is lost, if the
retransmission goes to different destination than the original, and the
path characteristics are as shown above. However, combining with duplicate
TSNs should work: F-RTO can be used to avoid unnecessary retransmissions
and draft-dsack-use can be used for congestion window undoing.

..or you could use timestamps, once specified. But back to the earlier
mail, wouldn't Eifel have some kind of problems in detecting a real
spurious RTO if retransmissions go to alternative path such as shown
above?

Thanks,
- Pasi

---

From: [email-masked]
Date: Tue, 22 Jul 2003 07:15:04 -0400 (EDT)

Hi Pasi,

> spurious RTO if retransmissions go to alternative path such as shown

You are right. I did stumble across this issue. Eifel "for TCP" if ported
"as it is" to SCTP did present these cases where in a multihomed scenario
(with such a setting as described in the earlier mail), Eifel did not
detect a spurious timeout when it was. But the vice-versa was not true,
i.e Eifel did not detect a loss as a spurious timeout. (Note that the case
is less pronounced in the event of a spurious fast-retransmit). However as
you also pointed out, integrating with the dup-tsn notifiation adds some
robustness.

Thanks,
Sourabh

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 22 Jul 2003 12:07:53 -0700

Pasi Sarolahti wrote:

 > And after the examples shown by Sourabh, I believe detecting spurious
 > RTOs with multihoming brings interesting questions for both F-RTO and
 > Eifel that I need to study more thoroughly.

I guess after all the discussions so far, SCTP implementors should
think about not to follow what is recommended in RFC 2960, timeout
retransmission to an alternate address.  Just a suggestion, it seems
that at least for the first timeout, the retransmission is probably
better sent to the original address.  And when this first retransmission
is also lost leading to another timeout, SCTP should start considering
using an alternate address.

-- 

						K. Poon.
						[email-masked]