Subject: [Rmt] TFMCC Building Block document comments

From: Brian Adamson <[email-masked]>
Date: Thu, 20 Feb 2003 12:49:31 -0500

Hello Jorg, Mark,

I've been updating the NORM BB & PI documents to be consistent with 
the RMT guidelines and to be consistent with the pertinent building 
block documents.  In the previous NORM PI draft I had taken a shot at 
specifying TFMCC-like congestion control for NORM based on the TFMCC 
SIGCOMM papers and other related work.  This was prior to the release 
of the TFMCC draft.  I'm changing terminology in the NORM draft to be 
consistent with the terminology in the TFMCC draft and making some 
other changes.

I think the TFMCC draft is very well-written, very well thought-out, 
and very clear.  However, there a couple of areas where the TFMCC 
document, as a "building block", perhaps over-specifies some details 
for RMT protocol instantiations using its techniques.   In several 
cases, different but functionally equivalent mechanisms might better 
fit a particular protocol instantiation (e.g. NORM).

In the case of NORM, the only significant deviations from the wording 
of the TFMCC document is that NORM uses a special congestion control 
message instead of including overhead in every packet and that 
individual RTT measurements are performed by the sender and provided 
to the receiver group.  With respect to the TFMCC approach, 
equivalent functionality is provided.  (The previous NORM PI didn't 
have a "receiver_leave" flag specified, but this is being remedied). 
For the most part, I think the remaining NORM/TFMCC inconsistencies 
can be addressed by adding some more general language to the TFMCC 
recommendation to allow for different data sender protocol 
assumptions.  The only other issues are that the NORM BB and TFMCC BB 
give two different RTT_max/GRTT estimation algorithms (see below) and 
a minor comment on sender rate adjustment in the absence of feedback 
from the CLR.

So, here are my comments on the TFMCC document:

---------------------

1) TFMCC Data Sender Protocol Assumptions

With respect to the NORM PI, the most significant semantic difference 
between it and TFMCC is that the NORM protocol uses a separate 
message (NORM_CMD(CC)) type to which is periodically sent once per 
CLR RTT (or RTT_max) to excite congestion control feedback from the 
receiver set rather than including overhead in every NORM_DATA 
message.  In simulations I have conducted, I seem to get congestion 
control performance comparable to TFMCC with this approach.

The reasons for this different approach is:

a)  With the overhead to accommodate FEC, NORM message overhead is 
already quite large and the congestion control additions start to 
significantly eat into the space available for user data given 
networks with limited MTUs.

b) Although congestion control operation will be mandatory for 
general Internet use, the anticipated NORM user base will have a 
great deal of application in specialized networks where TCP-friendly 
congestion control may not be necessary or desirable.  (The separate 
NORM_CMD(CC) message can allow for alternate congestion control 
schemes to be employed in these networks and its exclusion provides 
for fixed sender-rate operation).

c) In addition to collecting explicit feedback from the "current 
limiting receiver" (CLR), the NORM_CMD(CC) also provides for optional 
collection of explicit feedback from a small set of "potential 
limiting receivers" (PLRs)  (This was termed "worst path candidates" 
in the previous NORM draft).  The probing of the PLR set allows for 
rapid congestion control response to receivers which, although 
perhaps not the current CLR, have been somehow identified (perhaps 
even administratively) as critical in congestion control (or flow 
control) operation.  Our reliable multicast user base has expressed 
interest in this option.

Impact on TFMCC Document:

The TFMCC document could have language added to note that the data 
sender protocol might provide congestion control status to the group 
in a fashion as described above (at least once per CLR RTT or 
RTT_max).  This mechanism also allows for a slightly different 
control paradigm for the TFMCC notion of "feedback rounds".  In NORM, 
each NORM_CMD(CC) message sent by the data sender contains a 
"cc_sequence" number which basically identifies the "feedback round"

---------------------
2) Determining Maximum RTT

The NORM BB document had specified a different means for tracking 
R_max (the NORM BB had termed this "GRTT") which we have been using 
for some time.

The NORM GRTT algorithm provides a small amount of smoothing of GRTT 
peaks and reduces the GRTT only after multiple (3) consecutive rounds 
(thus allowing for feedback loss) with feedback indicators less than 
the current GRTT.

I don't see these two different algorithms (TFMCC and NORM-BB)  likey 
performing extremely differently for the most part, but they are 
different and either one should be selected for the RMT documents or 
some explanation given.  I don't think this issue has been thoroughly 
studied and I'm not sure of all of the advantages/disadvantages of 
either over the other.  The reason we smoothed the GRTT peaks a bit 
was that in experience, "user-space" code on a busy operating system 
would often give us large peaks and this seemed to help with the 
trade-off between protocol latency and feedback suppression.  Since 
the next step is Experimental RFC for these documents, I think that 
perhaps some discussion be provided.  I can do this in the NORM-BB. 
As for TFMCC, it may want to present its recommendation, but note 
this as an area for further study/experimentation?

---------------------

3) Adjusting the Sending Rate

As a congestion avoidance measure, it may be useful to also reduce 
the transmission rate when no feedback is received from the CLR (even 
though feedback might have been received from other receivers), prior 
to timing-out the CLR under the "crashed or left group" assumption. 
For example, if no feedback is received from the CLR after 2-3 RTT, 
the rate should be reduced as a congestion avoidance measure.

---------------------

4) Receiver-Side RTT Measurements.

In the NORM PI, the receivers echo back the sender timestamp (from 
previous NORM_CMD(CC) messages) and the sender performs the RTT 
measurement (it can use the receiver's provided previous RTT to 
smooth the measurement if the NORM_FLAG_RTT (equalt to "have_rtt") 
flag is set).  The sender provides the RTT to receivers in a 
subsequent NORM_CMD(CC) transmission  using the same sort of 
prioritization rules that are described in TFMCC to break ties.  Note 
that one-way delay adjustments can still be done with this approach. 
This approach used by NORM is a little more efficient with overhead 
(in the context of NORM) since the RTT field provide to receivers can 
be readily compressed.  Also this approach allows receivers to be 
aware of other receivers RTT which can potentially be used to make 
multi-sender RTT determination more efficient (This is perhaps 
outside of the immediate scope of one-to-many bulk transfer but may 
be applicable in the long run)

-- 
Brian
__________________________________
Brian Adamson
<http://manimac.itd.nrl.navy.mil>
<mailto:[email-masked]>
_______________________________________________
Rmt mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rmt

---

From: Joerg Widmer <[email-masked]>
Date: Fri, 28 Feb 2003 15:37:50 +0100

Hi Brian,
thanks for your comments on the TFMCC draft.

1) I don't see any problems with the proposed changes. Having the
congestion control information separate from the data packet seems fine
to me though it'll limit the number of possible RTT measurements per 
feedback round. I can add a corresponding paragraph to the draft.

2) As for how to determine the max RTT I'll implement the NORM version 
and run some simulations to better understand it's impact on TFMCC.

3) We could align this with the TFRC draft where the sending rate is 
reduced by half if a nofeedback timer expires (with timeout value 
4*RTT). Seems reasonable to me.

4) If sender-side RTT measurement are better suited for NORM we can add 
a paragraph that either sender- or receiver-side RTT measurements are 
possible as long as they allow for one-way RTT adjustments.

Mark, are you ok with this?

Cheers,
    Joerg

  >
  > In the case of NORM, the only significant deviations from the wording of
  > the TFMCC document is that NORM uses a special congestion control
  > individual RTT measurements are performed by the sender and provided to
  > the receiver group.  With respect to the TFMCC approach, equivalent
  > functionality is provided.  (The previous NORM PI didn't have a
  > "receiver_leave" flag specified, but this is being remedied). For the
  > most part, I think the remaining NORM/TFMCC inconsistencies can be
  > addressed by adding some more general language to the TFMCC
  > recommendation to allow for different data sender protocol assumptions.
  > The only other issues are that the NORM BB and TFMCC BB give two
  > different RTT_max/GRTT estimation algorithms (see below) and a minor
  > comment on sender rate adjustment in the absence of feedback from the
CLR.
  > between it and TFMCC is that the NORM protocol uses a separate message
  > (NORM_CMD(CC)) type to which is periodically sent once per CLR RTT (or
  > RTT_max) to excite congestion control feedback from the receiver set
  > rather than including overhead in every NORM_DATA message.  In
  > simulations I have conducted, I seem to get congestion control
  > performance comparable to TFMCC with this approach.
  >
  > significantly eat into the space available for user data given networks
  > with limited MTUs.
  >
  > b) Although congestion control operation will be mandatory for general
  > Internet use, the anticipated NORM user base will have a great deal of
  > application in specialized networks where TCP-friendly congestion
  > control may not be necessary or desirable.  (The separate NORM_CMD(CC)
  > message can allow for alternate congestion control schemes to be
  > employed in these networks and its exclusion provides for fixed
  > sender-rate operation).
  >
  > collection of explicit feedback from a small set of "potential limiting
  > receivers" (PLRs)  (This was termed "worst path candidates" in the
  > previous NORM draft).  The probing of the PLR set allows for rapid
  > congestion control response to receivers which, although perhaps not the
  > current CLR, have been somehow identified (perhaps even
  > administratively) as critical in congestion control (or flow control)
  > operation.  Our reliable multicast user base has expressed interest in
  > this option.
  >
  > sender protocol might provide congestion control status to the group in
  > a fashion as described above (at least once per CLR RTT or RTT_max).
  > This mechanism also allows for a slightly different control paradigm for
  > the TFMCC notion of "feedback rounds".  In NORM, each NORM_CMD(CC)
  > message sent by the data sender contains a "cc_sequence" number which
  > basically identifies the "feedback round"
  >
  >
  > ---------------------
  > The NORM BB document had specified a different means for tracking R_max
  > (the NORM BB had termed this "GRTT") which we have been using for some
  > time.
  >
  > (thus allowing for feedback loss) with feedback indicators less than the
  > current GRTT.
  >
  > either over the other.  The reason we smoothed the GRTT peaks a bit was
  > that in experience, "user-space" code on a busy operating system would
  > often give us large peaks and this seemed to help with the trade-off
  > between protocol latency and feedback suppression.  Since the next step
  > is Experimental RFC for these documents, I think that perhaps some
  > discussion be provided.  I can do this in the NORM-BB. As for TFMCC, it
  > may want to present its recommendation, but note this as an area for
  > further study/experimentation?
  >
  > ---------------------
  >
  > As a congestion avoidance measure, it may be useful to also reduce the
  > transmission rate when no feedback is received from the CLR (even though
  > feedback might have been received from other receivers), prior to
  > timing-out the CLR under the "crashed or left group" assumption. For
  > example, if no feedback is received from the CLR after 2-3 RTT, the rate
  > should be reduced as a congestion avoidance measure.
  >
  > ---------------------
  >
  > measurement (it can use the receiver's provided previous RTT to smooth
  > the measurement if the NORM_FLAG_RTT (equalt to "have_rtt") flag is
  > set).  The sender provides the RTT to receivers in a subsequent
  > NORM_CMD(CC) transmission  using the same sort of prioritization rules
  > that are described in TFMCC to break ties.  Note that one-way delay
  > adjustments can still be done with this approach. This approach used by
  > NORM is a little more efficient with overhead (in the context of NORM)
  > since the RTT field provide to receivers can be readily compressed.
  > Also this approach allows receivers to be aware of other receivers RTT
  > which can potentially be used to make multi-sender RTT determination
  > more efficient (This is perhaps outside of the immediate scope of
  > one-to-many bulk transfer but may be applicable in the long run)
  >
  >

---

From: Mark Handley <[email-masked]>
Date: Fri, 28 Feb 2003 10:09:32 -0800

>feedback round. I can add a corresponding paragraph to the draft.

We should probably state that it can be piggybacked, or in separate
packets.  Then we should state what the requirements are for how often
we need to measure, taking loss into account.  

>4*RTT). Seems reasonable to me.

You might want to be conservative on the nofeedback timer - you'd like
to take variance into account, not just RTT, but this isn't too easy
to do for TFMCC.

>possible as long as they allow for one-way RTT adjustments.

Although the receiver does need to have its RTT measurement to figure
out congestion events.  I have a slight preference for specifying one
way to do this only.  I think receiver-based is slightly preferable,
because it adds half an RTT less delay.  Is there a good reason NORM
can't do receiver-based?

 - Mark

---

From: Brian Adamson <[email-masked]>
Date: Fri, 28 Feb 2003 15:04:18 -0500

>because it adds half an RTT less delay.  Is there a good reason NORM

I think in both cases, NORM and TFMCC, the sender can determine the 
RTT to a receiver upon receiving a feedback message from the receiver 
since the receiver message has a previous sender timestamp in it. 
(The TFMCC receiver feedback message also has a receiver timestamp)

With NORM, the sender echoes the calculated RTT back to the receiver 
where in TFMCC the sender echoes the receiver timestamp back to the 
receiver ... in this respect the TFMCC receiver RTT measurement is 
half an RTT fresher.  I don't think this is significant with respect 
to congestion control performance.

NORM's sender-side approach gets rid of the need to communicate the 
receiver timestamp in feedback messages.   And in the messages sent 
by the NORM sender, the RTT can be compressed to a smaller field than 
the receiver timestamp.  The current NORM approach also allows other 
receivers to see the RTT of other receivers ... While this isn't 
currently useful in a one-to-many bulk transfer paradigm, knowledge 
of other NORM nodes' RTT might be leveraged for multi-way RTT 
determination in the future.  It's also a _little_ less CPU work for 
the sender to just provide the RTT measurement to tre receiver and 
not also keep track of the hold-time and adjust the receiver 
timestamp echoed.

NORM _could_ do receiver-based if you think it will significantly 
impact congestion control performance.  This could be done instead of 
or in addition to the sender-provided RTT.  But in my simulations, 
both seem to work equivalently well - but I haven't conducted an 
extensive study of this issue.