Subject: [Tsvwg] pr-sctp -- final thoughts from me

From: Mark Allman <[email-masked]>
Date: Mon, 07 Apr 2003 08:05:09 -0400

Folks-

I do not want to continue to repeat myself on the list.  After
reading through this thread and having some on-list and off-list
dicussions I am left with the following thoughts/clarifications
(/FUD - depending on your viewpoint).

  * I never said and hopefully did not insinuate that there was no
    use/need/application for PR-SCTP.  So, for all of the folks who
    have written me enumerating all sorts of applications: I agree,
    there are compelling apps for PR-SCTP (and, more generally
    partially reliable transports).

    I am not advocating throwing the baby out with the bath water.
    I am wondering about whether we should think about enforcing a
    little more network friendliness because we can.

  * Just because a transport sends at a rate that is dictated by CC
    doesn't mean it's doing so in a terribly useful or friendly
    manner.  Just because a transport is allowed to send at X bps
    should not give it the right to send just anything at X bps.  If
    lots of the traffic is redundant the transport is using network
    resources that could be put to better use (to accomplish useful
    work).  In my opinion, *** being a well-behaved transport is not
    just about sending at the "right" rate. *** (Although, note that
    sending at the "right" rate is *part* of being network
    friendly.)

  * PR-SCTP is network friendly in one sense because it can prevent
    needless retransmits from being sent (i.e., because an
    application can no longer use the data).  This is a Good Thing.

    But, if we're using this PR functionality to squeltch a high
    percentage of messages from being retransmitted then I think we
    turn this Good Thing into a Bad Thing.  Sure, the transport is
    still preventing needless retransmissions, but at the same time
    it is acknowledging that it's cluttering up the network with a
    lot of packets that are not needed.

    IMO, a little PR is fine, but lots of PR is not.  (Where I
    purposely leave "a little" and "a lot" ambiguous because I don't
    know how to precisely define them.)

  * My own personal opinion is there are a number of angles to look
    at how "network friendly" a transport protocol happens to be.
    I believe that one aspect of this is how much useless data the
    transport injects into the network.  

    In this context I believe we need to think about the
    *incentives* a transport protocol offers to applications using
    it.  In addition, we need to look at the *opportunity* the
    transport has to police the sending of redundant data.

    (And, note, that this is not something I have selectivly applied
    to SCTP.  I have explored this in TCP as well.  For instance, in
    work that Ethan Blanton and I have done on making TCP robust to
    reordering one of the goals is to adjust the duplicate ACK
    threshold so that TCP does not pollute the network with lots of
    needless retransmissions that are caused by packet reordering
    not packet loss.  I.e., I am not attempting to disparage only
    SCTP here.  This is a general issue, but I think PR-SCTP is a
    place to apply anti-pollution strategies.)

  * I want to discuss several transports in the context of the above
    bit about incentives and opportunities, since others have
    brought them up.

    If we look at UDP I don't think there is any dis-incentive
    offered by the transport to prevent the application from sending
    lots of redundant data into the network.  If the application
    sends every message twice in case one of them is lost there is
    really no transport-level problem with that (i.e., this behavior
    does not cost the transport anything).  But, the flip side is
    that there is no opportunity for the transport to do anything
    about it.  Each message sent from the application to the
    transport is just transmitted without any sort of accounting,
    data naming, etc.  So, the transport is ignorant of what is
    being sent.  So, while UDP can be abused by applications that
    choose to do so, there is pretty much nothing that UDP can do
    about it.

    Next, look at TCP (and, really, any ordered, reliable
    transport).  TCP provides a huge dis-incentive for applications
    to send multiple copies of a message for redundancy purposes.
    TCP ensures that all copies will arrive -- and in the order the
    were given to the sending transport by the sending application.
    There is head-of-line blocking so there is just no good reason
    to send an application message twice to try to get better
    performance (via some metric).  (Note: TCP itself can be more or
    less careful about what it resends and then starts getting into
    these issues.  For instance, early retransmit allows for the
    resending of data with a less robust loss indicator than we
    currently use.  That *could* have the impact of inserting a
    bunch of useless data into the network in the path that has
    persistent reordering.  This should be prevented.)

    Next up is unordered, reliable SCTP.  In this case, applications
    can send multiple copies of some message for some possible gain.
    Since there is no head-of-line blocking it may make sense
    (performance-wise) for an application to send multiple copies of
    some message to the transport and then throw any extra copies
    that arrive at the receiving application in the bit bucket.  The
    dis-incentive provided by the transport in this case is that the
    extra messages (some of which are useless) take space in
    buffers, take space in the congestion window and take
    (re)transmission opportunities from other data.  And, all
    messages take these resources until they are successfully
    delivered to the receiving transport.  Depending on the
    application these tradeoffs an application designer might think
    these tradeoffs are worth it and the abuse to the network will
    happen.  But, that is not the whole story.  The other side of
    this variant of SCTP is that the transport is "out of the loop"
    on what is redundant and what is not.  Since all the redundancy
    is happening in application land (as in the UDP case) SCTP has
    no real clue about what is going on and therefore has no
    opportunity to police the stream and prevent redundant data.

    Finally, we get to PR-SCTP -- where I believe incentive and
    opportunity finally cross paths.  Think of the application
    passing down multiple copies of the same message to the
    transport with short lifetimes.  So, the job of the extra copies
    of the message is to "back up" the first and get to the
    application if the first is lost (quicker than a retransmission
    of the first could).  Now the transport is giving an incentive
    to the application to do this and has an opportunity to prevent
    excessive (ab)use of it.  The dis-incentive to do this is quite
    similar to the case above for unordered, reliable SCTP -- that
    is, the redundant data takes cwnd space, buffer space and
    transmission opportunities.  But, PR-SCTP goes a step further
    because the application (in effect) gets to tell the transport
    to release the resources quickly.  So, rather than being
    required to use these resources until a loss is repaired they
    can be freed and used for other things.  This is a Good Thing in
    moderation and a Bad Thing in excess, IMO.  In my opinion, the
    dis-incentive for PR-SCTP is *less* than the dis-incentive for
    unordered SCTP.  But, the really important bit that distiguishes
    PR-SCTP from the unordered, reliable SCTP is that PR-SCTP is "in
    the loop" and knows exactly what is going on (i.e., which data
    was sent & lost, but will not be re-sent).  And, hence, PR-SCTP
    can do something about it to prevent excessive (ab)use of the PR
    mechanism.

    So, my opinion is still that PR-SCTP *does* provide *incentive*
    for application abuse *and* yet also has an *opportunity* to
    police it and to ensure that the use of the PR feature is not
    excessive.  And, I believe that having both of those is the
    fundemental discriminator between PR-SCTP and all of our
    previous (popular) transports.

  * It's been suggested that I am advocating things that can be
    accomplished with twisting of other protocols (e.g., multiple
    connection TCP).  Well, I am not going on a long rant here --
    although it is tempting. ;) In any case, fine, I'll admit that
    if you twist and warp things they can be abused.  So, what?
    Does that give us the license to produce lousy protocols?  Did
    the fact that mutliple TCP connections viloates standard TCP CC
    or that UDP does not use CC allow us to design bogus (or no) CC
    into SCTP?  No.  The argument that since we can abuse other
    things we should not have to be careful in the design of PR-SCTP
    is just bogus.  That's the end of my mini-rant -- I am sure you
    can fill in the rest! :)

There is an old tradition in the IETF that says that repeating your
argument over-and-over makes you more-and-more correct.  I don't buy
it (although, alas, sometimes fall into the trap) and so I intend
for this to be my last note in this thread (on tsvwg) because I
don't think I have said much new in here (but, hopefully, in a more
coherent and comprehensive way), or really since the thread
started.  (Besides, if you have made it this far you're all sick of
listening to my bullshit anyway!)

I am intersted in hearing what other folk have to say.

Thanks for listening,
allman

--
Mark Allman -- BBN/NASA GRC -- http://roland.grc.nasa.gov/~mallman/
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 7 Apr 2003 12:52:14 -0500

On 4/7/03, Mark Allman wrote:

> transport).  TCP provides a huge dis-incentive for
> applications to send multiple copies of a message for
> redundancy purposes. TCP ensures that all copies will
> arrive -- and in the order the were given to the sending
> transport by the sending application. There is
> head-of-line blocking so there is just no good reason to
> send an application message twice to try to get better
> performance (via some metric).  (Note: TCP itself can be
> more or less careful about what it resends and then
> starts getting into these issues.  For instance, early
> retransmit allows for the resending of data with a less
> robust loss indicator than we currently use.  That
> *could* have the impact of inserting a bunch of useless
> data into the network in the path that has persistent
> reordering.  This should be prevented.)

TCP is actually fully compatabible with this type of
cheating, without any cost to the abusive sender.

The sender simply opens multiple redundant connections
(leaving even less capacity for those pesky other users).
The same TCP segments are then sent on all paths.

Once *any* connection has acked material, the sender simply
discards the buffers. The retention is based on the
*fastest* ack, not the slowest.

Any connection that ultimately needs the now missing data is
simply reset, and then re-established. More network traffic,
but what's that compared to our reckless applications buffer
space? It takes time, so better have *several* spare
connections.

As you already concede, UDP does not have the ability to
detect redundant data transmission by the application.
Therefore it is fully compatible with abusive programs.

Most uses of PR-SCTP would have otherwise used UDP, not
TCP or SCTP. So there is certainly no redudction in the
networks ability to prevent abusive traffic.

Beyond that, what change would be made anyway?

And more importantly, wouldn't it be strictly a local
interface question? The wire messages that invalidate
the need to wait for previously sent data chunks do
not cite a reason.

Whatever safety checks/guidelines might be agreed upon,
they will not be wire visisble. I am also skeptical that
any useful policies could be drafted with any teath beyond
"SHOULD NOT"s. It is very hard to encode real-time
enforcement that NEVER has a false positive. And any
real-time enforcement MUST NOT block ANY legitimate
practice (no matter how strange).

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Michael A. Ramalho" <[email-masked]>
Date: Tue, 08 Apr 2003 09:47:22 -0400

Mark,

My comments, as promised. Comments are in-line.

Michael Ramalho

At 08:05 AM 4/7/2003 -0400, Mark Allman wrote:
>     should not give it the right to send just anything at X bps.

Let me understand you. I find TCP a form of socialism ... each
of N sessions get 1/Nth (on very rough average) of
the (bottleneck) bandwidth.

You desire to go beyond socialism ... and not allow what you
see as *applications created* redundant data ... even if you are
within your 1/Nth ... even, perhaps, if you are within the QoS
bandwidth you paid a service provider for? Many people would
disagree with your point of view here for all "better than best-effort"
application environments. I do note that many, if not the majority,
of sigtran environments are QoS provisioned.

Note: I have no problem with minimizing "redundant data" in
a given layer (your transport optimizations examples below).

>If

OK. This is a "generic goodness" argument .. but is hard or
impossible to enforce.

If an application has to send twice (or send FEC/redundant
information) in order to meet it's delay performance requirements
and it is otherwise abiding in a responsible manner (e.g., paid
for the bandwidth in a QoS-enabled environment or is otherwise
within it's designed transport bounds) .... who are we to say
that the application isn't "network friendly"?

I, as will be seen below, fully agree with minimizing duplicate
data being sent in a given layer (e.g., wholly within transport layer).

>     lot of packets that are not needed.

Beauty (of data) is in the eye of the beholder. I view most of the
data that pr-sctp decides to do without (because of staleness) *is* data
that we would have liked to have received. But given a choice of getting
"more relevant data now" or "getting the old, potentially irrelevant, data
now ... then the more relevant data even later" ... we choose the former.
That does not mean the data which we decide not to re-transmit was
"clutter" when it was originally sent.

>     transport injects into the network.

I agree in general. But note you said "transport layer injects"
(not "application layer injects").

>     transport has to police the sending of redundant data.

I'm a little confused here. Your next paragraph describes
mechanisms *in the transport layer* to limit the amount of
truly useless (re-transmitted) data sent *by the transport
layer*. I have no problem with such goals and wholeheartedly
agree.

But how does the transport layer know of redundant data sent
to it by the application? Surely the redundant data need not
be identical ... it could have been formatted differently? How
would the transport layer know?

It is not the transport's job to police this!

>     choose to do so, there is pretty much nothing that UDP can do

Agree.

>     performance (via some metric).

As noted previously ... an application could open multiple
TCP connections, though. Not only would this increase the
probability that the message got there with one-way delay, it
would also double it's "share" of the bandwidth (assuming
two TCP connections) and resources (ports, etc.).

We both agree this is bad - and that there is *nothing* we can
do about it from an *applications enforcement* perspective.
The transport layer cannot be expected to "police" for this
redundant data. Store this thought for use later.

>(Note: TCP itself can be more or

Of course, unordered reliable SCTP was designed for messages
that were *uncorrelated* ... you are trying to "abuse" the very
premise of design for unordered service (via 100% correlated
messages). Are you suggesting that because it *can* be
abused ... that it shouldn't exist for the services or applications
that it was designed for? I don't think so ... but I'd like to find
the alternative.

>     that arrive at the receiving application in the bit bucket.

The incentive *only exists* if the SCTP session *isn't presently
using it's fair share*. If it is ... then there is the appropriate
disincentive ..... less throughput/goodput and potentially more
delay of the "other" information.

To me ... this is an even greater *disincentive*. Be "greedy" and
it "hurts you more" during the congestion episode! Exactly
when you need performance ... your greediness hurts you!

I fail to see why you think this is an incentive for bad
applications behavior; as the application is "punished" via
poorer (average) performance when the going gets tough.

I understand your desire to minimize "redundant information"
for generic goodness purposes .... but just like we can't
mandate non use of some of the present day RTP/UDP
tricks ... we can't mandate that applications can't do this. Applications
that do as you suggest can potentially get better performance
during non-congestion periods in exchange for poorer performance
during congestion episodes. But at least they can't use more than
their fair share *during congestion* if they use SCTP. This seems
like pretty optimal behavior ... when the going "gets tough" ... you
protect "the network". When the network has excess capacity, you
live with some bad apples.

>The
>     happen.

But ... unlike the multiple TCP case ... it still only uses it's
(singular) fair share of the bandwidth.

>  But, that is not the whole story.  The other side of

True. It isn't the transport's job to police the data. SCTP doesn't
see the data as "redundant" .... and it still doesn't use more than
it's share of the bandwidth.

>     transport with short lifetimes.

Call me dense ... but if you have a few RTT's to get the data
across ... I still don't understand why you would do this?

Because ... if you have a few RTT's ... you would only send it
once with the appropriate lifetime.

If you do not have a few RTT's and you really should be
using UDP/DCCP with the appropriate FEC.

Am I wrong in thinking that you are attempting to "abuse"
PR-SCTP in a manner in which no transport protocol can protect
against? If you try to formulate a mechanism in which the
transport polices the application layer information to make
sure that it is not sending redundant information - what you
are suggesting is a clear layer violation.

I have no problem with the TCP fast/quick restart tweaks
you mentioned previously ... as they were fixes entirely
*within* the transport layer.

>So, the job of the extra copies
>     excessive (ab)use of it.

Even if an "application abuser" did this ... there is still less incentive
than multiple TCP sessions .. and a disincentive during congestion
episodes (as above).

>The dis-incentive to do this is quite
>     can be freed and used for other things.

Yes. By design.

>This is a Good Thing in
>     unordered SCTP.

OK. Now you send multiple identical messages with shorter lifetimes.
Let's look at the two cases.

Case One: Lifetimes greater than a few RTTs. There is *no incentive*
to do this ... as a singular message would make it under reasonable
packet error conditions.

Case Two: Lifetimes on the order of a two RTTs or less. The case for
abuse is there ... but .... PR-SCTP *still* won't use it's fair share of
bandwidth. And (PR-)SCTP is still ignorant of whether the data it
has been passed is redundant.

I fail to see how you can do better than this *without layer violating
application <-> transport policing*.

>But, the really important bit that distiguishes
>     excessive.

Please explain how PR-SCTP has an opportunity to police without
violating the application/transport. I don't quite get this ... and you
haven't provided an example of how the transport layer would
determine "redundant data".

Even if an app "abuses" PR-SCTP in the manner you envision ... it
still doesn't get more than it's bandwidth share.

>And, I believe that having both of those is the

I'm not trying to be difficult ... I just do not know how to protect
against what you seem to object to without layer violation.

>I am intersted in hearing what other folk have to say.

Michael A. Ramalho, Ph.D.
Office email: [email-masked]
Personal email: [email-masked]
Office: +1.941.708.4650
Cell: +1.941.544.2844

---

From: Mark Allman <[email-masked]>
Date: Tue, 08 Apr 2003 10:24:51 -0400

Folks-

I lied.  I am replying to this thread.  But, just as a clarification
because I chose some unfortunate language in my original note.

> violating the application/transport. I don't quite get this
> ... and you haven't provided an example of how the transport layer
> would determine "redundant data".

It is clear to me from Michael's note that I inter-mixed my examples
and my statements of principle too much.  The term "redundant data"
should have been used in the context of my examples only.  What I
was advocating was the transport layer policing "junk data"
("useless data", "data never delivered to the recieving
application"... pick your favorite).  In the case of PR-SCTP the
transport knows what information this is by what the application
tells the transport and what the transport knows about which bits of
data have been lost, retransmitted, not retransmitted, etc.

I *was not* advocating any sort of new communication between
transport and application.  And, I *was not* advocating that the
transport should try to mine extra information from the application
data passed down.

If the policing scheme I was suggesting is a layer viloation then so
is PR-SCTP (which I do not believe is the case).

allman