Subject: Re: [P2PSIP] Solution space for fragmentation, congestion control and reliability

From: Dean Willis <[email-masked]>
Date: Wed, 1 Apr 2009 13:02:17 -0500

On Mar 26, 2009, at 10:53 PM, Salman Abdul Baset wrote:

>
>> Your alternate solution is an assumption that the overlay will  
>> contain
>> enough TCP open-access nodes that are able to identify themselves as
>> open-access nodes and willing to bear the burden of their role on
>> behalf of all the other peers.   What I've seen is that UDP actually
>> works, and TCP doesn't.
>
> Skype works. It has enough nodes that can provide relay services.

The Skype operators apparently pay to put their supernodes into places  
from which they can provide relay services. This isn't inexpensive.

You have funding to support RELOAD supernodes?

--
Dean

---

From: Salman Abdul Baset <[email-masked]>
Date: Wed, 1 Apr 2009 16:06:35 -0400 (EDT)

On Wed, 1 Apr 2009, Dean Willis wrote:

>>> Your alternate solution is an assumption that the overlay will contain
> The Skype operators apparently pay to put their supernodes into places from 
> which they can provide relay services. This isn't inexpensive.
>

I don't. I am a Ph.D. student -:)

Just to be clear, I was refering to the relaying of RELOAD messages and 
not the media. An insight from our Skype analysis is that the Skype 
application fails to join the Skype network if it cannot establish a TCP 
connection with any Skype super node.

Nevertheless, relaying of p2p+signaling+media raises some important 
issues:

1) Why would Skype pay to put their super nodes into places where they
could provide relay services?

2) How do we know if these 'paid' nodes are being heavily or sparingly 
used as media relays?

For (1)
Skype would do that because they do not want calls to fail if they are 
unable to find an altruistic Skype node that can relay media. That is just 
a smart of designing a system. Failed calls = bad publicity.

For (2)
Clearly, Skype knows this. However, since we do not have this information, 
the next best thing is to perform measurements. We performed experiments 
where two Skype nodes were forced to select a relay for about ~18,000 
calls and we found 9,584 unique media relays. About 46% of calls were 
relayed through nodes in educational institutions. These two statistics 
seem to indicate that the likelihood of paid Skype nodes being heavily 
used as relays is fairly low. Meaning, that Skype has no shortage of 
altruistic media relays.

(See slide 8)
http://www1.cs.columbia.edu/~salman/presentations/skyperelay-gi08.ppt

and
http://www1.cs.columbia.edu/~salman/publications/skyperelay-gi08.pdf

I wonder how much money Skype is paying to operators for relay services.
My guess is that it should be significantly smaller than say what MSN 
messenger has to pay.

Regards,
Salman

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 1 Apr 2009 13:10:50 -0700

On Wed, Apr 1, 2009 at 1:06 PM, Salman  Abdul Baset <[email-masked]> wrote:

> where two Skype nodes were forced to select a relay for about ~18,000 calls
> and we found 9,584 unique media relays. About 46% of calls were relayed
> through nodes in educational institutions. These two statistics seem to
> indicate that the likelihood of paid Skype nodes being heavily used as
> relays is fairly low. Meaning, that Skype has no shortage of altruistic
> media relays.

It's important to recognize one very significant difference between
RELOAD and Skype:
Skype controls the software you run and so can make you act as a
supernode--or at
least make it fairly difficult to opt out. By contrast, RELOAD will
presumably be open
software (at least the protocols will be open) so you cannot force
users to run relays.

-Ekr

---

From: Salman Abdul Baset <[email-masked]>
Date: Wed, 1 Apr 2009 16:27:41 -0400 (EDT)

> software (at least the protocols will be open) so you cannot force

Skype does make an active effort to avoid media relaying. These include 
techniques like ICE-UDP and media-over-TCP for UDP-blocking firewalls. 
So in terms of designing protocols for directly sending media, Skype and 
RELOAD are no different.

Starting with v3.2, Skype provides a mechanism to turn off the super node 
feature. See this:
http://www.skype.com/security/universities/

-s

---

From: Enrico Marocco <[email-masked]>
Date: Wed, 1 Apr 2009 23:18:24 +0200

Eric Rescorla wrote:
> RELOAD and Skype: Skype controls the software you run and so can make
> you act as a supernode--or at least make it fairly difficult to opt
> out. By contrast, RELOAD will presumably be open software (at least
> the protocols will be open) so you cannot force users to run relays.

Well, most BitTorrent clients are open source software and users have
full control of the resources they share, but that doesn't prevent them
to share often more than what they consume. Of course file-sharing and
realtime communications are quite different beasts, but I don't think
that if Skype allowed users to turn the local relay off, many would
actually do that.

-- 
Ciao,
Enrico

---

From: Bruce Lowekamp <[email-masked]>
Date: Wed, 1 Apr 2009 19:55:02 -0400

I really feel like this conversation has diverged from any useful
purpose.  The first question we need to answer is:

- Do we need to support peers behind NATs?

Based on the charter and the concepts draft, this is true.  If you
wish to contest this, please start a new thread to revisit this
question in the concepts draft and charter.

2nd question: Do we need to have direct connections between peers most
of the time, or can we rely on relays most of the time?

This is a use case or application scenario question.  Unfortunately,
the group never adopted a draft, although there were several drafts
that listed various scenarios.  However, if you want believe p2psip
work should only address use cases that have plenty of relays
available, this should still be in a conversation thread labeled for
that purpose and probably intended to update the concepts draft.

3rd question: What protocols work in scenarios with peers behind NATs
trying to establish direct connections.

If you believe this is anything other than TCP when possible, but UDP
in an awful lot of cases, please direct your effort toward making
ICE-TCP work in MMUSIC, and refer to discussions in MMUSIC for
experiences with other solutions.

My reading of the current charter, concepts draft, and state of the
art for NAT traversal makes me believe that the RELOAD must define a
UDP-based overlay link protocol.  I hope we can discuss solutions to
this problem.  But if there is agreement that the requirements are
different or need to be changed, let's be explicit in what question
we're asking and what draft(s) need(s) to be changed.  Those aren't
RELOAD questions.

Bruce

---

From: Salman Abdul Baset <[email-masked]>
Date: Wed, 1 Apr 2009 23:11:26 -0400 (EDT)

Assuming TCP-over-UDP is needed, my understanding from the list discussion 
and from the meeting is that our options are:

1) Design TCP-over-UDP in TSV. RELOAD refers to it.
2) If that may not be possible, then
  a) Design TCP-over-UDP as a seperate draft in P2PSIP WG. RELOAD refers to it.
  b) Define TCP-over-UDP in the base document. A custom protocol that only
     RELOAD can use.

Now, what should be the performance requirements of TCP-over-UDP as 
compared to regular TCP?

-Clearly, it should not exceed the performance of TCP.
-It should not *only* rely on timeouts to recover losses. Otherwise, the 
performance is horrible.
-It should not assume anything about the number of fragments.

Do you think (1-3) are our only options for a TCP-over-UDP solution or 
did I miss something? Do you agree with the performance requirements of a 
TCP-over-UDP solution?

If TCP-over-UDP is the solution, I prefer (1) or (2-a) as they are less 
likely to stall progress on the RELOAD base document.

Now, the tricky question :)

> we're asking and what draft(s) need(s) to be changed.  Those aren't

I am totally willing to believe what you are saying but I want to arrive 
at the conclusion in a more systematic manner. Specifically, I am looking 
for a paper or a draft, that will exactly list the problems with the 
existing approaches (including Characterizing paper), and conclude that 
these approaches don't work or using these approaches is much expensive 
than a TCP-over-UDP solution. However, I am not able to find such a paper, 
or a draft in the BEHAVE WG. If there was a mailing list discussion, it 
will be helpful to have it in a draft form.

Regards,
Salman

---

From: Bruce Lowekamp <[email-masked]>
Date: Sun, 5 Apr 2009 23:07:27 -0400

Salman,

We're not doing TCP over UDP.  If you want to do that, then please by
all means go do it in TSVWG.

We have the option of simpy saying "use TFRC."  That will be good
enough performance, and require relatively little specification since
TSV has already put a lot of work into it.  It's also a bit
complicated.  A lot more complicated than is really needed for most
p2psip implementations/deployments.

So the motivation of the other options was to provide simpler options
that are going to provide enough performance for many/most
deployments.  You seem very focused on thinking that they're trying to
duplicate TCP.  They're not.  They are trying to be something that a
fairly typical development team can build and get right, and get
enough performance for their deployment scenarios to work.  They are
addressing congestion and reliability just like *every* UDP protocol
needs to.

If they need TCP-level performance out of UDP, they can implement
TFRC.  We can debate which features are needed or useful for
performance without being too complicated for a long time, but that
doesn't change the basic focus of what we're doing.

(I think there may need to be a tweak in the receiver algorithm as
speced now to let TFRCs reach full performance, but the intention is
certainly to make sure TFRC implementations have the information they
need.)

Bruce

On Wed, Apr 1, 2009 at 11:11 PM, Salman  Abdul Baset
<[email-masked]> wrote:
> Now, what should be the performance requirements of TCP-over-UDP as compared
> to regular TCP?
>
> Do you think (1-3) are our only options for a TCP-over-UDP solution or did I
> miss something? Do you agree with the performance requirements of a
> TCP-over-UDP solution?
>
> I am totally willing to believe what you are saying but I want to arrive at
> the conclusion in a more systematic manner. Specifically, I am looking for a
> paper or a draft, that will exactly list the problems with the existing
> approaches (including Characterizing paper), and conclude that these
> approaches don't work or using these approaches is much expensive than a
> TCP-over-UDP solution. However, I am not able to find such a paper, or a
> draft in the BEHAVE WG. If there was a mailing list discussion, it will be
> helpful to have it in a draft form.
>
>
> Regards,
> Salman
>

---

From: Salman Abdul Baset <[email-masked]>
Date: Mon, 6 Apr 2009 00:09:32 -0400 (EDT)

I am worried about the situation in which packets traversing peers 
that run TCP have to be sent to a peer behind NAT using RELOAD's custom 
transport and cc protocol. Some unexpected failures may happen if this 
reliable and cc protocol is not properly designed.

In the light of WG discussion, the questions I mentioned before need to be 
addressed before a full-blown solution, i.e.,

What are the performance requirements of this protocol?

Where should this protocol be specified?

-s

On Sun, 5 Apr 2009, Bruce Lowekamp wrote:

> _______________________________________________
> P2PSIP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/p2psip

---

From: Lars Eggert <[email-masked]>
Date: Mon, 6 Apr 2009 12:20:15 +0300

Hi,

On 2009-4-6, at 6:07, Bruce Lowekamp wrote:
> deployments.

I'd strongly urge you to use TFRC rather than rolling your own scheme.  
Don't underestimate the validation effort that is required to ensure  
that a congestion control scheme is safe to deploy. This has all been  
done for TFRC, and it must be done for any new scheme.

Lars

---

From: "Brian Rosen" <[email-masked]>
Date: Mon, 6 Apr 2009 09:18:18 -0400

Is "use TCP when it works, and TRFC when it doesn't" an answer?
Arguments like "it's too complex" don't work for me when we're talking
transport protocols that have to do congestion control, etc.  Congestion
control is complex.

Brian

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Lars Eggert
Sent: Monday, April 06, 2009 5:20 AM
To: Bruce Lowekamp
Cc: Salman Abdul Baset; [email-masked]
Subject: Re: [P2PSIP] Solution space for fragmentation, congestion control
and reliability

Hi,

I'd strongly urge you to use TFRC rather than rolling your own scheme.  
Don't underestimate the validation effort that is required to ensure that a
congestion control scheme is safe to deploy. This has all been done for
TFRC, and it must be done for any new scheme.

Lars

---

From: Bruce Lowekamp <[email-masked]>
Date: Mon, 6 Apr 2009 13:15:27 -0400

Yes, that's a valid option.  Probably the right one at this point.  In
fact, the solution space as I last sent it out looks like:

- stop and wait
- simplified AIMD
- TFRC
- (TCP over UDP might go here if a draft existed)
- TCP

The argument has mostly been about the simplified AIMD.  I kind of
hate to lose it, but you're right, it will be a lot less controversial
if we do.  Actually, TFRC is a pretty flexible protocol itself, so we
can probably just do something a bit more within that framework if we
want to have options.

stop and wait was always intended as maybe a development-type
protocol, not a real deployable protocol (although it would work OK in
a small office environment)

Bruce

---

From: Saverio Mascolo <[email-masked]>
Date: Mon, 6 Apr 2009 19:35:44 +0200

hi lars,

> TFRC, and it must be done for any new scheme.

is anyone  aware of applications using TFRC?

s

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 6 Apr 2009 13:50:42 -0400

I will note that TFRC was designed for long-lived streams (e.g., media  
streams), which probably differs a bit from our typical application in  
P2PSIP, where somewhat random pairs of nodes exchange data objects of  
a few dozen to a few hundred packets. For example, establishing a  
"fair" rate is not trivial under those conditions, and has high error  
margins.

Henning

On Apr 6, 2009, at 1:35 PM, Saverio Mascolo wrote:

> I'd strongly urge you to use TFRC rather than rolling your own  
> scheme. Don't underestimate the validation effort that is required  
> to ensure that a congestion control scheme is safe to deploy. This  
> has all been done for TFRC, and it must be done for any new scheme.
>
> Lars
>

---

From: "Brian Rosen" <[email-masked]>
Date: Mon, 6 Apr 2009 13:54:04 -0400

Is that a fatal flaw, or merely an expression of concern, needing some text?

Brian

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Henning Schulzrinne
Sent: Monday, April 06, 2009 1:51 PM
To: Saverio Mascolo
Cc: [email-masked]
Subject: Re: [P2PSIP] Solution space for fragmentation, congestion control
and reliability

I will note that TFRC was designed for long-lived streams (e.g., media
streams), which probably differs a bit from our typical application in
P2PSIP, where somewhat random pairs of nodes exchange data objects of a few
dozen to a few hundred packets. For example, establishing a "fair" rate is
not trivial under those conditions, and has high error margins.

Henning

hi lars,

I'd strongly urge you to use TFRC rather than rolling your own scheme. Don't
underestimate the validation effort that is required to ensure that a
congestion control scheme is safe to deploy. This has all been done for
TFRC, and it must be done for any new scheme.

Lars

s

---

From: Salman Abdul Baset <[email-masked]>
Date: Mon, 6 Apr 2009 14:02:43 -0400 (EDT)

TFRC is a congestion control protocol. What is intended is a reliable 
congestion control protocol over UDP, which does not exist as a 
specification. Reliability cannot merely be sprinkled on top of TFRC or 
any congestion control protocol.

-s

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 6 Apr 2009 17:00:25 -0400

It's not fatal, but it will essentially mean that TFRC will stay at  
the initial (RFC 3390) rate for almost all transfers. The problem is  
that TFRC is based on estimating loss rates, based on 8 loss  
intervals. If you have a loss rate of 1%, this translates, very  
roughly, into 800 packets or roughly 800 kB, before you can reliably  
estimate a better (higher) rate. Lower rates will kick in sooner,  
since you'll get the loss intervals more quickly.

I have to admit I'm confused as to why TFRC is seen as attractive for  
this application. The main benefit, smoothing rates, is of no  
importance for this application, and TFRC is significantly more  
complicated than the TCP mechanism. RFC 5348 (TFRC) only specifies a  
mechanism, so we still would have to engineer a new transport  
protocol, including reliability and the bits necessary to do round- 
trip time estimation (a necessary ingredient for TFRC as well as more  
classical TCP).

Thus, saying "just use TFRC" doesn't avoid the "design transport"  
issue - it just provides guidance on the congestion control algorithm,  
and probably is not even the right answer for our application.

Henning

On Apr 6, 2009, at 1:54 PM, Brian Rosen wrote:

> scheme. Don't

---

From: Bruce Lowekamp <[email-masked]>
Date: Mon, 6 Apr 2009 22:46:02 -0400

Fortunately, reliability is not important for the overlay link
protocol.  It's a link layer.

Bruce

On Mon, Apr 6, 2009 at 2:02 PM, Salman Abdul Baset <[email-masked]> wrote:
> specification. Reliability cannot merely be sprinkled on top of TFRC or any
> congestion control protocol.
>
> -s
>

---

From: Bruce Lowekamp <[email-masked]>
Date: Mon, 6 Apr 2009 23:14:44 -0400

Henning is right in that TFRC is intended for streaming protocols.  My
understanding is that there has been some consensus at applying it to
other protocols, but it is a bit of a stretch.  I'm not sure I agree
with Henning's analysis, though I haven't spent enough time with it to
fully understand all the implications.  Didn't 5348 address the issue
with data-limited senders and bursty data?

Here is literally what 5405 says:

------------
   Applications that perform bulk transmission of data to a peer over
   UDP, i.e., applications that exchange more than a small number of UDP
   datagrams per RTT, SHOULD implement TCP-Friendly Rate Control (TFRC)
   [RFC5348], window-based, TCP-like congestion control, or otherwise
   ensure that the application complies with the congestion control
   principles.
--------------

I'm a bit unclear what would be considered acceptable under the
"window-based, TCP-like congestion control" part of that statement.  I
would think that AIMD window-based congestion control would be
sufficient.

Bruce

On Mon, Apr 6, 2009 at 5:00 PM, Henning Schulzrinne <[email-masked]> wrote:
> It's not fatal, but it will essentially mean that TFRC will stay at the
> initial (RFC 3390) rate for almost all transfers. The problem is that TFRC
> is based on estimating loss rates, based on 8 loss intervals. If you have a
> loss rate of 1%, this translates, very roughly, into 800 packets or roughly
> 800 kB, before you can reliably estimate a better (higher) rate. Lower rates
> will kick in sooner, since you'll get the loss intervals more quickly.
>
> I have to admit I'm confused as to why TFRC is seen as attractive for this
> application. The main benefit, smoothing rates, is of no importance for this
> application, and TFRC is significantly more complicated than the TCP
> mechanism. RFC 5348 (TFRC) only specifies a mechanism, so we still would
> have to engineer a new transport protocol, including reliability and the
> bits necessary to do round-trip time estimation (a necessary ingredient for
> TFRC as well as more classical TCP).
>
> Thus, saying "just use TFRC" doesn't avoid the "design transport" issue - it
> just provides guidance on the congestion control algorithm, and probably is
> not even the right answer for our application.
>
> Henning
>
>> Don't

---

From: Salman Abdul Baset <[email-masked]>
Date: Mon, 6 Apr 2009 23:16:44 -0400 (EDT)

Are you saying hop-by-hop reliability is not needed? A received message 
is useless unless all fragments are reliably received.

If indeed hop-by-hop reliability is needed, then the link layer analogy is 
unfortunately not right.

-s

---

From: Saverio Mascolo <[email-masked]>
Date: Tue, 7 Apr 2009 14:03:24 +0200

hi bruce,

On Tue, Apr 7, 2009 at 5:14 AM, Bruce Lowekamp <[email-masked]> wrote:

> Henning is right in that TFRC is intended for streaming protocols.

why tcp is not good for streaming?  youtube i using TCP.

do you mean tfrc for real-time?

saverio

>  My
> this
> >> From: [email-masked] [mailto:[email-masked]] On
> Behalf
> >> Of
> >> Henning Schulzrinne

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 7 Apr 2009 10:00:31 -0400

"TCP-like" is presumably something like the Reno, Tahoe, Vegas or  
similar variations of AIMD implemented by TCP.

I don't think you advocated this, but implementing both TFRC and AIMD  
(running at the same time, not alternatives as in DCCP) seems  
difficult and far more complicated.

The issue of data-limited senders has nothing to do with the start-up  
issue I mentioned in my message. The inherent problem of TFRC is that  
it relies on estimating a loss rate, and this, by its very nature,  
takes a while unless loss are very high (which you obviously don't  
want). This isn't a problem in steady state for long-lived flows, such  
as a video stream, but P2PSIP does not match this model.

I agree that an AIMD mechanism (of which TCP is an instance) is  
probably the better choice for this application. TFRC is neither  
simpler nor "better" than AIMD, it just fits the unreliable transport  
model better (again, which is not what we need here). After all, TFRC  
is attempting to model AIMD behavior.

However, using AIMD means implementing the same thing as "TCP over  
UDP", with maybe the header bits re-arranged, as AIMD needs much of  
the TCP header information, for example.

This also has nothing to do with whether TCP can be used for real-time  
data, which is probably a topic of interest to working groups other  
than P2PSIP. Salman and I, with others, have a paper on this topic in  
SIGMETRICS, if you happen to care about this topic.

Henning

On Apr 6, 2009, at 11:14 PM, Bruce Lowekamp wrote:

> On Mon, Apr 6, 2009 at 5:00 PM, Henning Schulzrinne <[email-masked] 
> > wrote:
>> the
>> issue - it
>>> Is that a fatal flaw, or merely an expression of concern, needing  
>>> some
>>> text?
>>>
>>> Brian
>>>
>>>
>>>
>>> P2PSIP, where somewhat random pairs of nodes exchange data objects  
>>> of a
>>> few
>>> scheme.

---

From: Bruce Lowekamp <[email-masked]>
Date: Tue, 7 Apr 2009 22:09:06 -0400

I think that if messages are typically only a few (<= 3) fragments, we
can safely ignore this issue.  If the loss rate is high enough that it
matters, the network will collapse anyway.  Also remember that what
happens is there's a retransmission of all of the fragments, which
might waste some traffic, but the receiver can still have the other
fragments waiting, so they're not completely useless.

OTOH, if messages are going to typically be even larger (or need to
deal with higher loss rates), I would favor fixing the problem
end-to-end, not through hop-by-hop reliability.   I hesitated adding
the minimal semi-reliability to the proposal I wrote up.  It's just a
lot of complexity and latency added for something that I'm not sure is
a good addition to the overall stability of the network.

It's important to be willing to drop traffic mid-network.  That's the
only way the network can shed load.  The senders will back off.
(Currently they back off in a per-transaction question.  It's an
interesting question whether they should do more.)

Bruce

On Mon, Apr 6, 2009 at 11:16 PM, Salman  Abdul Baset
<[email-masked]> wrote:
> Are you saying hop-by-hop reliability is not needed? A received message is
> useless unless all fragments are reliably received.
>
>>>>> that
>>>>> a

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 7 Apr 2009 22:54:12 -0400

The problem is that this will collapse (say) 10 times sooner than in a  
non-P2P network (assuming 10 hops).

Plus, getting the round-trip time estimation right is going to be that  
much harder the more hops there are, leading to spurious  
retransmission. After all, each hop is likely, on average, to transit  
a fair chunk of the Internet if the P2P network is global. With hop-by- 
hop RTT estimation, you at least have a chance to re-use the same set  
of "links" reasonably frequently, e.g., to finger table neighbors, so  
that RTT estimates are at least plausible. Each end-to-end path will  
be one-off, so that the RTT can only be worst-case, and that has to be  
significantly larger than the initial TCP estimate, given that you're  
traversing good spans of the Internet several (but unknown number of)  
times.

Betting that all messages are small seems like a dangerous gamble.  We  
lost that one before. (Not just SIP - who would have thought that DNS  
messages could become fairly substantial?)

Dropping messages mid-stream is a great way to design a protocol that  
doesn't meet the reliability requirements that "competition" with  
client-server mechanisms requires, and makes diagnosis really hard, as  
you could have failures that depend on who is asking the question, and  
other random behavior (e.g., which replica is being asked for). P2P  
systems are hard enough to debug without adding more obscure failure  
modes.

My opinion is to either to do it right or not to do it at all. Trying  
to cut corners now, to show how "simple" designing transport is, will  
just lead to regret later. We've been there, and some of us don't care  
to re-visit that place.

In general, we're in great danger of designing a protocol that is far  
more complicated than needed for the simple URL-to-IP address mapping  
(you don't need "kinds" and customizable policies for that...), yet  
insufficient to handle anything more demanding.

Henning

On Apr 7, 2009, at 10:09 PM, Bruce Lowekamp wrote:

>>> On Mon, Apr 6, 2009 at 2:02 PM, Salman Abdul Baset <[email-masked] 
>>> >
>>> wrote:
>>>>
>>>>>> Subject: Re: [P2PSIP] Solution space for fragmentation,  
>>>>>> congestion
>>>>>> control
>>>>>> and reliability
>>>>>>
>>>>>> Hi,
>>>>>>

---

From: Bruce Lowekamp <[email-masked]>
Date: Wed, 8 Apr 2009 06:21:37 -0400

One of the (if not THE) fundamental decisions that makes the Internet
work is that reliability is end-to-end, not hop-by-hop.  Hops are
allowed to drop traffic due to congestion, and that is taken as
implicit feedback that there is congestion in the network.

The proposal I sent out drops traffic in two places.  First, each peer
maintains a limited queue for fragments and drops excess fragments
(the literal analog of drops due to congestion in routers).  Second,
the overlay link protocol is only semi-reliable (which I debated
having at all), with the assumption that on the Internet, loss is due
to congestion.  So we have two congestion signals to the peer, queue
length and link protocol drops.

Congestion needs to cause load shedding in a network (overlay or not)
or else it will collapse.  Arguing that we should do extra work for
hop-by-hop reliability makes no sense to me.

Bruce

On Tue, Apr 7, 2009 at 10:54 PM, Henning Schulzrinne
<[email-masked]> wrote:
> Plus, getting the round-trip time estimation right is going to be that much
> harder the more hops there are, leading to spurious retransmission. After
> all, each hop is likely, on average, to transit a fair chunk of the Internet
> if the P2P network is global. With hop-by-hop RTT estimation, you at least
> have a chance to re-use the same set of "links" reasonably frequently, e.g.,
> to finger table neighbors, so that RTT estimates are at least plausible.
> Each end-to-end path will be one-off, so that the RTT can only be
> worst-case, and that has to be significantly larger than the initial TCP
> estimate, given that you're traversing good spans of the Internet several
> (but unknown number of) times.
>
> Betting that all messages are small seems like a dangerous gamble.  We lost
> that one before. (Not just SIP - who would have thought that DNS messages
> could become fairly substantial?)
>
> client-server mechanisms requires, and makes diagnosis really hard, as you
> could have failures that depend on who is asking the question, and other
> random behavior (e.g., which replica is being asked for). P2P systems are
> hard enough to debug without adding more obscure failure modes.
>
> My opinion is to either to do it right or not to do it at all. Trying to cut
> corners now, to show how "simple" designing transport is, will just lead to
> regret later. We've been there, and some of us don't care to re-visit that
> place.
>
> In general, we're in great danger of designing a protocol that is far more
> complicated than needed for the simple URL-to-IP address mapping (you don't
> need "kinds" and customizable policies for that...), yet insufficient to
> handle anything more demanding.
>
> Henning
>
>>> is

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 8 Apr 2009 08:27:29 -0400

> implicit feedback that there is congestion in the network.

You may want to re-read the end-to-end argument paper. You do need end- 
to-end reliability to ensure reliability (since nodes in the middle  
can do bad things to packets), but performance often dictates "hop-by- 
hop", particularly if the "hop" is essentially the whole Internet.  
Since you like to use the notion of a link layer in P2P: this is the  
reason why 802.11 has link-layer reliability. By your argument, 802.11  
should dispense with that.

> length and link protocol drops.

Unfortunately, that assumption is only somewhat true, particularly  
with wireless links.

> hop-by-hop reliability makes no sense to me.

Please consult the literature; this is a topic that has received more  
than its share of real measurement work, albeit ten and twenty years  
ago, when this topic was of greater practical interest.

---

From: Bruce Lowekamp <[email-masked]>
Date: Wed, 8 Apr 2009 22:49:55 -0400

802.11 is a bad example.  It needs reliability because its losses
aren't well-correlated with congestion.

I'm not terribly concerned about TCP having reliability when used in
this context, although the stream-oriented nature of TCP is
unfortunate.  I don't want to go to a lot of effort adding reliability
to a UDP-based protocol, because I don't think it will add much (and
could hurt if done wrong).

Even if the overlay link protocol does have reliability, you still
need to shed load with finite-length queues (or push back) to avoid
congestion collapse/infinite queues.  I'm curious what literature
you're thinking off that argues otherwise.

Bruce

On Wed, Apr 8, 2009 at 8:27 AM, Henning Schulzrinne <[email-masked]> wrote:
> You may want to re-read the end-to-end argument paper. You do need
> end-to-end reliability to ensure reliability (since nodes in the middle can
> do bad things to packets), but performance often dictates "hop-by-hop",
> particularly if the "hop" is essentially the whole Internet. Since you like
> to use the notion of a link layer in P2P: this is the reason why 802.11 has
> link-layer reliability. By your argument, 802.11 should dispense with that.
>
>
>
>>
>>
> Unfortunately, that assumption is only somewhat true, particularly with
> wireless links.
>
>
>>
>>
> Please consult the literature; this is a topic that has received more than
> its share of real measurement work, albeit ten and twenty years ago, when
> this topic was of greater practical interest.
>
>>
>>
>> Bruce
>>
>
>

---

From: Enrico Marocco <[email-masked]>
Date: Thu, 9 Apr 2009 12:04:41 +0200

> aren't well-correlated with congestion.

I do not expect network congestion to be the main cause of packet losses
in distributed overlays either. Factors like peer failures and link
instability in wireless/ad-hoc networks will probably have greater
impact, no?

> you're thinking off that argues otherwise.

The fact that hop-by-hop reliability cannot replace end-to-end -- and I
don't think that's what Henning was arguing for -- doesn't mean that the
contrary is always true. It happens to be a good approximation in some
scenarios, e.g. when the hops are very stable hardware routers and the
links wires, but in the general case it is an oversimplistic assumption.

-- 
Ciao,
Enrico

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 9 Apr 2009 08:20:43 -0400

> aren't well-correlated with congestion.

I admit that I can't follow this logic. The reason for reliability in  
802.11 has nothing to do with congestion. If 802.11 did not have  
reliability built-in, end-to-end throughput at the network layer would  
be horrible. The same principle applies here as well.

> could hurt if done wrong).

Can you support your arguments with references to the literature or at  
least not contradict standard intro to networking course materials?

> you're thinking off that argues otherwise.

Yes, you do need (and will, by nature, have) finite buffers at  
intermediate hops, but they will drop P2P-layer messages very rarely,  
as long as they can store a full "message" worth.

---

From: Bruce Lowekamp <[email-masked]>
Date: Thu, 9 Apr 2009 13:47:55 -0400

On Thu, Apr 9, 2009 at 8:20 AM, Henning Schulzrinne <[email-masked]> wrote:
> I admit that I can't follow this logic. The reason for reliability in 802.11
> has nothing to do with congestion. If 802.11 did not have reliability
> built-in, end-to-end throughput at the network layer would be horrible. The
> same principle applies here as well.

I'm not sure how we are talking past each other here.  The basic
principle of how TCP works is that loss==congestion.  802.11 needs a
reliability protocol because it doesn't have this correlation, i.e.
loss can be caused by signal problems that are not related to
congestion.

The principle I was trying to apply a few messages ago was that if
loss in the overlay link layer is caused by congestion, and end-to-end
protocols react to loss by backing off, we have the same property as
TCP.  Your suggestion that if 802.11 needing reliability implies that
an overlay link protocol only makes sense if you believe that the
overlay link protocol experiences losses that are not due to
congestion.  I don't believe that is likely to be true, so 802.11 is a
bad example.

> Can you support your arguments with references to the literature or at least
> not contradict standard intro to networking course materials?

I have no idea what you're referring to here.  But since you have
claimed I am misapplying/unaware of the end-to-end argument, etc, I
will quote what I think is the most relevant part of Saltzer's paper
here:

-----------------
Clearly, some effort at the lower levels to improve network
reliability can have a
significant effect on application performance. But the key idea here
is that the lower levels need
not provide "perfect" reliability.

Thus the amount of effort to put into reliability measures within the
data communication system
is seen to be an engineering tradeoff based on performance, rather
than a requirement for
correctness. Note that performance has several aspects here.
---------------------

Obviously the paper goes on to discuss more aspects of the corner
cases and performance tradeoffs, but my initial suggestion that what
we should build is a semi-reliable protocol is entirely in line with
this, as far as I can tell.

Bruce

---

From: Enrico Marocco <[email-masked]>
Date: Fri, 10 Apr 2009 00:36:32 +0200

> congestion.  I don't believe that is likely to be true, so 802.11 is a

Uh?? In a ring based DHT with 1000 nodes, failure detection time of 1
minute, if peers fail on average every 60 minutes every query has a 1 -
(1 - 1 / 60) ^ (log2(1000) / 2) =~ 8% probability to find a failed hop
on its way. YMMV, but that's hardly irrelevant, especially if compared
to the inherent packet loss rate of 802.11 (which, IIRC, should be less
than 10%).

-- 
Ciao,
Enrico

---

From: Bruce Lowekamp <[email-masked]>
Date: Thu, 9 Apr 2009 19:03:22 -0400

Enrico,

My comment was that the reason 802.11 has reliability isn't the same
as the reason we may or may not need reliability in an overlay link
protocol.  802.11 is irrelevant, the loss characteristics of overlays
is not.

But in your example, the overlay link protocol having reliability
doesn't solve the problem either, if your application is real-time
communication.  If the failure detection time is 1 minute, 8% of your
queries are going to take an average of 30 seconds to complete
(assuming they succeed on the second try).  So you need to rely on
another mechanism to try to complete the query more quickly.  Whether
that is each hop replacing the failing peer in their routing table
before they officially fail it and stop trying to re-establish (after
1 minute) or the querying peer trying alternative routes or replicas,
you need to find an alternative route faster.

Bruce

---

From: Enrico Marocco <[email-masked]>
Date: Fri, 10 Apr 2009 13:35:33 +0200

Bruce Lowekamp wrote:
> protocol.

The reason why you may want to consider hop-by-hop reliability in RELOAD
is performance, just as in 802.11.

> you need to find an alternative route faster.

Or each hop forking a parallel route right after a few retransmissions
of the same packet; that would take much longer if retransmissions were
end-to-end (i.e. rather than constant, the time overhead would grow
linearly with the _estimated_ path length).

-- 
Ciao,
Enrico

---

From: Bruce Lowekamp <[email-masked]>
Date: Sun, 12 Apr 2009 22:25:40 -0400

> is performance, just as in 802.11.

Sure, everything ultimately comes down to performance.  But that's
reduction to the absurd.  Losses in wired networks mean there is
congestion and the senders should slow down.  Losses in wireless
networks don't necessarily mean anything.

There are certainly good reasons to consider overlay link level
retransmissions.  (IMHO the best is that most sessions are short, so
the sender backing off means little.)  But the issues need to be taken
beyond "802.11 has it, so we should too."

> linearly with the _estimated_ path length).

We've been avoiding having actual parallel forking used in the base
draft.  What I think we could include is a simple approach that if A
is doing link-level retrans to B and believes it has failed, A should
replace B in its routing table with the next-best match and let the
fragment be retransed to the new entry.  Meanwhile, continue trying to
re-establish with B and bring it back if it's still alive.

However, this type of approach would be incompatible with any pure tcp
or tcp-over-udp overlay link layer.

Bruce

---

From: Bruce Lowekamp <[email-masked]>
Date: Sun, 12 Apr 2009 22:53:56 -0400

> or tcp-over-udp overlay link layer.

Actually, I retract that last statement.  We already have
acknowledgements over tcp since its timeout is too high, so as long as
that value is equal to the point where you want to fallback to the
next best entry for the routing table, it's fine.

Bruce