Subject: Re: [rmcat] 5 tuples and rmcat-cc-requirements-01

From: Harald Alvestrand <[email-masked]>
Date: Fri, 03 Jan 2014 01:19:31 +0100

On 12/31/2013 07:37 AM, Dave Taht wrote:
> On Mon, Dec 30, 2013 at 6:09 AM, Harald Alvestrand <[email-masked]> wrote:
>> Dave, this is interesting .... I think characterizing what the connection
>> between type-of-traffic and applicable congestion is should be an rmcat
>> issue, but may feed back into RTCWEB and even MMUSIC - so I'm sending this
>> to rmcat only.
>>
>>
>>
>> On 12/30/2013 01:02 AM, Dave Taht wrote:
>>> I finally had a chance to read over
>>>
>>> http://tools.ietf.org/html/draft-ietf-rmcat-cc-requirements-01
>>>
>>> and also work at reproducing some of the difficulties present in the
>>> current implementation of chrome in light of the two papers presented
>>> at the last ietf on google congestion control. I went through tons of
>>> tests and packet captures that pretty much reproduced the legion of
>>> problems documented in
>>>
>>>
>>> https://speakerdeck.com/vr000m/evaluating-googles-congestion-control-for-webrtc
>>> and the other paper presented at ietf...
>>>
>>> and also fed webrtc traffic through the fq_codel and pie queue
>>> management systems (with pretty good results). I have tons of captures
>>> from each experiment if anyone wants them.
>>>
>>> So, I have some comments on this requirements document. (I'm new to
>>> paying attention on rmcat, so a few of my questions are probably
>>> answered by some document I haven't read yet. Steers appreciated. for
>>> all I know I'm on the wrong mailing lists too. Sorry)
>>>
>>> In particular, the flows I was looking at multiplexed voice and video
>>> on the same 5 tuple. I am curious as to the justification for this, as
>>> voice (without silence suppression) is isochronous and video is not.
>>> (is silence suppression used in webrtc?)
>>
>> This is a curious statement - what characteristic are you thinking about
>> when you say that voice is "isochronous" and video is not?
>> Both of them are produced one time-frame at a time, and both of them cause
>> irritation at the receiver when the frames are displayed in anything but the
>> same cadence.
> On a per packet level, voice is typically one packet ranging in size
> from 110 bytes to 300 every 10 or 20ms. (the opus codec goes down to
> 2.5ms) Video, in comparison, consists of a variable number of (often)
> many packets depending on encoding, subframes, etc.
>
> So from this viewpoint voice can be isochronous (lacking silence
> suppression), and video is not.

OK, so synchronous production of single packets is isochronous, while 
synchronous production of group of packets is not. I don't mind, as long 
as we agree what we're saying :-)

>
>> I think the irritation from jitter on voice is quite a bit higher than on
>> video, but stuttering video isn't a good thing either.
> jitter and loss on voice is often quite annoying. On music it's intolerable.
>
> On video, I would not use the word "stuttering" which implies a
> max-headroom sort of effect. Partial corruption of the output, blocky
> output, etc are closer to words I would use.

Yup, those are more irritating artifacts in video than distortions in time.
In fact, most of the time, the most irritating time-related aspect of 
video is most often loss of synchronization with the audio.

>
>>> To me, multiplexing these two very different flows on the same ports
>>> is not a good idea except under extreme port pressure, and not useful
>>> in ipv6 or in non-natted environments at all. Even in a natted
>>> environment, saving a single port is a false savings as (for example)
>>> dns lookups punch dozens of holes through nat on a regular basis.
>>>
>>> Requiring that in most or all cases these two flow types be on unique
>>> 5 tuples would give smarter queue management techniques on routers
>>> (like sfq, or sqf (deployed at FT), or fq_codel (deployed at
>>> free.fr)), a fighting chance. In particular it gives voice much higher
>>> probability for drop-and-delay free behavior, and analyzing the delta
>>> in drops and delay between a separate voice tuple and video tuple
>>> could provide insight as to the congestion in the system.
>> How does this effect happen?
> Could you be more specific?
>
>> In particular, does the separation of flows itself create lower loss
>> probability for voice, or are you thinking of systems where the network
>> elements "know" what the flows contain?
> In nearly any packet scheduled system (drr, sfq,sqf, fq_codel, etc),
> the separation of flows on a 5 tuple will result in (much) less delay,
> jitter and loss for the lower rate flows. With sqf and fq_codel, if
> the rate is low enough there will be near zero queuing delay, jitter
> and loss on many workloads.

I think you answered my previous question here - what you're saying is 
that when sfq etc deals with several flows, it will prioritize delivery 
of packets in the smaller flows ahead of delivery of packets in the 
larger flows.

Which, if the audio flow is smaller than the video flow (true almost all 
the time), will have the effect of causing video corruption to happen 
before audio corruption.

>
> I try to encourage people to just try it to get a feel for what
> happens. openwrt, cerowrt, any modern linux for all of the above,
> cisco routers do drr, many dsl devices do sfq or sqf in their
> firmware....
>
> separation of flows does nothing terrible useful on a strict drop tail
> system, but no harm either.
>
>> What about multiplexing between different flows of the same media type?
> Good question. There are many things I don't understand about webrtc's
> intended audience.
>
> What excites me most about it is the ability to do in-the-building
> videoconferences without having to go over the internet for them.  As
> for going over the internet... I think the cable industry is too
> broken (currently) for any e2e congestion control algorithm to work at
> all with serious cross traffic, with up and downlinks having 2 or more
> seconds of buffering each. (unless fixed by one of the algorithms
> above in a 3rd party router) FIOS and DSL users have a fighting
> chance, 3g and LTE less so, and wifi needs some serious work.
>
> So at the moment I'd like to be addressing the fixable e2e problems in
> gcc (it can't even compete with itself!) and wanted to be able to
> break things out on tuples to look at the data...
>
> before drawing any conclusions on how to multiplex video flows.
>
> My gut tells me you want one tuple per voice and per video flow, AND
> to manage their congestion indicators together, but depending on the
> level of statistical multiplexing of the video flows they could go on
> a single tuple.

That looks approximately sensible to me too.

I spent quite a bit of time arguing in AVT* that setting this kind of 
rule in the structure of the signalling was a grave architectural 
mistake (which was one of the impetuses for the BUNDLE work in the first 
place); that we needed to be free to experiment with this to figure out 
what worked, and change that in deployment, not in specs.

 From what you write above, it seems that one would wish to harness the 
network's (new) tendency to discrimiate against high-volume flows in 
order to get audio through with near zero loss - but I don't want to 
embed that assumption into the application; I'd much prefer that the 
application could make an explicit decision on what's important to it, 
and that ways be found to let the network respect that wish.

Which may lead us back very close to the guidelines that were enshrined 
in RTP and SDP specifications 20 years ago .... but then again, it might 
not.

[This looks so good as the end statement of the message, so it seems a 
pity that I need to add some specifics below. Other people commenting .. 
feel free to delete below this line.]

>
>>> In current implementations, is it possible to force flows onto
>>> different tuples?
>>
>> Absolutely - just don't negotiate bundling.
> Pointer to a working example?

People have been doing this by editing out the line in the SDP that 
starts off
"a=group:BUNDLE"

More elegant solutions should be deployed.

>
> keep hoping there's working ipv6 code out there too, getting the stun
> and turn stuff out of the loop long enough to look at simpler cases.

I think you're already copied on 
https://code.google.com/p/webrtc/issues/detail?id=1406

>
>>>    What are the other flaws in suggesting or requiring
>>> a different tuple?
>>
>>
>>> (note I'm not suggesting bulk data traffic get unique 5 tuples per flow)
>>
>> I wouldn't either.
>> In current deployments, it's multiplexed with the media traffic on the same
>> 5-tuple too - what would you suggest advising it to be bundled with, and
>> why?
> My understanding of the use case for that is that it is for p2p file
> transfer. Given how prone to failure setup has been for this in other
> protocols it seems to make some sense to bundle it on the video
> tuple...
>
>> (dropping the rest of the message - this should be enough for one thread)
>>
>
>

---

From: Michael Welzl <[email-masked]>
Date: Fri, 3 Jan 2014 09:25:13 +0100

>> On Mon, Dec 30, 2013 at 6:09 AM, Harald Alvestrand <[email-masked]> wrote:

[snip]

> That looks approximately sensible to me too.

Not so much to me. Dave's gut seems to argue for separate treatment of audio and video in the network - what for? To protect them from each other, because the sending application doesn't know what it's doing?

> I spent quite a bit of time arguing in AVT* that setting this kind of rule in the structure of the signalling was a grave architectural mistake (which was one of the impetuses for the BUNDLE work in the first place); that we needed to be free to experiment with this to figure out what worked, and change that in deployment, not in specs.
> 
> From what you write above, it seems that one would wish to harness the network's (new) tendency to discrimiate against high-volume flows in order to get audio through with near zero loss - but I don't want to embed that assumption into the application; I'd much prefer that the application could make an explicit decision on what's important to it, and that ways be found to let the network respect that wish.

+1.
Separating tuples can help to let the network protect a flow against *any* other flow, but for prioritization between flows coming from the same application, you should be much better off with joint congestion management, where you have 100% control over priorities and get other benefits too. My argument is: "send stuff into the network the right way rather than hoping for the network to correct your mistakes."

> Which may lead us back very close to the guidelines that were enshrined in RTP and SDP specifications 20 years ago .... but then again, it might not.

Cheers,
Michael

---

From: Dave Taht <[email-masked]>
Date: Fri, 3 Jan 2014 23:18:02 -0500

On Fri, Jan 3, 2014 at 3:25 AM, Michael Welzl <[email-masked]> wrote:
> Not so much to me. Dave's gut seems to argue for separate treatment of audio
> and video in the network - what for? To protect them from each other,
> because the sending application doesn't know what it's doing?

0) audio behaves way differently than video, and should be differentiated.
I personally care not a whit if my video goes to hell once in a while. I care
a lot if my audio does. Videoconferences do contain a lot of visual information
but audio, despite being truly tiny relatively, is critical.

1) To differentiate them from other flows on the network. The sending
application
can't know what those flows are doing.

2) To make analysis easier from tools like wireshark (my immediate need).
What my gut says and what experiment shows is often very different. I
can see some problems with splitting tuples like this, like adding an
extra system call every 10ms or so, and having to explicitly be aware of
joint time stamps from two streams, but I think they are small problems
relative to the potential benefits.

3) To take advantage, where possible, of different scheduling and classification
techniques in the network. A strict drop tail or AQM system won't be
affected by adding in this support, although I foresee better mixing on the
endpoints as being a benefit across any network technology. I'd like
to be fiddling with ECN as well.

What I think I see now is enormous bursts of video and audio on periods
far greater than 10ms in the data I have, with bad collapses, bursty packet
loss, and lousy recovery time. So I'm poking at sources...

> I spent quite a bit of time arguing in AVT* that setting this kind of rule
> in the structure of the signalling was a grave architectural mistake (which
> was one of the impetuses for the BUNDLE work in the first place); that we
> needed to be free to experiment with this to figure out what worked, and
> change that in deployment, not in specs.

I am in part merely talking about stuff that worked in the 90s and 00s.

I wasn't paying attention whenever this discussion took place. link?

> network's (new) tendency to discrimiate against high-volume flows in order

Not "new". drr and weighted red were widely deployed in the 90s, and so far
 as I know still deployed today. free.fr deployed sfq in 2005, and FT
has specified sqf
for nearly that long. Most ISPs are reticent about their secret sauce, and there
are several others whose methods of AQM and packet scheduling I cannot
talk to. Wondershaper came out in 2002 and many other packet scheduling
and qos systems followed.

I only recently learned that free.fr adopted fq_codel summer 2012, for
example. I may end up using their deployed system (which also
does 3 tiers of service classification) for a best practices document.

Aside: I recently ranted about what's wrong in wondershaper today, here:

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

...

I wish more data was available.

Despite pushing the major measurement toolmakers (netanalyzr,
samknows, etc) to measure bufferbloat, and detect scheduling and aqm,
they still don't, and tools like shaperprobe aren't effective either. There
is much work to be done in the testing arena.

Wifi uses 4 levels of classification in 802.11e which in linux at least is
mapped to CS0 through CS7.

> to get audio through with near zero loss - but I don't want to embed that

I'd like to see it get through with near zero jitter, delay, and loss in
roughly that order. non-bursty loss is easily compensated for.

> assumption into the application; I'd much prefer that the application could
> make an explicit decision on what's important to it, and that ways be found
> to let the network respect that wish.

I would like something of the inverse, I'd like the presence of packet
scheduling to
be detected by the application and utilized if seen, by correlating the
differences in behavior between a high rate and a low rate flow in particular.

I'm also very interested in determining the effects of the leading
AQM and packet scheduling candidates on videoconferencing in general.

At the moment, however, what I'd mostly like is the ability to experiment.
Building chromium has thus far been no fun, although I think great strides
could be made with only a few dozen lines of added code.

> +1.
> Separating tuples can help to let the network protect a flow against *any*
> other flow, but for prioritization between flows coming from the same
> application, you should be much better off with joint congestion management,
> where you have 100% control over priorities and get other benefits too. My
> argument is: "send stuff into the network the right way rather than hoping
> for the network to correct your mistakes."

I think it is best to do both.

I think it is incorrect to use the word prioritization when it comes
to the effects of packet scheduling. When I think of prioritization
I think of using diffserv or dpi to classify packets into bins. When
I think of scheduling, fair queuing, flow queueing, etc.,
I think of identifying flows and doing better mixing between them;
of turning packets back into packets again.

Certainly, folk are discovering that sending large bursts into the
network is a bad idea. See for example the recent "Bullet Train"
paper.

http://conferences.sigcomm.org/co-next/2013/program/p133.pdf

Although I feel adding AQM and packet scheduling are necessary for
the current and next generation of nearly all network technologies,
I am tracking improvements in how we do server and client side
stuff closely also.

An endpoint result of all the back and forth about aqm, packet scheduling,
burstyness, and bufferbloat over the last 3 years has been huge reductions in
buffering across the linux stack, a switch to a more pull
model for tcp in particular gradually between linux 3.3 and 3.12,
and the recent addition of a paced fq packet scheduler in 3.12
which is yielding quite excellent results for tcp and video
streams thus far.

http://thread.gmane.org/gmane.linux.network/281023

I think it is probable that this will evolve to become a default qdisc
for linux in general, and that applications should be
twidding with the pacing value in order to get good results
out of the OS on granularities lower than 10ms.

(does anyone know the new setsockopt for this?)

That said, certain kinds of layer 2 technologies (wifi, cable,
3g) *require* bursts in order to function at their highest bandwidth,
and we've trimmed so much fat from Linux in particular that
we're at having to fix device drivers and devices again:

http://www.spinics.net/lists/stable-commits/msg30001.html

So yes, I strongly support good mixing from the application level
and for it to be as self aware as possible of its total impact on the
network.

But: although web browsers are complex and for many users
their entire OS (sigh), there is not enough information about
the hosts's other applications, the network and all the devices
held there to do all the job, thus giving the network more
information about flows in general is a good idea.

And I also strongly encourage experimentation with this stuff.
Me being me, I designed a bunch of experiments, emulated
every technology I could (DSL, cable, wifi, ethernet),
got a lot of data, started looking at captures long before
I started looking at source code and started raising questions
here on rmcat. I figure that's backwards from how most
people operate.

> Which may lead us back very close to the guidelines that were enshrined in
> RTP and SDP specifications 20 years ago .... but then again, it might not.

Cite? I still dimly remember an internet that ran on more ports than 53, 22,
80 and 443. I remember when sip ran on a specific port and RTP was
constrained to a specific port range. The fomer seemed a good idea,
the latter didn't work out so well in the post-nat era.

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

---

From: Michael Welzl <[email-masked]>
Date: Sat, 4 Jan 2014 11:28:29 +0100

Hi,

You answer both Harald and me in this email; I can't respond to the parts where you address Harald, and so I'll just cut some stuff away below.

[snip]

> but audio, despite being truly tiny relatively, is critical.

Tell this to an owner of website that offers WebRTC-supported striptease shows  :-)
My point being: I agree that you describe a common requirement, but this decision should really be up to the application designer (via prioritiziation, via the JS API).

> can't know what those flows are doing.

I don't think we need this.

Case 1: consider two RMCAT applications (browsers doing WebRTC) talking to each other across one bottleneck link. If you multiplex flows, irrespective of what they are, onto a UDP 5-tuple, you shouldn't need any help from the network: you're in complete control over priorities on the sender side, and you have only one bottleneck, i.e. something that acts like a single congestion controller should give you the best behavior. Remember that we're envisioning a delay-based cc. here that minimizes queuing delay anyway. In such a setting, I can't see how it would help to use different tuples for audio and video so that e.g. fq_codel could separate them.

Case 2: consider the same scenario, but with other flows on the bottleneck that recklessly push up the queue. Then, if you have only one common tuple for video and audio, e.g. sfq is going to give this one tuple one bin, and one queue, and we're back at what I described for case 1 -- separating them would only help to protect the browser's own audio from the browser's own video by putting them in different bins.

You mentioned that you'd like to do both, help from the network side *and* from within the application by coupling the congestion controls. Why, how? You might end up with the sender prioritizing flows in one way and the network working against that with sfq. I'd say that we should rather try to prevent sfq from doing its job on the sub-flows of an rtcweb application. This is not at all to say that fq_codel is a bad thing! It's great if it protects rtcweb traffic from everything else - but I think that letting it operate on rtcweb sub-flows could give us a behavior that we don't want, for no real gain. The fairness of e.g. sfq may be inappropriate for individual rtcweb flows (it may go against what the person writing the JS app wanted), and the potential delay benefit should pretty much disappear if the common congestion control is designed to minimize queuing delay anyway.

You keep pointing at the value of real-life tests. Do note that if you run tests with Chromium today, fq_codel may help you: as you know, google's congestion control was found to have issues, and there is no congestion control coupling happening yet. But we're trying to develop something new here, so measuring what's out there right now won't help.

> 2) To make analysis easier from tools like wireshark (my immediate need).

Agreed, this is a point in favor of separating. Then again, the multiplexing-or-not debate is old and I would guess that such things have already been debated? Anyway there are many more pro's and con's to this whole thing than what we're discussing here (a lot of it is related to middlebox behavior).

> to be fiddling with ECN as well.

See above, I don't think that you'd have a benefit, but potentially a disadvantage.

> I think it is best to do both.

- discussed above.

> of turning packets back into packets again.

http://tools.ietf.org/html/draft-ietf-rtcweb-use-cases-and-requirements-12

"   F34     The browser must support prioritization of
           streams and data."

That's my basis for talking about prioritization. AFAIK, the fact that it's listed that way in this document means that it's going to be explicitly supported by the JS API.

> information about flows in general is a good idea.

I disagree because of what I wrote above. When a browser can bundle flows together, it means that it can 100% control their impact of these flows on each other, and then the network can't do things any better than the browser could do by itself.

> here on rmcat. I figure that's backwards from how most

Personally, I would be cautious about drawing conclusions from experiments with things that we know to be moving targets. For example, I can't see that you would be able to get an answer to the questions we're discussing here from experimenting with Chromium today.

Cheers,
Michael

---

From: Dave Taht <[email-masked]>
Date: Sat, 4 Jan 2014 15:54:37 -0500

> I don't think we need this.

I think you are wrong.

> Case 1: consider two RMCAT applications (browsers doing WebRTC) talking to each other across one bottleneck link. If you multiplex flows, irrespective of what they are, onto a UDP 5-tuple, you shou…

Who is we? I'm not envisioning a delay based cc, but only a partially
based one. Delay based cc tcps don't work vs loss based. next
question.

> You mentioned that you'd like to do both, help from the network side *and* from within the application by coupling the congestion controls. Why, how? You might end up with the sender prioritizing flows in one way and the network working against that with sfq. I'd say that we should rather try to prevent sfq from doing its job on the sub-flows of an rtcweb application.

I think that all on one tuple should be allowed, and on different
tuples allowed also, and let the application designer choose.

>This is not at all to say that fq_codel is a bad thing! It's great if it protects rtcweb traffic from everything else - but I think that letting it operate on rtcweb sub-flows could give us a behavior that we don't want, for no real gain. The fairness of e.g. sfq may be inappropriate for individual rtcweb flows (it may go against what the person writing the JS app wanted), and the potential delay benefit should pretty much disappear if the common congestion control is designed to minimize queuing delay anyway.

I don't see how you can get sufficient insight into the underlying TCP
stack to have a common congestion control picture across that and
webrtc.

> You keep pointing at the value of real-life tests. Do note that if you run tests with Chromium today, fq_codel may help you: as you know, google's congestion control was found to have issues, and t…

What was measured in this series of experiments was chromium vs cubic
tcp, under a variety of dsl, cable, wifi emulations that seem to be
pretty
accurate.

The whole point of the experiment was to get a grip on what happened
today and get insight as to be what could be done tomorrow.

> Agreed, this is a point in favor of separating. Then again, the multiplexing-or-not debate is old and I would guess that such things have already been debated?

I was too busy fixing routers to pay attention. Cites to relevant
arguments in the past on other mailing lists desired.

>Anyway there are many more pro's and con's to this whole thing than what we're discussing here (a lot of it is related to middlebox behavior).

I agree middlebox behavior is a serious problem. I had qualified this
conversation to be talking about two tuples rather than one for voice
and video, and also pointed out the non natted and ipv6 cases as being
the ones I was looking at primarily. I don't see a problem keeping two
ports open rather than one, even with middleboxes, actually, but am
open to opposing arguments.

> See above, I don't think that you'd have a benefit, but potentially a disadvantage.

I basically already demonstrated a benefit in the captures I took, and some
additional simulations of isochronous traffic vs variable.

I'll take time out to write a paper after I get more code working. Maybe.

Further experiments are good.

> That's my basis for talking about prioritization. AFAIK, the fact that it's listed that way in this document means that it's going to be explicitly supported by the JS API.

Thank you for the steer to the relevant document. I will read.

> I disagree because of what I wrote above. When a browser can bundle flows together, it means that it can 100% control their impact of these flows on each other

Agree and there is much fat to be sliced in the codebases I'm looking
at in favor of better mixing.

>, and then the network can't do things any better than the browser could do by itself.

disagree.

> Personally, I would be cautious about drawing conclusions from experiments with things that we know to be moving targets. For example, I can't see that you would be able to get an answer to the que…

I long ago drew a conclusion that current networks have way too much
buffering for any form of congestion control we know of to work right
for videoconferencing vs deployed tcps on cable, wifi, 3g and LTE.
Some hope on DSL & gpon. DSL because most that I have tried already
have some form of aqm or packet scheduling, gpon as it seems pretty
good on uploads, and not horrible on downloads.

As for experimentation, certainly answers can be obtained from the
moving targets!

Chromium is open source, so anybody can hack on it. I got a build done
last night. Adding in the bits of code and packet handling I'm
discussing experimenting with is fairly trivial. (finding where to add
it and working through the overlarge set of c++ abstractions is thus
far proving difficult.)

I am interested in the following experiments:

finding the "right" rate quickly
dealing with non-congestive (e.g. wifi) losses
improving recovery after a congestive loss
better analysis via rtp on separate tuples
looking at the effects of various proposed classification schemes, and
how to detect if they are preserved end to end
what ecn could be used for (and testing for end to end preservation also)
looking at the interactions of what network types exist today
(dsl,cable, wifi, 3g) in general
looking at aqm and packet scheduling vs drop tail behaviors.

Most of the core code we need is already written elsewhere. For
example being able to pay attention to classification, preservation
thereof, and
ECN is handled by these bits of code in mosh. (side note, TOS setting
is wrong for ipv6 in chrome & mosh, you need to use IPPROTO_IPV6 with
IPV6_TCLASS instead of IPPROTO_IP & IP_TOS, easy fix pending)

  /* request explicit congestion notification on received datagrams */
#ifdef HAVE_IP_RECVTOS
  int tosflag = true;
  socklen_t tosoptlen = sizeof( tosflag );
  if ( setsockopt( _fd, IPPROTO_IP, IP_RECVTOS, &tosflag, tosoptlen ) < 0 ) {
    perror( "setsockopt( IP_RECVTOS )" );
  }
#endif
}

string Connection::recv_one( int sock_to_recv, bool nonblocking )
{
  /* receive source address, ECN, and payload in msghdr structure */
  struct sockaddr_in packet_remote_addr;
  struct msghdr header;
  struct iovec msg_iovec;

  char msg_payload[ Session::RECEIVE_MTU ];
  char msg_control[ Session::RECEIVE_MTU ];

  /* receive source address */
  header.msg_name = &packet_remote_addr;
  header.msg_namelen = sizeof( packet_remote_addr );

  /* receive payload */
  msg_iovec.iov_base = msg_payload;
  msg_iovec.iov_len = Session::RECEIVE_MTU;
  header.msg_iov = &msg_iovec;
  header.msg_iovlen = 1;

  /* receive explicit congestion notification */
  header.msg_control = msg_control;
  header.msg_controllen = Session::RECEIVE_MTU;

  /* receive flags */
  header.msg_flags = 0;

  ssize_t received_len = recvmsg( sock_to_recv, &header, nonblocking ?
MSG_DONTWAIT : 0 );

  if ( received_len < 0 ) {
    throw NetworkException( "recvmsg", errno );
  }

  if ( header.msg_flags & MSG_TRUNC ) {
    throw NetworkException( "Received oversize datagram", errno );
  }

  /* receive ECN */
  bool congestion_experienced = false;

  struct cmsghdr *ecn_hdr = CMSG_FIRSTHDR( &header );
  if ( ecn_hdr
       && (ecn_hdr->cmsg_level == IPPROTO_IP)
       && (ecn_hdr->cmsg_type == IP_TOS) ) {
    /* got one */
    uint8_t *ecn_octet_p = (uint8_t *)CMSG_DATA( ecn_hdr );
    assert( ecn_octet_p );

    if ( (*ecn_octet_p & 0x03) == 0x03 ) {
      congestion_experienced = true;
    }
  }

-- 
Dave Täht

-- 
Dave Täht

---

From: John Leslie <[email-masked]>
Date: Sat, 4 Jan 2014 17:23:22 -0500

--
John Leslie <[email-masked]>

---

From: Michael Welzl <[email-masked]>
Date: Mon, 6 Jan 2014 11:37:25 +0100

+1.

I'll answer the longer email in more detail anyway because of questions asked and issues raised, but indeed I think this is the solution that we can all live with.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Mon, 6 Jan 2014 12:09:40 +0100

> based one.

With "we", I meant rmcat, based on its charter. However, my phrasing was imprecise: the requirement for RMCAT's congestion control as stated in the charter only reads "Low delay and low jitter for the case where there is no competing traffic using other algorithms".

I think it's pretty obvious that such an algorithm would incorporate some delay measure in its decision (as all 3 current proposals do), and - as you also say - only partially base decisions on delay, i.e. not ignore other signals such as loss (as all 3 current proposals do). Thus I made this assumption about the algorithm that "we" envision.

> Delay based cc tcps don't work vs loss based. next
> question.

Mechanisms can work reasonably well in the presence of TCP, yet minimize delay in its absence. All 3 current proposals try to do that in some way, and there are several more examples out there. Such mechanisms of course work best when they don't even have to compete against TCP because e.g. fq_codel gives them their own bin. So that's all fine!

Anyway, the requirement for RMCAT's congestion control is to "low delay and jitter", and that's my whole point - the congestion controller should produce low delay, and saying that it won't or it can't means that you question the RMCAT charter. If that's not your intention, let's work from the assumption of a low-delay-producing congestion controller.

> stack to have a common congestion control picture across that and

I don't understand why I need to know anything beyond webrtc if all of webrtc's traffic is on one tuple and gets its own bin.

> arguments in the past on other mailing lists desired.

I also only know that it was debated, and I guess it can be found in the rtcweb list archives. Finding it there is only going to take 2-3 days or so  :-)   that list has been extremely noisy.

> additional simulations of isochronous traffic vs variable.

My argument in our discussion (#1) is that coupled congestion control would eliminate the need for the network to help rtcweb's individual flows to get low delay (and in probably do a better job), and therefore it's probably best to multiplex all of rtcweb's traffic on one 5-tuple for the sake of delay.

Your argument (#2) is the opposite - separate tuples can be better supported by the network, which will give rtcweb's flows lower delay.

In that context, how can you demonstrate a benefit of #2 over #1 when #1 isn't even implemented yet?

> for videoconferencing vs deployed tcps on cable, wifi, 3g and LTE.

That's nice but really not what we're discussing here.

> last night.

My point is, AFAIK it does *not* do coupled congestion control, so if you measure its behavior you're not measuring what we're talking about. Yes we would like to add code on coupled congestion control to it when the time comes. We take it step by step.

Cheers,
Michael

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Mon, 6 Jan 2014 14:03:33 +0000

Michael/Dave,

I am trying to save this thread, as you both have important points.

However, when Michael talks about coupled congestion control he almost exclusively talks about coupling of the congestion control of flows ON ONE HOST. With that perspective, the coupling of the congestion control of the individual flows ON ONE HOST makes sense (even if they are undifferentiated on the sent 5-tuple).

And when Dave talks about coupled congestion control, he argues for de-coupling of the 5-tuples because he knows that coupling the flows ACROSS MULTIPLE HOSTS isn't possible on the time scales required (tens of milliseconds) and the best that can be done is some network-based decoupling of the individual voice and video flows. And Dave thinks that any cross-flow coupling is near hopeless whenever a "long-enough-lived" TCPs crank up (and I agree - and hope I paraphrased his position correctly).

In other words, Dave is talking about traffic AGGREGATES from many hosts on a given (usually bottleneck, bufferbloated) link.

I have had this communication issue with Dr. Welzl in the past (me talking about traffic aggregates and he talking about congestion control coupling on a given host).

Now ... with this as a background ... don't you both sound correct from your perspectives?

[Off Soapbox, hope this helped.]

Ramalho

-----Original Message-----
From: rmcat [mailto:[email-masked]] On Behalf Of Michael Welzl
Sent: Monday, January 06, 2014 6:10 AM
To: Dave Taht
Cc: Harald Alvestrand; [email-masked]
Subject: Re: [rmcat] 5 tuples and rmcat-cc-requirements-01

> Delay based cc tcps don't work vs loss based. next question.

>>> What my gut says and what experiment shows is often very different. 
>>> I can see some problems with splitting tuples like this, like adding 
>>> an extra system call every 10ms or so, and having to explicitly be 
>>> aware of joint time stamps from two streams, but I think they are 
>>> small problems relative to the potential benefits.
>>> 
>>> 3) To take advantage, where possible, of different scheduling and 
>>> classification techniques in the network. A strict drop tail or AQM 
>>> system won't be affected by adding in this support, although I 
>>> foresee better mixing on the endpoints as being a benefit across any 
>>> network technology. I'd like to be fiddling with ECN as well.
>> 
> I basically already demonstrated a benefit in the captures I took, and 
> some additional simulations of isochronous traffic vs variable.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Mon, 6 Jan 2014 16:21:56 +0100

> However, when Michael talks about coupled congestion control he almost exclusively talks about coupling of the congestion control of flows ON ONE HOST. With that perspective, the coupling of the co…

Confirmed.

> In other words, Dave is talking about traffic AGGREGATES from many hosts on a given (usually bottleneck, bufferbloated) link.

Well... frankly I assume that Dave is thinking about fq_codel, and fq_codel would give every tuple a bin. So the scenario in my mind is:
1 tuple from 1 host getting a bin in fq_codel
vs.
multiple tuples from 1 host getting a bin each in fq_codel
... and then, whether there are even more tuples from even more hosts or not doesn't matter much for our discussion I think.

> I have had this communication issue with Dr. Welzl in the past (me talking about traffic aggregates and he talking about congestion control coupling on a given host).

Sorry!  :-(

> [Off Soapbox, hope this helped.]

Dunno, hope I could clarify...

Cheers,
Michael

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 8 Jan 2014 08:13:07 +0000

Hi

Have not followed the whole thread due to an abundance of XMAS but anyway a few comments. 
I agree that fq_codel and similar is very good as it can help to maintain very low latency and packet loss for small flows, the challenge is however deployment. Same goes with ECN, it helps to do away with packet drops as congestion signal but adoption of ECN is very slow. Perhaps the emerging work around Immediate ECN can increase the interest but it is anyway a long way to go..

Both more advanced AQMs and ECN suffer from the deplyment problem and for that reason it is necessary to use RMCAT congestion controls that are a hybrid of delay and loss/ECN based and the congestion control is then likely to need to rely on delay measurements for a foreseable future.

Delay based CC algos are not bullet proof though, it is easy for a loss based TCP to starve out an application that use delay based CC. One can of course borrow ideas such as Hamilton Delay TCP but it is not completely without complications.

In the best of worlds I would see something like Immediate ECN or at least wide AQM deployment but until that becomes reality I guess we will have to stick to delay based CC and make that work as good as possible.

/Ingemar

> From: Michael Welzl [mailto:[email-masked]]
> Sent: den 6 januari 2014 12:10
> To: Dave Taht
> >> Case 1: consider two RMCAT applications (browsers doing WebRTC)
> talking to each other across one bottleneck link. If you multiplex flows,
> irrespective of what they are, onto a UDP 5-tuple, you shouldn't need any
> help from the network: you're in complete control over priorities on the
> sender side, and you have only one bottleneck, i.e. something that acts like a
> single congestion controller should give you the best behavior. Remember
> that we're envisioning a delay-based cc. here that minimizes queuing delay
> anyway. In such a setting, I can't see how it would help to use different
> tuples for audio and video so that e.g. fq_codel could separate them.
> >
> With "we", I meant rmcat, based on its charter. However, my phrasing was
> imprecise: the requirement for RMCAT's congestion control as stated in the
> charter only reads "Low delay and low jitter for the case where there is no
> competing traffic using other algorithms".
> 
> I think it's pretty obvious that such an algorithm would incorporate some
> delay measure in its decision (as all 3 current proposals do), and - as you also
> say - only partially base decisions on delay, i.e. not ignore other signals such
> as loss (as all 3 current proposals do). Thus I made this assumption about the
> algorithm that "we" envision.
> 
> 
> Mechanisms can work reasonably well in the presence of TCP, yet minimize
> delay in its absence. All 3 current proposals try to do that in some way, and
> there are several more examples out there. Such mechanisms of course
> work best when they don't even have to compete against TCP because e.g.
> fq_codel gives them their own bin. So that's all fine!
> 
> Anyway, the requirement for RMCAT's congestion control is to "low delay
> and jitter", and that's my whole point - the congestion controller should
> produce low delay, and saying that it won't or it can't means that you
> question the RMCAT charter. If that's not your intention, let's work from the
> assumption of a low-delay-producing congestion controller.
> 
> 
> >> This is not at all to say that fq_codel is a bad thing! It's great if it protects
> rtcweb traffic from everything else - but I think that letting it operate on
> rtcweb sub-flows could give us a behavior that we don't want, for no real
> gain. The fairness of e.g. sfq may be inappropriate for individual rtcweb flows
> (it may go against what the person writing the JS app wanted), and the
> potential delay benefit should pretty much disappear if the common
> congestion control is designed to minimize queuing delay anyway.
> >
> I don't understand why I need to know anything beyond webrtc if all of
> webrtc's traffic is on one tuple and gets its own bin.
> 
> 
> >> Agreed, this is a point in favor of separating. Then again, the multiplexing-
> or-not debate is old and I would guess that such things have already been
> debated?
> >
> I also only know that it was debated, and I guess it can be found in the
> rtcweb list archives. Finding it there is only going to take 2-3 days or so  :-)
> that list has been extremely noisy.
> 
> 
> My argument in our discussion (#1) is that coupled congestion control would
> eliminate the need for the network to help rtcweb's individual flows to get
> low delay (and in probably do a better job), and therefore it's probably best
> to multiplex all of rtcweb's traffic on one 5-tuple for the sake of delay.
> 
> Your argument (#2) is the opposite - separate tuples can be better supported
> by the network, which will give rtcweb's flows lower delay.
> 
> >> Personally, I would be cautious about drawing conclusions from
> experiments with things that we know to be moving targets. For example, I
> can't see that you would be able to get an answer to the questions we're
> discussing here from experimenting with Chromium today.
> >
> My point is, AFAIK it does *not* do coupled congestion control, so if you
> measure its behavior you're not measuring what we're talking about. Yes we
> would like to add code on coupled congestion control to it when the time
> comes. We take it step by step.
> 
> Cheers,
> Michael
>

---

From: John Leslie <[email-masked]>
Date: Wed, 8 Jan 2014 07:40:19 -0500

Ingemar Johansson S <[email-masked]> wrote:
> 
> I agree that fq_codel and similar is very good as it can help to
> maintain very low latency and packet loss for small flows, the
> challenge is however deployment.

   I suggest working on diagnostic tools then -- to determine where
fq_codel/similar is or is not deployed at a bottleneck.

> Same goes with ECN, it helps to do away with packet drops as
> congestion signal but adoption of ECN is very slow. Perhaps the
> emerging work around Immediate ECN can increase the interest but
> it is anyway a long way to go..

   Again, work on diagnostic tools would help -- to determine where
Immediate ECN is or is not deployed.

   And BTW, writing a definition of Immediate ECN would help...

---

From: Randell Jesup <[email-masked]>
Date: Wed, 22 Jan 2014 13:55:27 -0500

I'm going to make deeper comments once I read through everything, but 
some detail points while I have them in front of me:

One big issue for bundle is that ICE takes time, and it takes time 
proportional to the number of ports you have to open.  From experience, 
the more ports needing to be opened the more probably you can get.  For 
Hangout-like conferencing, "classic" RTP setup would mean 5 + 4(n-1) 
ports where N is the number of participants at least, or more likely 7 + 
6(n-1) (assuming we do simulcast and do it on a separate RTP/RTCP 
stream).  It also means 4 or 6 times the amount of TURN ports needed by 
a service offering WebRTC. (One day, ICE and TURN will die the horrible 
death they should when IPV6 is the norm - but even then it may be needed 
for things like firewalls that block UDP.)

This drives a) RTP/RTCP mux (already being done, and predates WebRTC), 
and BUNDLE.  With both, you can get 2 or 1 ports for a conference 
(depending on if you bundle the DataChannel flow).

> but audio, despite being truly tiny relatively, is critical.

Audio is critical in a conference, true.  However when we're constrained 
for bandwidth we may want to reduce audio bandwidth targets rather than 
drop video totally or have sucky video - especially if we're running 
higher-than-current-quality audio (Opus and WebRTC (will) support 
multi-channel audio at virtually lossless levels - but that takes a lot 
more bandwidth than current default audio settings).

Also, perceptually a conversation over video feels "real" when the video 
is high frame rate (>20 consistently, pref 30 rock solid). letting it 
occasionally get nuked (or even drop frame rate noticably) takes you out 
of that mode of perception.

The relative priority of these things is something that feeds back to 
the application needs, which will vary.

> relative to the potential benefits.

Wireshark can separate flows on the same tuple using SSRC & payloads.  
There's an issue for Wireshark in that the payloads and signaling are 
all encrypted/hidden/application-specific - oh well. Same applies to 
encrypted SIP calls.  The only addition is SSRC muxing onto one 5-tuple, 
which wireshark can understand if it wants.  It can if it likes infer 
audio or video from the payload and size (and Mark), of course it can be 
wrong.

> loss, and lousy recovery time. So I'm poking at sources...

Typical audio is close to isochronous at 20ms.  Video will be bursts 
(perhaps metered onto the wire) every 33-100ms or so.  Below ~300K it 
will be 1-packet bursts typically.  Note: this gets messier with the 
default behavior in current WebRTC to use retransmission on loss reports 
if the RTT is low enough.  I have issues with this behavior for recovery 
of video; typically I prefer a dynamic refresh using a pframe against an 
older reference frame (or other non-iframe/IDR recovery method).  It 
avoids making congestion problems causing loss worse...  I know Justin 
Uberti (of Google) strongly prefers to use retransmission based on their 
experience with Hangouts (which (were) using H.264 SVC).  They also may 
be able to put servers closer to users that can short-circuit 
retransmissions without going all the way back to the source (in a 
conference).

> Wifi uses 4 levels of classification in 802.11e which in linux at 
> least is mapped to CS0 through CS7. 

WiFi is a whole other headache.... and an important one.

> roughly that order. non-bursty loss is easily compensated for.

Audio single-packet loss concealment is pretty good in Opus (and iLBC, 
and I believe in iSAC).  Ok to so-so in other codecs.  And single-packet 
losses are easy enough to hide that generally the experience will be 
better allowing a moderate amount of them INSTEAD of video losses.  Odd, 
but true.

I'll note that video being delayed behind audio is NOT a good thing; 
it's actually bad.  The reason is lip-sync.  Humans hate seeing video 
that lags audio - they're somewhat used to audio lagging video (see 
something in the distance, then hear it), but not tolerant of the 
reverse.  So if video is delayed, we have to artificially delay audio to 
match (run deeper jitter buffers than needed, basically). If you're 
doing non-realtime streaming, it's not an issue, but for realtime the 
delay for playback is the *worst* of the synchronized sub-streams.

Video is rather more tolerant of loss -- skipped frames aren't 
super-noticable, and often a lost packet doesn't cause major artifacts 
(if you do decode-with-errors-until-recovery instead of 
freeze-until-recovery, which gives much longer freezes but no errors, or 
freeze-until-retransmit which can lead to video high-speed "catchup" 
when the lost packet gets through, which is kinda odd).

> differences in behavior between a high rate and a low rate flow in particular.

Different applications have different needs on the streams.  Some may 
want to drop a stream rather than get by with less bits for other 
streams, or drop video rather than degrade both it and audio, or drop 
secondary video channels to keep the primary happy.  A VRS service may 
put video (and consistent frame rate!) above audio in importance.

> AQM and packet scheduling candidates on videoconferencing in general.

Absolutely!  and please reach out to me again; our attempts to get a 
call in before xmas got lost in the busy period that led to Mozilla's 
holiday shutdown (and some personal issues).

> could be made with only a few dozen lines of added code.

It's much easier to build Firefox.... ;-)  Though it can still take some 
time.  Install mercurial, clone mozilla-central (our trunk), install 
some dependencies if needed (generally not needed), and "./mach build".  
There are more things you can do, but that's the basics.

> http://conferences.sigcomm.org/co-next/2013/program/p133.pdf

This is a scale of bursts way beyond anything in a WebRTC application 
(even video streaming, which they mention, is 64KB-2MB per burst).  A 
"very large" burst for WebRTC would be maybe 10-20KB; rarely would it 
get to that level except at super-high bitrates (and then rarely).  So 
nothing much from this is applicable (at a quick reading).

> http://thread.gmane.org/gmane.linux.network/281023

a) 90% of our traffic in WebRTC is Windows.  b) most of the rest is 
Mac.  c) that's TCP traffic with an "infinite" source it appears; we 
have paced senders.  Even when we use TURN-TCP we'll have a (regular) 
bursty flow with idle times.

Not to say that isn't good - it's good!  and it also makes other 
realtime things competing with TCP much, much happier.

> information about flows in general is a good idea.

Agreed.  But it's not all simple...

> here on rmcat. I figure that's backwards from how most

:-)

> the latter didn't work out so well in the post-nat era.

RTP was never constrained that I know of; many applications limited RTP 
to things like >16384 for no real apparent reason other than maybe to 
make it easier to write router/NAT/firewall rules.  :-)

-- 
Randell Jesup -- rjesup a t mozilla d o t com

---

From: Randell Jesup <[email-masked]>
Date: Wed, 22 Jan 2014 14:22:29 -0500

> good on uploads, and not horrible on downloads.

No realtime flow in the current environment (buffering, greedy TCP) can 
do really well; see the papers and notes from the IAB Realtime 
Congestion Control Workshop that occurred with the BOF for RMCAT 
(Vancouver a year and a half ago).  TCP will try to eat your lunch 
eventually unless someone separates it out (even if there's no 
buffering), unless you're nasty and refuse.  In droptail/etc esp. with 
bloat, this is Bad, as TCP will put you in a queue.  With AQM, you may 
lose but the queuing won't be bad, and you'll see loss you can react 
to.  (And note that current experiments likely haven't tuned the loss 
behavior for AQM yet, though the requirements say they'll need to.)

> As for experimentation, certainly answers can be obtained from the

Just be careful to remember the moving targets may not be stable or 
indicative of the behavior envisioned by the requirements.

> finding the "right" rate quickly

A known flaw in Google's algorithm currently.  My old algorithm 
(Worlgate/Ojo) was very aggressive at the start of a flow and would 
overshoot, then notice queuing developing and back off fast to drain.  
After ~30 seconds it got more conservative about putting it's toes over 
the apparent available bandwidth (high water); when recovering from 
local dips it would be moderately aggressive up to the high-water-mark 
for a stable flow with no delay buildup. Startup behavior was discussed 
a lot 2 years ago on rtcweb.  Also, people who do HD conferences on 
high-bandwidth links and local LANs really, really want to start much 
higher (and adapt up faster).

> dealing with non-congestive (e.g. wifi) losses

Yup.  I used to allow a small amount of loss, then apply modifiers on 
top of apparent delay over a loss threshold (progressively).  You 
generally don't know the cause of a loss (especially in AQM).

> (dsl,cable, wifi, 3g) in general

Yup, all interesting.

> looking at aqm and packet scheduling vs drop tail behaviors.

Absolutely and called for in the requirements and the testing doc from 
Varun.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 24 Jan 2014 11:08:42 -0500

On 1/22/2014 1:55 PM, Randell Jesup wrote:
> proportional to the number of ports you have to open.  From 
> experience, the more ports needing to be opened the more probably you 
> can get.  For Hangout-like conferencing, "classic" RTP setup would 
> mean 5 + 4(n-1) ports where N is the number of participants at least, 
> or more likely 7 + 6(n-1) (assuming we do simulcast and do it on a 
> separate RTP/RTCP stream).  It also means 4 or 6 times the amount of 
> TURN ports needed by a service offering WebRTC. (One day, ICE and TURN 
> will die the horrible death they should when IPV6 is the norm - but 
> even then it may be needed for things like firewalls that block UDP.)
>

Amplifying my own point, this clearly impacts a lot of important 
performance aspects of media exchange.  Classically people answered 
handsets and it takes time to get them to the ear/mouth; that gave you 
100-500ms to finish call setup after "accept".  With GUIs and headsets 
or speakerphones, people will press and say "hello"; at best you have 
150ms. I've found you'll get clipping in speakerphones with 
press-to-accept if streams aren't flowing in 200ms, preferably less.  
300ms is quite noticeable.

Sometimes you can hide this delay by warming up channels before the user 
accepts.  However, that's not always possible, and also may fall afoul 
of other requirements (avoiding exposing physical location by IP without 
user interaction, avoid DoS attacks (incoming request produces a burst 
of ICE traffic in response as well as port reservations in the 
firewall), etc).

Also, there are more chances you'll run into port conflicts that cause 
NATs to switch "mode"; some non-symmetric NATs will port-preserve if 
it's not in-use/mapped, but when forced to map to another port will 
switch to symmetric NAT.  Really.  So with a large number of ports you 
might get different characteristics in the NAT (some requiring TURN, 
some not), and also even in simpler cases if there are multiple network 
interfaces you might end up with some ports on one interface and some on 
another, or some randomly deciding that a longer-delay candidate is 
better, etc.  The odds get worse the larger the number of ports.

---

From: Dave Taht <[email-masked]>
Date: Fri, 24 Jan 2014 11:38:42 -0500

On Fri, Jan 24, 2014 at 11:08 AM, Randell Jesup <[email-masked]> wrote:
>> I'm going to make deeper comments once I read through everything, but some
>> detail points while I have them in front of me:
>>
>> proportional to the number of ports you have to open.  From experience, the
>> more ports needing to be opened the more probably you can get.  For
>> Hangout-like conferencing, "classic" RTP setup would mean 5 + 4(n-1) ports
>> where N is the number of participants at least, or more likely 7 + 6(n-1)
>> (assuming we do simulcast and do it on a separate RTP/RTCP stream).  It also
>> means 4 or 6 times the amount of TURN ports needed by a service offering
>> WebRTC. (One day, ICE and TURN will die the horrible death they should when
>> IPV6 is the norm - but even then it may be needed for things like firewalls
>> that block UDP.)
>>
>> This drives a) RTP/RTCP mux (already being done, and predates WebRTC), and
>> BUNDLE.  With both, you can get 2 or 1 ports for a conference (depending on
>> if you bundle the DataChannel flow).
>
>
> Amplifying my own point, this clearly impacts a lot of important performance
> aspects of media exchange.  Classically people answered handsets and it
> takes time to get them to the ear/mouth; that gave you 100-500ms to finish
> call setup after "accept".  With GUIs and headsets or speakerphones, people
> will press and say "hello"; at best you have 150ms. I've found you'll get
> clipping in speakerphones with press-to-accept if streams aren't flowing in
> 200ms, preferably less.  300ms is quite noticeable.
>
> accepts.  However, that's not always possible, and also may fall afoul of
> other requirements (avoiding exposing physical location by IP without user
> interaction, avoid DoS attacks (incoming request produces a burst of ICE
> traffic in response as well as port reservations in the firewall), etc).
>
> Also, there are more chances you'll run into port conflicts that cause NATs
> to switch "mode"; some non-symmetric NATs will port-preserve if it's not
> in-use/mapped, but when forced to map to another port will switch to
> symmetric NAT.  Really.  So with a large number of ports you might get
> different characteristics in the NAT (some requiring TURN, some not), and
> also even in simpler cases if there are multiple network interfaces you
> might end up with some ports on one interface and some on another, or some
> randomly deciding that a longer-delay candidate is better, etc.  The odds
> get worse the larger the number of ports.

I am beginning to feel a lot more of your pain. :/

However I still stand by

A) wanting to ensure that requirements driven by ICE/turn/nat negotiation
hassles don't affect direct e2e problems, and to ensure the API exposes
the ability to do saner things e2e in the absence of of
ice/turn/nat/middleboxes,
and (especially) over ipv6.

B) wanting to clearly separate audio and video tuples.

C) wanting world peace.

D) from a congestion control and analysis perspective I'd dearly like
to be able to
work without all these middleboxware for a while, and it is also coming
clear to me that I'd like to take a hard look at their behaviors also.

-- 
Dave Täht

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 24 Jan 2014 17:56:36 +0100

Hi,

one more comment on the requirements that just came to my mind while reading 
this: To be able to quickly grab capacity, other flows on the link, that 
might already utilize the link fully, need also to release capacity quickly. 
Of course we cannot change existing congestion control but we can at least 
require this for rmcat. The document already says that the flow should adapt 
quickly but this is more or less just referred to case where more bandwidth 
becomes available. Sensing that the traffic situation changed and now more 
flows are competing is much harder and thus I would maybe have this as a 
separate requirement.

By the way: are the requirements currently ordered by priority? Or is there a 
specific order? This should maybe be explained at the beginning. And if there 
is some kind of priority, I would see requirement 10 on the startup behavior 
somewhere at the beginning (right after low delay and competing with other 
TCPs).

Mirja

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: Randell Jesup <[email-masked]>
Date: Fri, 24 Jan 2014 17:58:42 -0500

On 1/24/2014 11:56 AM, Mirja Kuehlewind wrote:
> require this for rmcat.

Yes, though I'll note that releasing capacity quickly is almost always a 
requirement to avoid big problems (triply so for realtime code that 
doesn't want to build queues).  If you don't release fast, you are in 
deep trouble fast unless you're just a tiny bit over-bandwidth.  My 
algorithm tried to estimate how far it was over (by the slope of the 
apparent queue (delay) increase), and modify how far it cut back 
according to that (22ms per 20ms packet implies ~10% over-bandwidth, so 
cut bandwidth by (say) 15% immediately). (Yes, real stuff would be more 
complex, but you get the idea.) Better to overshoot on cutback; and this 
tends to help self-fairness as well, especially if there are different 
rules fr established flows compared to new ones.

> The document already says that the flow should adapt

I'll look.  Much of this is implicit in the requirement to keep delay 
down and to not block new flows from getting a share.

> somewhere at the beginning (right after low delay and competing with other

They are not in priority order; I think priorities are a judgement call 
and part of the WG's job in evaluating the results; there are tradeoffs 
between all of these in every algorithm, and it's the overall result 
that's important, not really the sub-pieces on their own.  (IMHO).

   Randell