Subject: Re: [Ice] Trickle: What is Section 16 about?

From: Christer Holmberg <[email-masked]>
Date: Thu, 1 Mar 2018 14:01:26 +0000

Hi,

Ok, I get it why components need to be trickled in order.

But, the text also talks about “streams”. I don’t think you need to wait until all candidates for one stream have been trickled until you trickle the candidates for another stream.

Also, the SDP example seem to assume that the order of SDP candidate attributes matter. But, in SDP the order does not matter, and intermediaries could even change the order between parsing and incoming and generating an outgoing message…

Regards,

Christer

From: Taylor Brandstetter <[email-masked]<mailto:[email-masked]>>
Date: Wednesday 28 February 2018 at 20:34
To: Christer Holmberg <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Ice] Trickle: What is Section 16 about?

Does it mean that, when an agent gathers a new candidate, it must not send
it to the peer in case there will later be a candidate with a lower ID
within the same foundation?

That's how I interpret it; the example below explains how an RTCP candidate for a particular foundation must be conveyed after (or simultaneously with) the RTP candidate.

As for "why", I think that's explained by the paragraph above:

   One important aspect of regular ICE is that connectivity checks for a
   specific foundation and component are attempted simultaneously by
   both agents, so that any firewalls or NATs fronting the agents would
   whitelist both endpoints and allow all except for the first
   ("suicide") packets to go through.  This is also important to
   unfreezing candidates at the right time.  While not crucial,
   preserving this behavior in Trickle ICE is likely to improve ICE
   performance.

Trickling candidates in a specific order helps guarantee that pairs are formed in the same order by both peers, so that they'll perform connectivity checks for the same pairs simultaneously.

Though given the updates to section 8.1.1 (Inserting a New Pair in a Check List), this may not be as important as it used to be. The rules in 8.1.1 guarantee that the "topmost" pair for each foundation will always be unfrozen. So I believe if candidates were trickled out of order, the worst that could happen is that one peer ends up with more pairs unfrozen than the other. For example:

1. Bob trickles RTCP candidate, then RTP.
2. Alice trickles RTP candidate, then RTCP.
3. Alice receives Bob's candidates, forming RTCP pair first (unfreezing it), then RTP pair (unfreezing it too, since it becomes the new "topmost" pair).
4. Bob receives Alice's candidates, forming RTP pair first (unfreezing it), then RTCP pair (leaving it frozen).
5. Alice now has both pairs unfrozen, while Bob only has the RTP pair unfrozen.

On Tue, Feb 27, 2018 at 11:13 PM, Christer Holmberg <[email-masked]<mailto:[email-masked]>> wrote:
Hi,

Section 16 says the following:

   "To achieve this, when trickling candidates, agents MUST respect the
   order in which the components and streams appear (implicitly or
   explicitly) as they have been negotiated by means of the relevant
   candidate information.²

I can¹t parse this sentence. What ³order² is the text talking about?

   "Therefore candidates for a given component MUST NOT be conveyed prior
to
    candidates for a component with a lower ID number within the same
foundation.²

Does it mean that, when an agent gathers a new candidate, it must not send
it to the peer in case there will later be a candidate with a lower ID
within the same foundation? I thought the whole idea of trickle was to
provide candidates to the peer in ³real-time², once they become available.

If the order in which candidates are placed in SDP matters, that should be
described in trickle-ice-sdp.

My suggestion would be to remove Section 16 completely, but I would like
someone to tell me why we need it, and what exactly it is trying to say :)

Regards,

Christer

_______________________________________________
Ice mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ice

---

From: Taylor Brandstetter <[email-masked]>
Date: Thu, 1 Mar 2018 09:04:01 -0800

>
> I don’t think you need to wait until all candidates for one stream have
> been trickled until you trickle the candidates for another stream.
>

What makes "streams" different than "components" in this regard? Both
determine what pair is unfrozen first. You can take the example above and
replace "RTP" with "stream 1" and "RTCP" with "stream 2" and it still
applies.

It doesn't say you need to wait for *all* candidates for one stream before
trickling *any* candidate for another stream, though. This is only talking
about candidates "within the same foundation". It would be fine to legal
them in the order "stream 1 host", "stream 2 host", "stream 1 srflx",
"stream 2 srflx".

On Thu, Mar 1, 2018 at 6:01 AM, Christer Holmberg <
[email-masked]> wrote:

> But, the text also talks about “streams”. I don’t think you need to wait
> until all candidates for one stream have been trickled until you trickle
> the candidates for another stream.
>
> Also, the SDP example seem to assume that the order of SDP candidate
> attributes matter. But, in SDP the order does not matter, and
> intermediaries could even change the order between parsing and incoming and
> generating an outgoing message…
>
> Regards,
>
> Christer
>
> From: Taylor Brandstetter <[email-masked]>
> To: Christer Holmberg <[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> That's how I interpret it; the example below explains how an RTCP
> candidate for a particular foundation must be conveyed after (or
> simultaneously with) the RTP candidate.
>
> Trickling candidates in a specific order helps guarantee that pairs are
> formed in the same order by both peers, so that they'll perform
> connectivity checks for the same pairs simultaneously.
>
> Though given the updates to section 8.1.1 (Inserting a New Pair in a Check
> List), this may not be as important as it used to be. The rules in 8.1.1
> guarantee that the "topmost" pair for each foundation will always be
> unfrozen. So I believe if candidates were trickled out of order, the worst
> that could happen is that one peer ends up with more pairs unfrozen than
> the other. For example:
>
> 3. Alice receives Bob's candidates, forming RTCP pair first (unfreezing
> it), then RTP pair (unfreezing it too, since it becomes the new "topmost"
> pair).
> On Tue, Feb 27, 2018 at 11:13 PM, Christer Holmberg <
> [email-masked]> wrote:
>
>> Hi,
>>
>> [email-masked]

---

From: Taylor Brandstetter <[email-masked]>
Date: Thu, 1 Mar 2018 09:05:37 -0800

(Meant to say "fine to *trickle* them"...)

On Thu, Mar 1, 2018 at 9:04 AM, Taylor Brandstetter <[email-masked]>
wrote:

> It doesn't say you need to wait for *all* candidates for one stream
> before trickling *any* candidate for another stream, though. This is only
> talking about candidates "within the same foundation". It would be fine to
> legal them in the order "stream 1 host", "stream 2 host", "stream 1 srflx",
> "stream 2 srflx".
>
>> Though given the updates to section 8.1.1 (Inserting a New Pair in a
>> Check List), this may not be as important as it used to be. The rules in
>> 8.1.1 guarantee that the "topmost" pair for each foundation will always be

---

From: Christer Holmberg <[email-masked]>
Date: Thu, 1 Mar 2018 17:26:05 +0000

Hi,

>(Meant to say "fine to trickle them"...)

I've fixed the text below :)

>I don’t think you need to wait until all candidates for one stream have been trickled until you trickle the candidates for another stream.
>
>What makes "streams" different than "components" in this regard? 
>
>Both determine what pair is unfrozen first. You can take the example above and replace "RTP" with "stream 1" and "RTCP" with "stream 2" and it still applies.

If you get a candidate for component 2, you KNOW that you at some point will get a candidate for component 1  (otherwise things won't work), so you can wait for that.

But, you don't know how many candidates you are going to get for each stream, so I don't see how you would determine whether to trickle a candidate or not based on what is happening on other streams. Also, the text doesn't say how/when you are supposed to wait when it comes to streams.

>It doesn't say you need to wait for all candidates for one stream before trickling any candidate for another stream, though. This is only talking about candidates "within the >same foundation". It would be fine to trickle them in the order "stream 1 host", "stream 2 host", "stream 1 srflx", "stream 2 srflx".

Assume the agent gets "stream 1 host", "stream 2 host" and "stream 2 srflx". Should it wait for "stream 1 srflx" before trickling "stream 2 srflx"? How does the agent even know that it will get "stream 1 srflx"?

My point is that is if you have the following text:

   "achieve this, when trickling candidates, agents MUST respect the
   order in which the components and streams appear (implicitly or
   explicitly) as they have been negotiated by means of the relevant
   candidate information."

...you should explain how it impacts streams. Currently the text only talks about components.

Regards,

Christer

On Thu, Mar 1, 2018 at 6:01 AM, Christer Holmberg <[email-masked]> wrote:
Hi,

Regards,

Christer

On Tue, Feb 27, 2018 at 11:13 PM, Christer Holmberg <[email-masked]> wrote:
Hi,

Regards,

Christer

---

From: Taylor Brandstetter <[email-masked]>
Date: Thu, 1 Mar 2018 10:52:19 -0800

>
> Assume the agent gets "stream 1 host", "stream 2 host" and "stream 2
> srflx". Should it wait for "stream 1 srflx" before trickling "stream 2
> srflx"?

That was my interpretation.

> How does the agent even know that it will get "stream 1 srflx"?

If one candidate of a particular foundation was gathered successfully, the
agent can expect the rest of the candidates with that foundation to be
gathered successfully as well.

Again, I don't see the difference between streams and components here. You
could ask the same question of components: "How does the agent know it will
get 'RTP srflx' if it gets 'RTCP srflx'?"

> ...you should explain how it impacts streams. Currently the text only

Ah, so is the issue that the next sentence only mentions components?

>    Therefore candidates for a given component
>    MUST NOT be conveyed prior to candidates for a component with a lower
>    ID number within the same foundation.

Streams are mentioned further below:

   Similar considerations apply at the level of media streams in
>    addition to foundations; this is covered by the requirement to always
>
>    start unfreezing candidates starting from the first media stream as
>
>    described under Section 5.2 <https://tools.ietf.org/html/draft-ietf-ice-trickle-17#section-5.2>.
>
>
But this appears out of date. Section 5.2 used to say:

If [the first]
> checklist is still empty however, agents will hold off further
> processing until this is no longer the case.

.... Which ensures the first stream is always unfrozen first. But now it
says:

   If that first component of the first media stream does
>    not contain candidates for one or more of the currently known pair
>    foundations, and if candidate pairs already exist for that foundation
>    in one of the following components or media streams, then the agent
>    unfreezes the first of those candidate pairs.

Meaning section 5.2 no longer enforces the stream ordering requirement.

So, I'd suggest:

   - Removing the outdated "Similar considerations apply..." paragraph.
   - Updating the rest of Section 16 to make it clear that candidates
   should be trickled in both stream and component order.

On Thu, Mar 1, 2018 at 9:26 AM, Christer Holmberg <
[email-masked]> wrote:

> >Both determine what pair is unfrozen first. You can take the example
> above and replace "RTP" with "stream 1" and "RTCP" with "stream 2" and it
> still applies.
>
> If you get a candidate for component 2, you KNOW that you at some point
> will get a candidate for component 1  (otherwise things won't work), so you
> can wait for that.
>
> But, you don't know how many candidates you are going to get for each
> stream, so I don't see how you would determine whether to trickle a
> candidate or not based on what is happening on other streams. Also, the
> text doesn't say how/when you are supposed to wait when it comes to streams.
>
> >It doesn't say you need to wait for all candidates for one stream before
> trickling any candidate for another stream, though. This is only talking
> about candidates "within the >same foundation". It would be fine to trickle
> Assume the agent gets "stream 1 host", "stream 2 host" and "stream 2
> srflx". Should it wait for "stream 1 srflx" before trickling "stream 2
> srflx"? How does the agent even know that it will get "stream 1 srflx"?
>

---

From: Christer Holmberg <[email-masked]>
Date: Thu, 1 Mar 2018 19:19:48 +0000

Hi,

>> Assume the agent gets "stream 1 host", "stream 2 host" and "stream 2 srflx". Should it wait for "stream 1 srflx" before trickling "stream 2 srflx"?
>
> If one candidate of a particular foundation was gathered successfully, the agent can expect the rest of the 
> candidates with that foundation to be gathered successfully as well.

I don't think there is any rule saying that, for a given foundation, you will get candidates for all streams (see example in section 6.1.2.6. of 5245bis). For example, perhaps you gather relay candidates only for the audio streams, because you want to use a relay for your video (you rather don't use video, if it won't work without a relay).

> Again, I don't see the difference between streams and components here. You could ask the same question of components: "How 
> does the agent know it will get 'RTP srflx' if it gets 'RTCP srflx'?"

You don't know - unless trickle mandates it (5245bis does not mandate it, as far as I know).

> Ah, so is the issue that the next sentence only mentions components?
 
Correct.

>   described under Section 5.2.
>
> .... Which ensures the first stream is always unfrozen first. But now it says:
>
> • Removing the outdated "Similar considerations apply..." paragraph.
> • Updating the rest of Section 16 to make it clear that candidates should be trickled in both stream and component order.

You would have to explain what "stream order" really means. Because, it does not mean that you first trickle all candidates for stream 1, then all candidates for stream 2, etc.

As I said above, you seem to suggest that, for each stream and each foundation, there will be the same set of candidates. There is no such requirement anywhere, as far as I know. 

Regards,

Christer

On Thu, Mar 1, 2018 at 9:26 AM, Christer Holmberg <[email-masked]> wrote:
Hi,

Regards,

Christer

Regards,

Christer

Regards,

Christer

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 1 Mar 2018 13:56:11 -0700

> As I said above, you seem to suggest that, for each stream and each foundation, there will be the same set of candidates. There is no such requirement anywhere, as far as I know. 

I suggest that we remove all references to the poorly defined concept of
"stream order". But if Emil thinks this is crucial then I'll defer to
him. (I did connect with him offlist and he promises to weigh in soon.)

Peter

---

From: Taylor Brandstetter <[email-masked]>
Date: Thu, 1 Mar 2018 13:18:11 -0800

>
> For example, perhaps you gather relay candidates only for the audio
> streams, because you want to use a relay for your video (you rather don't
> use video, if it won't work without a relay).
>

I agree there's nothing preventing this. In this case the ICE agent would
be able to trickle the audio relay candidate immediately.

> Again, I don't see the difference between streams and components here.
> You could ask the same question of components: "How

The ICE agent doesn't know what candidates it's attempting to gather? Or it
doesn't know whether the pending STUN transaction will fail?

You would have to explain what "stream order" really means.

It would be the order of the rows in section 8.1; ordered first by stream
then by component.

As I said above, you seem to suggest that, for each stream and each
> foundation, there will be the same set of candidates. There is no such
> requirement anywhere, as far as I know.
>

 I didn't mean to suggest that. Section 16 allows one foundation to have a
smaller set of candidates as long as they're trickled in order.

> "stream order".

So keep the component order but not stream order? Just because it's better
defined, or is there another reason?

On Thu, Mar 1, 2018 at 12:56 PM, Peter Saint-Andre <[email-masked]>
wrote:

> > I don't think there is any rule saying that, for a given foundation, you
> will get candidates for all streams (see example in section 6.1.2.6. of
> 5245bis). For example, perhaps you gather relay candidates only for the
> audio streams, because you want to use a relay for your video (you rather
> don't use video, if it won't work without a relay).
> >
> >> Again, I don't see the difference between streams and components here.
> You could ask the same question of components: "How
> >> • Updating the rest of Section 16 to make it clear that candidates
> > You would have to explain what "stream order" really means. Because, it
> does not mean that you first trickle all candidates for stream 1, then all
> candidates for stream 2, etc.
> >
> > As I said above, you seem to suggest that, for each stream and each
> foundation, there will be the same set of candidates. There is no such
> requirement anywhere, as far as I know.
>

---

From: Christer Holmberg <[email-masked]>
Date: Thu, 1 Mar 2018 21:23:49 +0000

Hi,

>>> If one candidate of a particular foundation was gathered 
>>> successfully, the agent can expect the rest of the candidates with that foundation to be gathered successfully as well.
>> 
>> I don't think there is any rule saying that, for a given foundation, you will get candidates for all streams 
>> (see example in section 6.1.2.6. of 5245bis). For example, perhaps you gather relay candidates only for the 
>> audio streams, because you want to use a relay for your video (you rather don't use video, if it won't work without a relay).
>> 
>>> Again, I don't see the difference between streams and components 
>>> here. You could ask the same question of components: "How does the agent know it will get 'RTP srflx' if it gets 'RTCP srflx'?"
>> 
>>>    MUST NOT be conveyed prior to candidates for a component with a 
>>>    lower ID number within the same foundation.
>>>
>>> If [the first] checklist is still empty however, agents will hold off further 
>> You would have to explain what "stream order" really means. Because, it does not mean that you first trickle 
>>all candidates for stream 1, then all candidates for stream 2, >> etc.
>> 
>> As I said above, you seem to suggest that, for each stream and each foundation, there will be the same set of 
>> candidates. There is no such requirement anywhere, as far as I know. 
>
> I suggest that we remove all references to the poorly defined concept of "stream order". But if Emil thinks this is crucial 
> then I'll defer to him. (I did connect with him offlist and he promises to weigh in soon.)

Maybe there is something important behind it, but we need to understand what it means.

MAYBE it means that, throughout all streams, all foundations are trickled in the same order (and for each foundation the components are trickled in the same order). But, again, that assumes that you actually know what foundations you are going to have for each stream when you start, and I'm not sure we can make such assumption.

Regards,

Christer

---

From: Christer Holmberg <[email-masked]>
Date: Thu, 1 Mar 2018 21:40:03 +0000

Hi,

>> For example, perhaps you gather relay candidates only for the audio streams, because you don't want to use 
>> a relay for your video (you rather don't use video, if it won't work without a relay).
>
> I agree there's nothing preventing this. In this case the ICE agent would be able to trickle the audio relay candidate immediately.

Yes, but in this case it is assumed that the ICE agent knows that it will never get a relay candidate for video (or, if it gets one, it will not trickle it).

>> The ICE agent doesn't know what candidates it's attempting to gather? Or it doesn't know whether the pending STUN transaction will fail?

The ICE agent does not know what candidates it will eventually get. Maybe it can indicate (using an API or configuration) what it wants, maybe it will get whatever is available, but I don't think we can assume that it knows what it will eventually get. 

>It would be the order of the rows in section 8.1; ordered first by stream then by component. 

If you are using SDP, the initial offer will obviously order the streams (m- lines) in a certain order. 

But, when you then start to trickle, I don't understand what "stream order" means. Assume the agent gets candidates for only one stream. How do you trickle those in "stream order"? Again, if you know that you are going to get the same type of candidates also for the "previous" streams, you can wait. But if you don't know? How long will you wait until you trickle?

Regards,

Christer

On Thu, Mar 1, 2018 at 12:56 PM, Peter Saint-Andre <[email-masked]> wrote:

Peter

---

From: Taylor Brandstetter <[email-masked]>
Date: Thu, 1 Mar 2018 14:54:35 -0800

>
> The ICE agent does not know what candidates it will eventually get.
>

I think this is the root of our misunderstanding. But can you clarify, do
you mean that the ICE agent doesn't know what candidates it will *attempt* to
gather? Or that it doesn't know whether a pending STUN transaction will
fail?

If you're talking about the former case, then I think we have a different
concept of what the "ICE agent" is. I interpreted the ICE agent as being
responsible for candidate gathering (including deciding which candidates it
will attempt to gather).

If you're talking about the latter case, then I agree there's a question of
"how long do I wait for the pending STUN transaction before giving up and
trickling the candidates I have?" Which I assumed would be an
implementation choice.

On Thu, Mar 1, 2018 at 1:40 PM, Christer Holmberg <
[email-masked]> wrote:

> streams, because you don't want to use
> > I agree there's nothing preventing this. In this case the ICE agent
> would be able to trickle the audio relay candidate immediately.
>
> Yes, but in this case it is assumed that the ICE agent knows that it will
> never get a relay candidate for video (or, if it gets one, it will not
> trickle it).
>
> >> The ICE agent doesn't know what candidates it's attempting to gather?
> Or it doesn't know whether the pending STUN transaction will fail?
>
> The ICE agent does not know what candidates it will eventually get. Maybe
> it can indicate (using an API or configuration) what it wants, maybe it
> will get whatever is available, but I don't think we can assume that it
> knows what it will eventually get.
>
> But, when you then start to trickle, I don't understand what "stream
> order" means. Assume the agent gets candidates for only one stream. How do
> you trickle those in "stream order"? Again, if you know that you are going
> to get the same type of candidates also for the "previous" streams, you can
> wait. But if you don't know? How long will you wait until you trickle?
>
> Regards,
>
> Christer
>
>
>
>

---

From: Christer Holmberg <[email-masked]>
Date: Fri, 2 Mar 2018 07:36:48 +0000

Hi,

>>ICE agent does not know what candidates it will eventually get.
>
>you mean that the ICE agent doesn't know what candidates it will
>attempt to gather? Or that it doesn't know whether a pending STUN
>transaction will fail?

The agent probably knows what candidates it TRIES to gather. But, as you
say, a pending STUN transaction may fail. The STUN/TURN server might be
overloaded, or out-of-service for some other reason.

> If you're talking about the former case, then I think we have a
>different concept of what the "ICE agent" is. I interpreted the ICE agent
>as being responsible for candidate gathering (including deciding which
>candidates it will attempt to gather).
>
> If you're talking about the latter case, then I agree there's a question
>of "how long do I wait for the pending STUN transaction before giving up
>and trickling the candidates I have?" Which I assumed would be an
>implementation choice.

Perhaps, but then we need to document it. Based on our discussion, I
assume ³stream order" actually means ³foundation order throughout all
streams². I.e., you need to trickle all available f1 candidates before you
trickle any f2 candidates, before you trickle any f3 candidates, etc etc.

Regards,

Christer

On Thu, Mar 1, 2018 at 1:40 PM, Christer Holmberg
<[email-masked]> wrote:

Hi,

>It would be the order of the rows in section 8.1; ordered first by stream

If you are using SDP, the initial offer will obviously order the streams
(m- lines) in a certain order.

But, when you then start to trickle, I don't understand what "stream
order" means. Assume the agent gets candidates for only one stream. How do
you trickle those in "stream order"? Again, if you know that you are going
to get the same type of candidates also
 for the "previous" streams, you can wait. But if you don't know? How long
will you wait until you trickle?

Regards,

Christer

On Thu, Mar 1, 2018 at 12:56 PM, Peter Saint-Andre <[email-masked]>
wrote:
On 3/1/18 12:19 PM, Christer Holmberg wrote:
>audio streams, because you want to use
 a relay for your video (you rather don't use video, if it won't work
without a relay).
>> € Removing the outdated "Similar considerations apply..." paragraph.
>> € Updating the rest of Section 16 to make it clear that candidates
>does not mean that you first trickle all candidates for stream 1, then
>all candidates for stream 2, etc.
>

Peter

---

From: Taylor Brandstetter <[email-masked]>
Date: Fri, 2 Mar 2018 10:29:19 -0800

>
> Based on our discussion, I

It would be the other way around: the order of the rows in Section 8.1.1,
not the columns. Pairs for different foundations are meant to be attempted
simultaneously.

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 20 Mar 2018 20:30:27 -0600

> trickle any f2 candidates, before you trickle any f3 candidates, etc etc.

While puzzling over this topic, I noticed that Section 5 of
draft-ietf-ice-trickle-17 says:

   A Trickle ICE agent initially considers all candidate pairs in all
   check lists to be frozen.  It then inspects the first check list and
   attempts to unfreeze all candidate pairs it has received so far that
   belong to the first component on the first media stream (i.e., the
   first media stream that was reported to the ICE implementation from
   the using application).  If that first component of the first media
   stream does not contain candidates for one or more of the currently
   known pair foundations, and if candidate pairs already exist for that
   foundation in one of the following components or media streams, then
   the agent unfreezes the first of those candidate pairs.

By contrast, Section 16 says:

   To achieve this, when trickling candidates, agents MUST respect the
   order in which the components and streams appear (implicitly or
   explicitly) as they have been negotiated by means of the relevant
   candidate information.  Therefore candidates for a given component
   MUST NOT be conveyed prior to candidates for a component with a lower
   ID number within the same foundation.  In addition, candidates MUST
   be paired, following the procedures in Section 8.1.1, in the same
   order they are conveyed.

The concepts of "stream order" here don't exactly line up - in Section 5
the relevant consideration is the order of reporting streams from the
using application to the ICE agent, whereas in Section 16 it's the order
of negotiating streams between the local and remote ICE agents (perhaps
from the inititiating agent to the receiving agent?).

My inclination is still to remove the concept of stream order, unless
Emil convinces us it is needed.

Peter