Subject: [Wish] Initial comments on draft-murillo-whip-02

From: Christer Holmberg <[email-masked]>
Date: Tue, 3 Aug 2021 19:16:39 +0000

Hi,

I read the draft, and I have some comments.

As the draft is still in an early stage I will skip pure editorial comments, unless the text is unclear.

General:
=======

Q_GEN_1:

The motivation behind WHIP seems to be broadcasting.

But, unless I've missed something, what the draft does is defining how to do SDP O/A using HTTPS. So, it could be used for any SDP O/A use-case. It doesn't even have to be WebRTC, as long as the client and server support the required extensions etc.

Section 1:
=======

Q_SEC-1_1:

The text says:

   "WebRTC intentionally does not specify a signaling transport protocol
   at application level, while RTCWEB standardized the signalling
   protocol itself (JSEP, SDP O/A) and everything that was going over
   the wire (media, codec, encryption, ...)."

RTCWEB did not standardize the signaling protocol.

Q_SEC-1_2:

The text says:

   "RTSP, which is based on RTP and maybe the closest in terms of features to webrtc, is not compatible with WebRTC
   SDP offer/answer model."

What is "the WebRTC offer/answer model"?

Q_SEC-1_3:

The text says:

   "This document proposes a simple protocol for supporting WebRTC as
   ingest method which is:

   ...

   o  Fully compliant with Webrtc and RTCWEB specs."

Based on some of the suggested restrictions, related to e.g., SDP O/A transactions (only allowing the initial O/A), ICE (related to trickle and restart), and the SDP setup attribute misalignment (allowing setup:active in the initial offer) etc, I don't think that is "fully compliant". Or, if I have misunderstood, what exactly are you referring to?

Section 4:
========

Q_SEC-4_1:

The text talks about usage of the Location header field in the 201 response. But, I assume usage is optional, and that a server can choose to use the WISH endpoint for the whole session?

Section 4.1:
==========

Q_SEC-4-1_1:

The text says:

   "In order to simplify the protocol, there is no support for exchanging
   gathered trickle candidates from media server ICE candidates once the
   SDP answer is sent.  So in order to support the WHIP client behind
   NAT, the WHIP media server SHOULD be publicly accessible."

Just because the server doesn't support trickle, it doesn't mean it has to be publicly accessible. Not supporting trickle just means that the server needs to collect all candidates before sending the answer.

Q_SEC-4-1_2:

The text says:

   "A WHIP client receiving a 405 response for an HTTP PATCH request
   SHALL not send further request for ICE trickle or restart.  If the
   WHIP client gathers additional candidates (via STUN/TURN) after the
   SDP offer is sent, it MUST send STUN request to the ICE candidates
   received from the media server as per [RFC8838] regardless if the
   HTTP PATCH is supported by either the WHIP client or the WHIP
   resource."

This sounds really strange.

The text says that if the client receives 405, because the WHIP resource does not support trickle, the client can still trickle candidates per RFC 8838 (by sending STUN requests from the new candidates).

Creating peer-reflexive candidates by sending STUN requests from NEW candidates, even if trickle isn't used, sounds like a hack to me.

Regards,

Christer

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 23 Aug 2021 09:07:14 +0200

Hi Christer, just got back from vacation, comments inline

El mar, 3 ago 2021 a las 21:16, Christer Holmberg (<christer.holmberg=
[email-masked]>) escribió:

> But, unless I’ve missed something, what the draft does is defining how to
> do SDP O/A using HTTPS. So, it could be used for any SDP O/A use-case. It
> doesn’t even have to be WebRTC, as long as the client and server support
> the required extensions etc.
>
>
>

The SDP O/A is restricted to one audio and/or video track in sendonly,
indeed anyone could do an SDP O/A in an HTTP POST with whatever semantics
they want, but the WISH wg is chartered to specifically cover the ingestion
use case only.

>
> Section 1:
>
> =======
>
>
>
> Q_SEC-1_1:
>
>
>
> The text says:
>
>
>
>    “WebRTC intentionally does not specify a signaling transport protocol
>
>    the wire (media, codec, encryption, ...).”
>
>
>

You are right,  adding a github issue to track and solve it.
https://github.com/murillo128/webrtc-http-ingest-protocol/issues/12

>
> Q_SEC-1_2:
>
>
>
> The text says:
>
>
>
>    “RTSP, which is based on RTP and maybe the closest in terms of features
> to webrtc, is not compatible with WebRTC
>
>    SDP offer/answer model.”
>
>
>
> What is “the WebRTC offer/answer model”?
>
>
>

This seems editorial to me, do you have an alternative wording for this?

>
> Q_SEC-1_3:
>
>
>
> The text says:
>
>
>
>    “This document proposes a simple protocol for supporting WebRTC as
>
>    ingest method which is:
>
>
>
>    …
>
>
>
>    o  Fully compliant with Webrtc and RTCWEB specs.”
>
>
>
> Based on some of the suggested restrictions, related to e.g., SDP O/A
> transactions (only allowing the initial O/A), ICE (related to trickle and
> restart), and the SDP setup attribute misalignment (allowing setup:active
> in the initial offer) etc, I don’t think that is “fully compliant”. Or, if
> I have misunderstood, what exactly are you referring to?
>
>
>

That any webrtc compliant endpoint can implement WHIP.

>
>
> Section 4:
>
> ========
>
>
>
> Q_SEC-4_1:
>
>
>
> The text talks about usage of the Location header field in the 201
> response. But, I assume usage is optional, and that a server can choose to
> use the WISH endpoint for the whole session?
>

No, it is not optional, the url of the created resource is returned in the
201 response and it must be used for doing the ice trickle and termination
request. Nothing prevents the server from using the same url as the whip
endpoint one.

>
>
>
>
> Section 4.1:
>
> ==========
>
>
>
>
>
> Q_SEC-4-1_1:
>
>
>
> The text says:
>
>
>
>    “In order to simplify the protocol, there is no support for exchanging
>
>    NAT, the WHIP media server SHOULD be publicly accessible.”
>
>
>
> Just because the server doesn’t support trickle, it doesn’t mean it has to
> be publicly accessible. Not supporting trickle just means that the server
> needs to collect all candidates before sending the answer.
>
>
>

We can remove the  last sentence as it is superfluous, adding a new github
issue
https://github.com/murillo128/webrtc-http-ingest-protocol/issues/13

>
> Q_SEC-4-1_2:
>
>
>
> The text says:
>
>
>
>    “A WHIP client receiving a 405 response for an HTTP PATCH request
>
>    resource.”
> The text says that if the client receives 405, because the WHIP resource
> does not support trickle, the client can still trickle candidates per RFC
> 8838 (by sending STUN requests from the new candidates).
>
>
>
> Creating peer-reflexive candidates by sending STUN requests from NEW
> candidates, even if trickle isn’t used, sounds like a hack to me.
>

That was already agreed to be removed. ICE lite servers would just need to
ignore the signaled candidates on the HTTP PATCH requests and reply only to
the incoming stun requests from the new candidates.

Best regards
Sergio

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 25 Aug 2021 20:34:14 +0000

Hi,

See inline.
 
General:
=======
 
Q_GEN_1:
 
>> But, unless I’ve missed something, what the draft does is defining how to do SDP O/A using HTTPS. So, it could be used for any SDP O/A use-case. It doesn’t even have to be WebRTC, as long as the client and server support the required extensions etc.
>
> The SDP O/A is restricted to one audio and/or video track in sendonly, indeed anyone could do an SDP O/A in an HTTP POST with whatever semantics they want, but the WISH wg is chartered to specifically cover the ingestion use case only.

Maybe it would be good to say that the mechanism can be enhanced for other usages too, but those are outside the scope of the document.

Also, if you are not planning on using the WebRTC data channel I think it would be good to explicitly indicate that.

Section 1:
=======
 
Q_SEC-1_1: 
 
>https://protect2.fireeye.com/v1/url?k=772ed376-28b5ea32-772e93ed-861d41abace8-803b50005df852cb&q=1&e=d10bad7b-9658-47e5-8a4e-85689e02a197&u=https%3A%2F%2Fgithub.com%2Fmurillo128%2Fwebrtc-http-ingest-protocol%2Fissues%2F12

Thanks!

Q_SEC-1_2: 
 
>> The text says:
>> 
>>   “RTSP, which is based on RTP and maybe the closest in terms of features to webrtc, is not compatible with WebRTC
>>   SDP offer/answer model.”
>> 

I assume what you want to say is that the features used by WebRTC have not been defined for RTSP. But, it is not only about offer/answer, it is also about e.g., ICE etc.

I guess it would also be good to say "which is ALSO based on RTP".

Q_SEC-1_3: 
 
>> Based on some of the suggested restrictions, related to e.g., SDP O/A transactions (only allowing the initial O/A), ICE (related to trickle and restart), and the SDP setup attribute
>> misalignment (allowing setup:active in the initial offer) etc, I don’t think that is “fully compliant”. Or, if I have misunderstood, what exactly are you referring to?
> 

I think it would be better to say that WHIP is designed in order to be compliant with WebRTC, but that a set of restrictions apply. 

Section 4:
========
 
Q_SEC-4_1:
 
> No, it is not optional, the url of the created resource is returned in the 201 response and it must be used for doing the ice trickle and termination request. Nothing prevents the server from using the same url as the whip endpoint one.

Isn´t that the implicit meaning if the Location header is not present?

Section 4.1:
==========

Q_SEC-4-1_1: 
 
>> Just because the server doesn’t support trickle, it doesn’t mean it has to be publicly accessible. Not supporting trickle just 
>> means that the server needs to collect all candidates before sending the answer.
> 
> We can remove the  last sentence as it is superfluous, adding a new github issue
> https://protect2.fireeye.com/v1/url?k=26929152-7909a816-2692d1c9-861d41abace8-717a11d6e023f0d4&q=1&e=d10bad7b-9658-47e5-8a4e-
> 85689e02a197&u=https%3A%2F%2Fgithub.com%2Fmurillo128%2Fwebrtc-http-ingest-protocol%2Fissues%2F13

Also perhaps saying something like "no support for exchanging further candidates (ICE trickle)"

Q_SEC-4-1_2: 
 
>> Creating peer-reflexive candidates by sending STUN requests from NEW candidates, even if trickle isn’t used, sounds like a hack to me.
>
> That was already agreed to be removed. ICE lite servers would just need to ignore the signaled candidates on the HTTP PATCH requests and reply only to the incoming stun requests from the new candidates.
 
Ok.

I have a couple of more generic comments.

Q_GEN_2:

As a frequent user of a broadcasting/ingestion application myself, once the connection has been established, I am able to send at least START, STOP, PAUS and RESUME commands. In case of sport events, you are also able to send commands regarding scores etc.

There doesn't seem to be any of that in WHIP. Instead, at least as far as the server is concerned, the media ingestion starts once the connection has been established, ends once the connection is terminated, and there is no way to exchange any kind of non-media data during the session. Perhaps that is good enough when teenagers want to ingest something with their mobile, but I doubt it will be useful for more professional usages. 

If that is true, I think it would be good to point out in the draft.

Q_GEN_3:

There is almost no text regarding WHY someone should use this mechanism, instead of existing ones. There is a sentence saying "and content providers still rely heavily on protocols like RTMP". Ok, so why is that a bad thing? Why should one use WHIP instead? Now it seems like the only reason is because it is an IETF standard, but I hope there is more than that.

Regards,

Christer

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Sun, 29 Aug 2021 12:26:22 +0200

El mié, 25 ago 2021 a las 22:34, Christer Holmberg (<
[email-masked]>) escribió:

> >> But, unless I’ve missed something, what the draft does is defining how
> to do SDP O/A using HTTPS. So, it could be used for any SDP O/A use-case.
> It doesn’t even have to be WebRTC, as long as the client and server support
> Maybe it would be good to say that the mechanism can be enhanced for other
> usages too, but those are outside the scope of the document.
>
>
The protocol is only targeting ingest, I think it is a bad idea to open the
possibility of enhancing it for other usages without explicting mentioning
them too. Implementing an SDP O/A over HTTP for the viewing side will have
not much similarities with current whip proposal to consider it an
enhancement, even if they share HTTP as a transport layer.

> Also, if you are not planning on using the WebRTC data channel I think it
> would be good to explicitly indicate that.
>

Nothing prevents an extension to include datachannel support in the future,
although it may be worth adding this as note.

>
>
>
> Q_SEC-1_2:
>
> >> The text says:
> >>
> >>   “RTSP, which is based on RTP and maybe the closest in terms of
> features to webrtc, is not compatible with WebRTC
> >>   SDP offer/answer model.”
> >>
> I assume what you want to say is that the features used by WebRTC have not
> been defined for RTSP. But, it is not only about offer/answer, it is also
> about e.g., ICE etc.
>

Seems fine to me.

> restart), and the SDP setup attribute
> >> misalignment (allowing setup:active in the initial offer) etc, I don’t
> think that is “fully compliant”. Or, if I have misunderstood, what exactly
> are you referring to?
> >
> I think it would be better to say that WHIP is designed in order to be
> compliant with WebRTC, but that a set of restrictions apply.
>

Fine for me too

> > No, it is not optional, the url of the created resource is returned in
> the 201 response and it must be used for doing the ice trickle and
> termination request. Nothing prevents the server from using the same url as
> the whip endpoint one.
>

I would prefer to add as few optionalities to the protocol as possible. It
would be trivial for the server to just return the current URL and not to
have to test each interoperability with each client for both when the
location header is present or not.

> >> Just because the server doesn’t support trickle, it doesn’t mean it has
> to be publicly accessible. Not supporting trickle just

Looks good to me

> Q_GEN_2:
>
> As a frequent user of a broadcasting/ingestion application myself, once
> the connection has been established, I am able to send at least START,
> STOP, PAUS and RESUME commands. In case of sport events, you are also able
> to send commands regarding scores etc.
>
> There doesn't seem to be any of that in WHIP. Instead, at least as far as
> the server is concerned, the media ingestion starts once the connection has
> been established, ends once the connection is terminated, and there is no
> way to exchange any kind of non-media data during the session. Perhaps that
> is good enough when teenagers want to ingest something with their mobile,
> but I doubt it will be useful for more professional usages.
>
In terms of features, the WHIP protocol matches the ones offered by RTMP
which is still widely used and offered on most profesional hardware
encoders and main open source broadcasting software (i.e.
ffmpeg/OBS/vmixer/...). Not sure how that is not suited for "professional
usages".

Regarding the metadata exchanged during the session, which could be done
either by datachannel or by a new whip protocol extension with another http
url, the main problem is about standardizing the metadata that could be
used widely and implemented  in a service agnostic way.

> Q_GEN_3:
>
> There is almost no text regarding WHY someone should use this mechanism,
> instead of existing ones. There is a sentence saying "and content providers
> still rely heavily on protocols like RTMP". Ok, so why is that a bad thing?
> Why should one use WHIP instead? Now it seems like the only reason is
> because it is an IETF standard, but I hope there is more than that.
>
> We could add more info about the benefits of using webrtc end to end.

Best regards
Sergio

---

From: Juliusz Chroboczek <[email-masked]>
Date: Sun, 29 Aug 2021 22:39:42 +0200

>     As a frequent user of a broadcasting/ingestion application myself,
>     once the connection has been established, I am able to send at least
>     START, STOP, PAUSE and RESUME commands.

Could you please clarify the requirement?  I'd like to understand whether
STOP is different from closing the PeerConnection, and whether PAUSE is
different from muting all tracks (setting MediaStreamTrack.enabled to
false in the JavaScript API)?

>     In case of sport events, you are also able to send commands
>     regarding scores etc.

Looks like a can of worms to me.  Once we start doing that, we might end
up adding a full chat protocol.

-- Juliusz

---

From: Christer Holmberg <[email-masked]>
Date: Mon, 30 Aug 2021 17:49:09 +0000

Hi,

General:
=======

Q_GEN_1:

> The protocol is only targeting ingest, I think it is a bad idea to open the possibility of enhancing it for other usages
> without explicting mentioning them too. Implementing an SDP O/A over HTTP for the viewing side will have not much similarities
> with current whip proposal to consider it an enhancement, even if they share HTTP as a transport layer.
>>
> Nothing prevents an extension to include datachannel support in the future, although it may be worth adding this as note.

I think that would be good, because I am sure the question would come up.

---

Q_SEC-1_2:

>> I assume what you want to say is that the features used by WebRTC have not been defined for RTSP.
>> But, it is not only about offer/answer, it is also about e.g., ICE etc.
>>

---

>> I think it would be better to say that WHIP is designed in order to be compliant with WebRTC, but that a set of restrictions apply.

---

Section 4:
========

Q_SEC-4_1:

>>>> The text talks about usage of the Location header field in the 201 response. But, I assume usage is optional,
>>>> and that a server can choose to use the WISH endpoint for the whole session?
>>>
>>> No, it is not optional, the url of the created resource is returned in the 201 response and it must be used
>>> for doing the ice trickle and termination request. Nothing prevents the server from using the same url as
>>> the whip endpoint one.
>>
> I would prefer to add as few optionalities to the protocol as possible. It would be trivial for the server
> to just return the current URL and not to have to test each interoperability with each client for both
> when the location header is present or not.

I just think it is not a good idea to make changes to core HTTP, so in my opinion it is more about HTTP interoperability than WHIP interoperability.

And, what if you are using an HTTP library that will remove the header if the endpoint isn't changed?

---

Section 4.1:
==========

>> Also perhaps saying something like "no support for exchanging further candidates (ICE trickle)"

---

Q_GEN_2:

>> As a frequent user of a broadcasting/ingestion application myself, once the connection has been
>> established, I am able to send at least START, STOP, PAUS and RESUME commands. In case of sport
>> events, you are also able to send commands regarding scores etc.
>>
>> There doesn't seem to be any of that in WHIP. Instead, at least as far as the server is concerned,
>> the media ingestion starts once the connection has been established, ends once the connection is
>> terminated, and there is no way to exchange any kind of non-media data during the session. Perhaps
>> that is good enough when teenagers want to ingest something with their mobile, but I doubt it will
>> be useful for more professional usages.

> In terms of features, the WHIP protocol matches the ones offered by RTMP which is still widely used
> and offered on most profesional hardware encoders and main open source broadcasting software
> (i.e. ffmpeg/OBS/vmixer/...). Not sure how that is not suited for "professional usages".

RTMP supports PAUSE, RESUME etc commands, doesn't it? :)

Let me explain how the sport event ingestion software I am using works:

First you establish the session, choose what sports event you are going to stream etc.

After that, the media ingestion starts. But, the server does not start the broadcasting of the media yet. It only broadcasts the stream to the one doing the ingestion, so that he/she can check that everything works.

After that, the user sends a command to the server to start the actual broadcasting.

Etc.

So, the user has explicit control of when the actual broadcasting begins, even if the media ingestion has began earlier.

As I am not the implementer of the software I don't know all the protocol level details, other than it uses RTMP.

> Regarding the metadata exchanged during the session, which could be done either by datachannel
> or by a new whip protocol extension with another http url,

I had a datachannel in mind, and that's why I asked about it earlier :)

But, if commands are outside the scope of WHIP, I think it would be good to explcitly say it.

---

Q_GEN_3:

>> There is almost no text regarding WHY someone should use this mechanism, instead of existing ones.
>> There is a sentence saying "and content providers still rely heavily on protocols like RTMP". Ok,
>> so why is that a bad thing? Why should one use WHIP instead? Now it seems like the only reason is

I think that would be useful.

Regards,

Christer

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 30 Aug 2021 20:52:51 +0200

El lun, 30 ago 2021 a las 19:49, Christer Holmberg (<
[email-masked]>) escribió:

> > to just return the current URL and not to have to test each
> interoperability with each client for both
> I just think it is not a good idea to make changes to core HTTP, so in my
> opinion it is more about HTTP interoperability than WHIP interoperability.
>
Can you point me to the HTTP spec that defines that behaviour of the Link
attribute?

> which is still widely used

Hard to know as AMF commands on top of RTMP are quite undocumented. If you
do some archeology on action script reference you can see that the
play/pause are commands send from the player to the media server when
reproducing a video, they are not intended to be used for the publish
command:

https://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/net/NetStream.html#publish()

> After that, the media ingestion starts. But, the server does not start the
> broadcasting of the media yet. It only broadcasts the stream to the one
> doing the ingestion, so that he/she can check that everything works.
>
> So, the user has explicit control of when the actual broadcasting begins,
> even if the media ingestion has began earlier.
>
That workflow makes a lot of sense, but not sure how it maps the
pause/resume operation. Typically you will control it from a control panel
on the server side which is not at the encoder.

> either by datachannel
The way I see it, the problem again is defining and standardizing the
metadata, not about how to transport it. We can state that it is not
supported by this WHIP version.

Best regards
Sergio

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 31 Aug 2021 16:28:06 +0000

Hi,

>Can you point me to the HTTP spec that defines that behaviour of the Link attribute? 

I assume you mean Location header? I have not commented on any Link attribute :)

The semantics of the Location header, in a 201 (Created) response is that a NEW resource has been created. As far as I know, if a new resource is NOT created, HTTP does not mandate a Location header to be included in the 201 response.

I can look if this is explicitly defined somewhere, but if there is someone having better HTTP knowledge than myself he/she may want to chime in.

---

> Hard to know as AMF commands on top of RTMP are quite undocumented. If you do some archeology on action script reference you can see that the play/pause are commands send from the player to the media server when reproducing a video, they are not intended to be used for the publish command:
>
> https://protect2.fireeye.com/v1/url?k=972f57bf-c8b46e83-972f1724-86b1886cfa64-94d0ba57ffd7f0e9&q=1&e=6b5239de-60be-4332-8f7c-cc2e3371d5b9&u=https%3A%2F%2Fhelp.adobe.com%2Fen_US%2FFlashPlatform%2Freference%2Factionscript%2F3%2Fflash%2Fnet%2FNetStream.html%23publish%28%29

You may be right, and I haven't debugged any RTMP app in order to see what exactly is sent. Just because my app has a button that says "PAUSE" it doesn't mean it triggers an RTMP Pause command :) 

However, my main issue is not the specific PAUSE and RESUME commands, but the generic ability of being able to send commands in the first place. 

> That workflow makes a lot of sense, but not sure how it maps the pause/resume operation. Typically you will control it from a control panel on the server side which is not at the encoder. 

The above is probably a bad example for PAUSE/RESUME. In the app there is a "PAUSE/RESUME" button, but it does not affect the media ingestion and broadcasting. It only affects the sports game watch (which runs in the server).

But, the example shows the usage of commands for controlling different steps (ingest-no-broadcast, ingest-with-broadcast etc). In WHIP, as far as the server is concerned, the ingestion and broadcasting will start once the O/A and ICE procedure has concluded.

---

> The way I see it, the problem again is defining and standardizing the metadata, not about how to transport it. We can state that it is not supported by this WHIP version.

I was not thinking about standardizing all metadata, but more about a framework with e.g., an IANA registry where commands/meta-data can be registered.

BUT, having said all that, if the community thinks it's outside the scope of this WHIP version, I will not argue against that. I do think it would be good to document, though, because I think it is useful information for implementers evaluating different protocols etc.

Regards,

Christer

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 31 Aug 2021 16:35:09 +0000

Hi,

> Could you please clarify the requirement?  I'd like to understand whether STOP is different from closing the PeerConnection, and whether PAUSE is different from muting all tracks (setting MediaStreamTrack.enabled to false in the JavaScript API)?

Please discard the specific START, STOP etc commands. In my reply to Sergio I did give an example where I think commands are useful. But, instead of talking about specific commands, I think the question is whether sending of commands to begin with should be supported. If not, I think it is good to document.

> Looks like a can of worms to me.  Once we start doing that, we might end up adding a full chat protocol.

Unless we'd agree on a clear set of requirements, I agree there is a risk for that.

Regards,

Christer