Subject: [AVT] Comments on draft-boyaci-avt-app-sharing-00

From: Cullen Jennings <[email-masked]>
Date: Fri, 14 Nov 2008 18:52:03 -0700

I like the idea of an applications share protocol that can be set up  
over SIP and glad to see this submission.

Few comments

At a high level, why not just standardize something like VNC. It seems  
like this would be better than inventing something new.

I also wondered about why run it over RTP vs just be it's own  
protocol. It was not clear that RTP was a good match.

Some more small details ... you might want to look at how X deals with  
mouse and keyboard events - particularly compression of mouse motion.

Make sure that you have some way for the protocol to send a CTRL-ALT- 
DEL to the far side without the user actually having to simultaneously  
press CTRL, ALT and DEL.

You probably want your PLI to be able to specify a partial rectangle  
to update.

Cullen in my individual contributor roll

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 14 Nov 2008 21:25:16 -0500

Cullen,

thanks for your comments. Some quick responses below:

On Nov 14, 2008, at 8:52 PM, Cullen Jennings wrote:

> At a high level, why not just standardize something like VNC. It  
> seems like this would be better than inventing something new.

We have tried this (our SIP client [sipc] can invoke vnc) - vnc is  
difficult to integrate into a session-based system, has a rather  
primitive security mechanism and a very narrow remote entry facility,  
among other problems. We should probably create a note summarizing the  
issues with vnc.

> protocol. It was not clear that RTP was a good match.

That's a fairly fundamental discussion we should have in more detail  
soon... One could probably run most of RTP over just plain UDP (needed  
to support multicast, which is very useful for teaching situations) or  
TCP, but one would need many of the elements, such as payload  
discrimination.

Timing synchronization with audio and video is probably the largest  
reason to do this, but I'm ready to be convinced that either RTP  
should be an optional transport (in addition to, say, MSRP).

>
>
> Some more small details ... you might want to look at how X deals  
> with mouse and keyboard events - particularly compression of mouse  
> motion.
>
> DEL to the far side without the user actually having to  
> simultaneously press CTRL, ALT and DEL.

This seems potentially dangerous, albeit necessary for remote terminal  
operation. Not sure how vnc handles this, for example.

---

From: Omer Boyaci <[email-masked]>
Date: Fri, 14 Nov 2008 23:36:44 -0500

Hi Cullen,

Thanks for your comments. I want to mention some more issues with the  
VNC.

1) VNC uses a client-pull mechanism which results in poor performance  
in high RTT situations.
     Our proposal uses a server-push mechanism.

2) VNC uses the same encoding for all updates.
      This new system may use different encodings for updates in the  
same session.
      For example, if the user plays a movie, our implementation  
detects that and uses Theora or MJPEG just for this movie region.

3) VNC uses a separate compressor for each connected user. This  
results high CPU usage if more than one person connects.
      Also due to this compressor issue, one can not use these  
encodings in a multicast session (decompressor needs all the packets).
      TeleTeachingTool uses Hextile encoding (which means uncompressed  
data and bandwidth inefficiency) to overcome this limitation.

      Our proposed system compresses each update only once. The  
compressed updates are stored with metadata such as the top-left  
coordinates, width and height.
      Then ready-to-go updates are transmitted to participants. For  
low bandwidth clients, the system skips some updates if they are  
overwritten by the following updates.

-Omer Boyaci

---

From: Cullen Jennings <[email-masked]>
Date: Sun, 16 Nov 2008 06:59:18 -0600

Thanks for the answers from Henning and you - it makes more sense now  
- glad I asked.

On Nov 14, 2008, at 22:36 , Omer Boyaci wrote:

> Hi Cullen,
>
> Thanks for your comments. I want to mention some more issues with  
> the VNC.
>
> 1) VNC uses a client-pull mechanism which results in poor  
> performance in high RTT situations.
>> I like the idea of an applications share protocol that can be set  
>> up over SIP and glad to see this submission.
>>
>> Few comments
>>
>> Make sure that you have some way for the protocol to send a CTRL- 
>> ALT-DEL to the far side without the user actually having to  
>> You probably want your PLI to be able to specify a partial  
>> rectangle to update.
>>

---

From: Cullen Jennings <[email-masked]>
Date: Sun, 16 Nov 2008 07:01:59 -0600

On Nov 14, 2008, at 20:25 , Henning Schulzrinne wrote:

> This seems potentially dangerous, albeit necessary for remote  
> terminal operation. Not sure how vnc handles this, for example.

Obviously critical to be able to logon to most windows machines which  
is really important with all the virtualization stuff going on. VNC  
implementations did not initially care about this and the support is  
erratic and confusing. IMHO It's something that for better or worse  
needs to be designed in from the beginning.

---

From: Henning Schulzrinne <[email-masked]>
Date: Sun, 16 Nov 2008 11:12:07 -0500

I agree that this is needed. I suspect that this needs to be something  
that is transmitted when requested by a user interface, as just typing  
CTRL-ALT-DEL in the client window is unlikely to have the right  
effect. (There are probably other such keys, such as the Fxx keys,  
that are not handled by the in-focus window.)

On Nov 16, 2008, at 8:01 AM, Cullen Jennings wrote:

> Obviously critical to be able to logon to most windows machines  
> which is really important with all the virtualization stuff going  
> on. VNC implementations did not initially care about this and the  
> support is erratic and confusing. IMHO It's something that for  
> better or worse needs to be designed in from the beginning.
>
>
>

---

From: Omer Boyaci <[email-masked]>
Date: Sun, 16 Nov 2008 11:54:33 -0500

On Nov 16, 2008, at 11:12 AM, Henning Schulzrinne wrote:

> I agree that this is needed. I suspect that this needs to be  
> something that is transmitted when requested by a user interface, as  
> just typing CTRL-ALT-DEL in the client window is unlikely to have  
> the right effect. (There are probably other such keys, such as the  
> Fxx keys, that are not handled by the in-focus window.)
>

Exactly, there should be an element in the client GUI (like button or  
menu item) which sends these key combinations to the host.
The protocol itself supports all key combinations. This is an  
implementation detail.
The Client application must allow the client to generate these events  
somehow (Via menus, buttons, or other GUI elements).

---

From: "Roni Even" <[email-masked]>
Date: Sun, 16 Nov 2008 20:06:43 +0200

Hi,
I also share Cullen concerns about why using RTP.
To me it looks like you are using RTP as a transport for a protocol. But
note that RTP over UDP or multicast is not reliable.
The semantics of RTCP PLI you define is not according to the PLI semantics,
PLI is an indication from the receiver leaving the sender a free choice how
to handle it. If you want a request you either use FIR from the CCM feedback
RFC or define a different request.

As for the data, you suggest using ,multiple multicast for different
resolution, a better solution may be based on scalable video codec (SVC).
One of the problems in the past for using video for screen data was the low
resolution making the text unreadable, with SVC using HD resolution you can
get high quality.

Roni Even 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Cullen
Jennings
Sent: Saturday, November 15, 2008 3:52 AM
To: IETF AVT WG
Subject: [AVT] Comments on draft-boyaci-avt-app-sharing-00

Few comments

---

From: Omer Boyaci <[email-masked]>
Date: Sun, 16 Nov 2008 12:44:41 -0600

Hi Roni,

Thanks for your comments.

> RFC or define a different request.

Initially I was using FIR, but then thought that PLI is more generic.  
I will check the FIR, again.
We also want to give a choice to the sender how to handle new-comers.
Transmitting the full screen state as soon as getting a PLI may not be  
feasible due to bandwidth considerations.

> resolution making the text unreadable, with SVC using HD resolution  

The system does not use video for screen data.
We suggest to use PNG for the screen data and we submit an ID for png- 
rtp http://tools.ietf.org/html/draft-boyaci-avt-png-00.
The system uses a video codec only for the regions where a movie is  
playing.
For example, if the user is playing a movie in youtube, the system  
detects this and uses movie based encoding just for this region.
The rest of the screen will be transmitted via PNG.

The system detects (via mirror driver) and transmits only the changed  
regions of the screen.

-Omer

---

From: Colin Perkins <[email-masked]>
Date: Sat, 15 Nov 2008 15:39:20 -0600

> protocol. It was not clear that RTP was a good match.

This was the main issue raised back at IETF 62 when this was last  
discussed. The minutes from the AVT meeting back then state:

    draft-lennox-avt-app-sharing-00.txt
    draft-schulzrinne-mmusic-sharing-00.txt

    Henning Schulzrinne discussed some work his group has done on  
protocols
    for application and desktop sharing. This is not a work item of  
AVT but
    was presented for information, to determine if it might be  
appropriate
    to adopt as a future work item.

    The motivation for this work is to allow remote viewing and  
access of
    unmodified applications, without sharing application state. An  
aim is
    to integrate with the IETF session architecture, unlike the old T. 
120
    protocol. Henning outlined the requirements and summarised an  
initial
    protocol design outline using RTP to convey bitmap display output  
and
    keystroke/mouse input.

    After a number of questions for clarification, there was  
considerable
    discussion of the "big picture" open issues. Carsten Bormann  
wondered
    if this is now the right time to solve the problem, and if AVT is  
the
    right home for the work? Carsten was unconvinced by the arguments  
for
    using RTP: there are definitely some requirements for real-time  
data,
    but general remote access is not always perceived as a real-time
    application (indeed, the protocol uses RTP over TCP). The landscape
    of requirments varies: some applications have latency and timing
    constraints, particularly for input; others do not. It is not clear
    that the appropriate trade-off has been made, or that we have a good
    handle on the problem space. Carsten also noted that different types
    of operating system have become more similar over time, so the  
problem
    might be easier than when previous solutions were last developed,  
and
    that now might be a good time to attempt a new solution.

    Keith Lantz expressed similar concerns: this is clearly an important
    problem to solve, but he is not sure this is the right  
architecture and
    AVT may not be the right home. Perhaps we need to better  
understand the
    architecture to know if this is the appropriate solution?

    Some interest was also expressed in scaling one small screen  
devices,
    although it is unclear how that use case fits into this model.

That previous work wasn't adopted in AVT, due to those concerns, and  
I don't see anything in the current draft that addresses them either.

-- 
Colin Perkins
http://csperkins.org/

---

From: Stephen Casner <[email-masked]>
Date: Sun, 16 Nov 2008 19:40:56 -0800 (PST)

On Fri, 14 Nov 2008, Omer Boyaci wrote:

>     same session.  For example, if the user plays a movie, our
>     implementation detects that and uses Theora or MJPEG just for
>     this movie region.

There are multiple implementations of VNC that use different
encodings.

                                                        -- Steve

---

From: Henning Schulzrinne <[email-masked]>
Date: Sun, 16 Nov 2008 22:59:27 -0500

At the same time, i.e., within the same session?

---

From: Randell Jesup <[email-masked]>
Date: Tue, 18 Nov 2008 11:57:05 -0500

>    draft-schulzrinne-mmusic-sharing-00.txt
...
>    After a number of questions for clarification, there was  considerable
>    discussion of the "big picture" open issues. Carsten Bormann  wondered
>    if this is now the right time to solve the problem, and if AVT is  the
>    right home for the work? Carsten was unconvinced by the arguments  for
>    using RTP: there are definitely some requirements for real-time  data,
>    of operating system have become more similar over time, so the  problem
>    might be easier than when previous solutions were last developed,  and

App-sharing can make sense to setup with SIP.  The data streams are or can
be a mixed bag - the primary ones are not or should not have much in common
with what we do in AVT (i.e. RTP and the like).  Some of the data streams
do map over (video and audio); hopefully those can make use of existing
payload formats.  Some might map over in that there's timing info, though
other transport protocols probably make more sense (mouse, keyboard, etc).

So my question for this is what form would the audio and video channels
take (when active).  For the main UI imagery, I would use some other
transport than RTP/AVP in the SDP media lines.  It doesn't *have* to be
TCP-based - UDP has some advantages in high loss/long-delay/etc situations,
so long as each packet (or at least small group of packets) are
self-contained *and* you have a robust recovery mechanism.  But normally
TCP would be prefered, and is certainly a lot simpler to implement.  The
protocol should include NTP timestamping so the UI (and input streams like
mouse and keyboard) can be synchronized with each other and audio/video
streams.

>    problem to solve, but he is not sure this is the right  architecture and
>    AVT may not be the right home. Perhaps we need to better  understand the
>    Some interest was also expressed in scaling one small screen  devices,

That seems to be a application/client UI issue.

>That previous work wasn't adopted in AVT, due to those concerns, and  I
>don't see anything in the current draft that addresses them either.

Pretty much I agree.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)