Subject: RE: Remote UI BoF at IETF63

From: [email-masked]
Date: Fri, 1 Jul 2005 12:24:35 +0300

Hi,

Please find attached a Remote UI BoF FAQ which explains some of the problems with existing protocols and clarifies the scope of proposed work. To discuss this in more detail, a mailing list is available at [email-masked] and you can subscribe here https://www1.ietf.org/mailman/listinfo/remoteui.

Also, a starting point for this work can be found here 
http://www.ietf.org/internet-drafts/draft-stirbu-lrdp-00.txt.

Regards,
Vlad

Frequently Asked Questions - Remote UI BoF
------------------------------------------

1. What problem is the Remote UI protocol trying to solve?

The Remote UI is a mechanism that enables user interfaces to be rendered on other devices than those that have the application logic. The Remote UI protocol provides the mechanism which enable a client device the generate a UI, received from a server device, using the client device's native UI capabilities, and keeping the UI synchronised with the application logic.

2. Who needs this solution?

Device manufacturers invest a lot of energy in trying to optimise their devices for certain environments. Because the devices are intended for a diverse range of purposes, their UI capabilities can vary considerably; screen size and ratio, color depth, windowing system with various widget sets, input methods are making the environment highly heterogeneous. At the same time, application developers and UI designers are trying to create user interfaces that are high optimised for the rendering platform, so that the user experience is improved by  having the respective application easy to learn and use.

Therefore, when an UI is rendered on another device than the one that is running the application logic provisions need to be made, so that the user can perceive the UI as a local application making it intuitively usable. For 
example an application UI may appear differently when remoted on an Windows XP desktop vs. a mobile phone.

3. Why this cannot be done with existing protocol?

Existing protocols can be split in two categories: framebuffer-level and graphics-level protocols.

In the framebuffer-level protocol, the contents of the framebuffer (i.e. the individual pixels) are copied across the network to a framebuffer on the client. In order to avoid sending the full screen every time something changed on the screen, these protocols typically send only the pixels that are changed inside the clipping regions to the client. Examples of such protocols are VNC and protocols based on T.120, like Microsoft's RDP.

In the graphics-level protocol, the drawing request to the graphical device interface (GDI), such as DrawLine(), DrawString(), etc. are copied across the network. The client is responsible for interpreting these commands and rendering the lines, rectangles, strings, etc. in its framebuffer. Example of such protocol is X Windows.

The problem with these approaches is that, in order to render the UI, the clients are following blindly the instructions received from the server; they don't have means to influence the appearance of the UI, they just render the UI using the graphical elements/instructions that are provided by the server and are specific to the server platform.

4. What are the pieces, building blocks that are needed to build this solution?

In order to correct the problems pointed out with the existing protocols, a mechanism is needed so that information about the widgets or widget trees is sent to the client, so that it can generate the UI using native client's widgets. This mechanism has usually three components: UI description language, UI remoting protocol and session setup.

UI description language contains the descriptions of the widgets, their properties and relationships between the widgets. Typically the widget-level UI descriptions are augmented with stylesheets containing hints such as preferred colours to use, which layout to use, which background picture, etc. Examples of such UI 
descriptions languages are Mozillla's XUL and OASIS' UIML.

The UI remoting protocol is the transport protocol that is responsible with communicating (partial) UI updates from the server to the client and UI events triggered through changes in widget states made by the user from the client to the server. Usually an UI update can have three meanings:
	
	* UI Initialisation - an application can send the full description of the UI at session setup.
	
	* UI Fragment - an application is sending partial descriptions of the UI when the application is becoming aware of which parts of the UI will be used depending on the runtime status.
	
	* UI Update - an application is sending notifications when widget properties are changing due to changes in application state.

The actual structures that are exchanged with UI updates and UI events are described using the UI description language.

Session setup is responsible with identifying compatible servers and clients and initiating the UI remoting session between them. A client is compatible with a server when they support the same UI remoting protocol and the same UI description language. Examples of session setup protocols are SIP/SDP and UPnP Remote UI.

5. What is the goal of the Remote UI effort in IETF?

Create a UI remoting protocol for applications that are using UI Description Languages.

> -----Original Message-----
> From: ext Pekka Savola [mailto:[email-masked]]
> Sent: 30 June, 2005 20:42
> To: Dave Crocker
> Cc: Stirbu Vlad (Nokia-TP-MSW/Tampere); [email-masked]
> Subject: Re: Remote UI BoF at IETF63
> 
> 
> On Thu, 30 Jun 2005, Dave Crocker wrote:
> [...]
> > Are there any materials to use as input, either as 
> exemplars or possibly even 
> > as a beginning specification?  This latter would be 
> particularly helpful for 
> > the success of the effort?
> 
> FWIW, when I read the BoF description, I started wondering about the 
> same things.  Specifically I started thinking about the half a dozen 
> protocols that already exist in this space (vnc, rdp, ...).  It'd be 
> very good to understand the scope of the work, and how the existing 
> protocols fit (or don't fit) the bill.
> 
> -- 
> Pekka Savola                 "You each name yourselves king, yet the
> Netcore Oy                    kingdom bleeds."
> Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
> 

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: David Hopwood <[email-masked]>
Date: Fri, 01 Jul 2005 12:34:49 +0100

[email-masked] wrote:
> In the framebuffer-level protocol, the contents of the framebuffer (i.e. the
> individual pixels) are copied across the network to a framebuffer on the client.
> In order to avoid sending the full screen every time something changed on the
> screen, these protocols typically send only the pixels that are changed inside
> the clipping regions to the client. Examples of such protocols are VNC and
> protocols based on T.120, like Microsoft's RDP.
> 
> In the graphics-level protocol, the drawing request to the graphical device
> interface (GDI), such as DrawLine(), DrawString(), etc. are copied across the
> network. The client is responsible for interpreting these commands and
> rendering the lines, rectangles, strings, etc. in its framebuffer. Example of
> such protocol is X Windows.

Framebuffer-level protocols can be viewed as a special case of graphics-level
protocols where the drawing commands are restricted to bitblt-like commands.

> The problem with these approaches is that, in order to render the UI, the
> clients are following blindly the instructions received from the server;
> they don't have means to influence the appearance of the UI, they just
> render the UI using the graphical elements/instructions that are provided
> by the server and are specific to the server platform.

Having the UI adapt to a look-and-feel appropriate to the client device
(and user's preferences) doesn't automatically imply that it has to be
the client that does this adaptation. The client could send the server a
description of the preferred L&F. The advantage of this is that it allows
clients to be much simpler, putting the complexity on the server which is
likely to have more memory, processing power, etc.

-- 
David Hopwood <[email-masked]>

---

From: saravanan t s <[email-masked]>
Date: Mon, 04 Jul 2005 20:35:58 +0530

Hi Vlad & others

Just adding some opinions on this from my side, since I found this idea
interesting:

1. Even though today's clients are powerful enough to handle the widgets and
adaptation, maybe on the server side, certain computations and algos could
be clubbed to reduce the load on the pda's and other devices. The extra
resources (maybe read "power") on the pda's and other devices could be used
for adding additional features (or maybe extending battery life?)

2. How many look and feel "standards" or maybe "UI Languages" the server can
support? Does this imply that there will be only a limited set of L&F that
can be supported by the server and rendered on the client? Will this be a
limitation for different devices (talking of embedded) having different
levels of needs on widgets (qualitatively: very simple to quite complex)?

3. On the other hand, there may be one more advantage to have clients send
the description of the L&F sent to the server & server managing the client
based on its description - is it possible that the bandwidth will be used
more efficiently than the existing protocols that seem to achieve similar
purposes for the end user?

Regards,
Saravanan T S

>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>David Hopwood
>Sent: Friday, July 01, 2005 5:05 PM
>To: [email-masked]; [email-masked]
>> Existing protocols can be split in two categories:
>framebuffer-level and
>> graphics-level protocols.
>>