Subject: Re: [Widex] Working group last call (WGLC) for Requirements: draft-ietf-widex-requirements-01.txt

From: "Jin Yu" <[email-masked]>
Date: Wed, 10 May 2006 17:04:00 -0700

Hi all,

I have a few comments about the requirements draft, mostly based on my
experience in working on XUP and X11.

* Event Selection

In a networked UI environment, efficient event delivery is as important as
updating the UI itself. The requirements document already discussed that
certain events can be handled locally at the renderer (e.g. a mouse click on
a menu bar will display a submenu) and therefore there is no need to send
those events to the server. However, in addition to that, it is very
important that the renderer only sends events that are *interested* by the
application logic on the server side. For example, typing in a text field
generates a key press event for each key stroke. But the application may not
care about each key stoke; it only wants to receive the entire text when the
user clicks on a button labeled "submit". In this case, the application is
interested in the mouse click event on the "submit" button, but not
interested in the key press events in the text field.

So what's needed is an event selection mechanism, and I believe this is
needed at the protocol level, not at the XML UI language level. That is,
this event selection mechanism should be independent of the actual UI
language in use.

I think we could define a new type of message - widex event selection
object. This is a one directional message that goes from the server to the
renderer, allowing the application to instruct the renderer to send only
selected events. The widex event selection object would specify the type of
event as well as the UI component that fired the event. And the event type
would be a namespace-qualified event from any concrete XML UI languages (not
limited by DOM events; it could come from XUL, XAML, etc.)

Note that the new event selection object would be very different from
WO.Event. The event selection object specifies the event delivery criteria;
whereas WO.Event specifies the actual event fired, with timestamp, etc.

Similar design can be found in XUP, where it has the notion of "event
selectors" (<http://www.openxup.org/TR/xup/#elem_selector>).

In addition, X11 also allows the selection of events, via the library
function XSelectInput().

* UI State Synchronization

The diagram at the beginning of section 2 shows the UI update interface as
one way messages from the server to the renderer. However, for the UI state
to be synchronized, the renderer should also send UI update messages to the
server.

The server-side UI state is manipulated by application code (e.g. create a
new window, change background color of a label, etc.); this results server
to renderer UI updates messages.

On the other hand, the renderer's UI state is manipulated by the end user.
For example, when the user is typing text in a text field, the UI state of
that field is getting changed. This change needs to be synchronized to the
server. Other examples include resizing a window, checking a check box, etc.
Note that the synchronization needs not to be immediate, as the application
may just care about the entire text input, not each key stoke. But how and
when to synchronize should be left to the application to decide. Again, this
could be done by using the widex event selection object proposed above.

Regards,

Jin
________________________________________
Jin Yu
OpenXUP.org

_______________________________________________
Widex mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/widex

---

From: Dave Raggett <[email-masked]>
Date: Fri, 12 May 2006 14:52:41 +0100 (BST)

Hi Jin,

Some thoughts triggered by your email:

Event filtering could be:

   a) by prior arrangement
   b) negotiated at the start of the session
   c) dynamically updated during the session

Session initiation is out of scope for REX, but we could support
the means to dynamically add or remove event listeners during
the session via introducing a pair of new events for that purpose
as you suggest. I could draft something on this for the REX mailing 
list.

For Widex, we could explore the idea of negotiating event selection
at session initiation.

   a) a list of explicit event names
   b) a list of event catgories
   c) a reference to a list of event names/event categories

Event categories are something that was introduced by the W3C Voice 
Browser working group for VoiceXML and CCXML. They defined event 
names hierarchically and could name categories of events by 
reference to this hierarchy, e.g. there are a set of events

      error.connection.noauthorization
      error.connection.baddestination
      error.connection.noroute
      error.connection.noresource
      ...

you can specify behavior for a specific event, for all 
error.connection events, or more generally for all error events.

Another idea is to be able to indicate which events the server is 
interested in by reference to a URI for a list of events or event 
categories. The usual caching rules would apply to such lists. For 
devices that aren't connected to the Internet, such lists would have 
to be available on the local network, e.g. on the widex server. An 
advantage of identifying events by reference is that it can save 
time if the list is known to be always the same, since then you 
don't need to download it for each session.

One issue is where the event listener is attached to the DOM tree 
and whether it is invoked in the bubble or capture phase. This only 
matters when there is another event listener that is invoked prior 
to the server's event listener, as then there may the possibility of 
the event being cancelled by the earlier listener.

I don't believe it makes sense to allow the server's event listener 
to be able to cancel events, as this would introduce a significant 
latency into event handling in the client. For this purpose, the 
event object, that is passed to the server by the widex 
implementation, would have the DOM attributes bubbles and cancelable 
set to false.

In principle, we might also need to say something about which events 
the server can send to the widex renderer. The same negotiation 
mechanisms could be used as for events the server gets from the 
client.

You also talk about UI initialization. I think this can be handled 
as part of the UI markup language, and also by additional events 
sent from the server to the widex renderer. These events are outside 
the scope of Widex, and would need to have their semantics and XML 
serialization defined in separate specifications.

For REX, it will be easier if we stick to the existing DOM events
so that we can limit the scope of what needs to be specified for
the processing and serialization of these events. I think we will
be able to provide an algorithm for describing the serialization
of these events from their IDL definitions.

  Dave Raggett <[email-masked]>  W3C lead for multimodal interaction
  http://www.w3.org/People/Raggett +44 1225 866240 (or 867351)

On Wed, 10 May 2006, Jin Yu wrote:

> Hi all,
>
> I have a few comments about the requirements draft, mostly based 
> on my experience in working on XUP and X11.
>
> * Event Selection
>
> In a networked UI environment, efficient event delivery is as 
> important as updating the UI itself. The requirements document 
> already discussed that certain events can be handled locally at 
> the renderer (e.g. a mouse click on a menu bar will display a 
> submenu) and therefore there is no need to send those events to 
> the server. However, in addition to that, it is very important 
> that the renderer only sends events that are *interested* by the 
> application logic on the server side. For example, typing in a 
> text field generates a key press event for each key stroke. But 
> the application may not care about each key stoke; it only wants 
> to receive the entire text when the user clicks on a button 
> labeled "submit". In this case, the application is interested in 
> the mouse click event on the "submit" button, but not interested 
> in the key press events in the text field.
>
> So what's needed is an event selection mechanism, and I believe 
> this is needed at the protocol level, not at the XML UI language 
> level. That is, this event selection mechanism should be 
> independent of the actual UI language in use.
>
> I think we could define a new type of message - widex event 
> selection object. This is a one directional message that goes from 
> the server to the renderer, allowing the application to instruct 
> the renderer to send only selected events. The widex event 
> selection object would specify the type of event as well as the UI 
> component that fired the event. And the event type would be a 
> namespace-qualified event from any concrete XML UI languages (not 
> Note that the new event selection object would be very different 
> from WO.Event. The event selection object specifies the event 
> delivery criteria; whereas WO.Event specifies the actual event 
> fired, with timestamp, etc.
>
> Similar design can be found in XUP, where it has the notion of 
> "event selectors" 
> (<http://www.openxup.org/TR/xup/#elem_selector>).
>
> In addition, X11 also allows the selection of events, via the 
> library function XSelectInput().
>
> The diagram at the beginning of section 2 shows the UI update 
> interface as one way messages from the server to the renderer. 
> However, for the UI state to be synchronized, the renderer should 
> also send UI update messages to the server.
>
> The server-side UI state is manipulated by application code (e.g. 
> create a new window, change background color of a label, etc.); 
> this results server to renderer UI updates messages.
>
> On the other hand, the renderer's UI state is manipulated by the 
> end user. For example, when the user is typing text in a text 
> field, the UI state of that field is getting changed. This change 
> needs to be synchronized to the server. Other examples include 
> resizing a window, checking a check box, etc. Note that the 
> synchronization needs not to be immediate, as the application may 
> just care about the entire text input, not each key stoke. But how 
> and when to synchronize should be left to the application to 
> decide. Again, this could be done by using the widex event 
> selection object proposed above.
>
>
> Regards,
>
> Jin

---

From: "Jin Yu" <[email-masked]>
Date: Fri, 12 May 2006 11:58:21 -0700

>    c) dynamically updated during the session

Yes, and I think the dynamic part is essential, since most applications will
very likely change their event handling behavior at runtime. That is why I
see the need of a WO for event selection.

> error.connection events, or more generally for all error events.

Event categorization is a great concept. But that could be left to the
specific UI languages. The protocol should be just concerned with some XML
namespace qualified event name; whether that name is a concrete event or an
event category could be interpreted by the UI language in use.

> implementation, would have the DOM attributes bubbles and cancelable

I agree. Actually I believe the DOM event capturing / bubbling model may not
fit well with a network UI environment. I think we should not restrict
ourselves to DOM style events. What's more appropriate is a *delegation"
based event model, which is used by popular desktop GUI toolkits such as
.NET WinForms and Java Swing. In this model, one would attach event
listeners (or event handlers in .NET WinForms) to a specific UI component
for a specific event type. This allows for a loose coupling of UI events and
application logic, and it also minimizes event traffic.

> serialization defined in separate specifications.

What do you mean by "UI initialization"? I think I was talking about UI
state synchronization in my message. Are you referring to something in the
XUP spec?

Regards,

Jin

---

From: <[email-masked]>
Date: Sun, 14 May 2006 20:45:44 +0300

Hi Jin,

We started the requirements draft with the goal of having a minimal set
of requirements that a system exporting the UI to another system must
fulfill but stop short of going deeper in describing the system. Having
these goals in mind we described the high level interactions between the
Widex Server and Widex Renderer as having just two basic interfaces: 

a) update interface, used to send information from the server to the
renderer that is basically generated by the application logic;
b) event interface, used to send information from the renderer to the
server that is basically generated by the user interaction with the
rendered UI.

It is quite likely that the terms used so far in the requirements draft
will not map directly to the specification, as is the case with the
proposed Widex framework
(http://www.ietf.org/internet-drafts/draft-stirbu-widex-framework-00.txt
) where practically all messages exchanged between the Widex Server and
the Widex Renderer are events described using REX.

I think that being able to dynamically adjust which events server is
interested in is a good feature but that is a design issue and Dave
already mentioned several alternatives. We'll have to asses these
alternatives and select the optimal one for our environment.

As a bottom line, I think that your comments are very useful but I think
that they should be considered for the Widex Framework draft rather than
for the Requirements.

Regards,
Vlad

---

From: "Jin Yu" <[email-masked]>
Date: Mon, 15 May 2006 16:17:18 -0700

Hi Vlad,

My comments are based on reading both the January framework draft and the
April requirement draft. So I agree with you that the comments might be a
bit too detailed for the requirements draft, and indeed they may be more
appropriate for the framework draft.

Regards,

Jin

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Sunday, May 14, 2006 10:46 AM
> To: [email-masked]; [email-masked]
> Subject: RE: [Widex] Working group last call (WGLC) for
> Requirements:draft-ietf-widex-requirements-01.txt
> 
> Hi Jin,
>