Subject: Re: [ogpx] Synchronization of gestures (was: Limits to interoperability of scripted objects)

From: Han Sontse <[email-masked]>
Date: Fri, 4 Dec 2009 18:11:10 -0800

My apologies to all for this screwed up thread.   I didn't include the  
mail-host in the thread for the first part of this dialog.

On Dec 4, 2009, at 12:42 PM, Carlo Wood wrote:

> I know I'm not diplomatic, but you'll have to admit
> that in some cases and/or for some people there IS a
> need to wait till the slower ones have the data too.

With all due respect,  Carlo, I am finding that I disagree with your  
analysis.

I have a lot of practical experience in SL getting animations
as well as sound and textures to synchronize under a lot
of different conditions by taking into account the client's existing
cacheing behavior.   I'm drawing on that experience for my assertions.

I can't think of a single reasonable use case that would *require*  
clients
to hold for the slowest link.  The examples you have suggested
don't require it to achieve the outcome you desire.
The only reason that synchronization is a problem today in SL is because
there is no provision for properly notifying the client that two or more
isochronous assets are to be played as part of a sequence.

Signaling the client that some arbitrary bundle of resources are
required to be cached as a suite will handle a vast majority of
the synchronization issues, and a MIDI-like sequence handler can manage
the remainder of specialized cases where complex/strict client side  
isochronous
behavior is desirable.

The outlying cases (the unreasonable ones) are those that really are  
beyond the scope of
what is possible because communication over any appreciable distances  
(> 100km ) is unlikely to
ever be instantaneous in our life times.   While wave fronts in  
electric media do travel
near the speed of light, the switching fabric relaying and repeating  
those signals is
several orders of magnitude slower even under the ideal cases.   Inter- 
client isochronicity to the level
a performing musician might desire is simply out of reach, since the  
ear-brain interface
is sensitive to latencies as small as 2 to 5 ms, and acceptable  
performance almost completely
breaks down at about >50 ms.

So here I'll walk through an example... the hug:

To synchronize a hug all that is needed is for the
hug object to report that it contains bundled isochronous
assets during the normal process of the client caching the
surrounding area.  This bundled sequence might include animation for
the hugger and the huggee, and maybe a sound or two.  It include
possibly a client side script to move the target agents into the
correct relative locations to participate in the hug.

a)	In the simple case the caching of bundled resources has
	occurred.

b)	A permissions request would be sent to the huggee.

c)	The permissions are granted.  (were permission not granted
	a subsequent trigger event would be rejected by the SIM)

d)	The hugger's hug object issues a trigger event which is relayed  
through
	the Sim to all clients within audio/visual range.

e)	Each client on receiving the event applies the event to the indicated
	agents using the sequence bundle cached locally

f)	each client views the sequence isochronously as directed by the
	sequence bundle.   Timing for the sequence is local to each client.
	Each client views the hug as a seamless sequence.  Though the exact
	starting time of the event is different for each client.   In all but  
the most
	extreme cases of lag, a user viewing the rendered event will have the  
perception that
	the event occurred simultaneously for all witnessing/participating  
agents.

Ok now for a more complex case where one or more clients have not  
cached the
bundle before the event is triggered:

a) 	permission request is granted.

b)    The hugger's hug object issues a trigger event which is relayed  
through
	the Sim to all clients within audio/visual range.

c)   	As each client receives the trigger event it may find that it must
	request the sequence bundle indicated in the event.  The appropriate
	assets are requested.

d)	The sequence is played as soon as all the assets in the bundle are  
queued.

e)	Each client views the event as a seamless sequence.   In this case  
the delay is likely to
	be longer for some clients than others.  The exact wall clock delay  
does not matter since
	the end user has no idea when the event was actually sent.   Users on  
extremely slow links
	may wish to have their client reject events that have become too  
stale,  but this loss for one client has no
	impact on any other client witnessing the event.

To avoid e) in the second example it might be desirable for a client  
to request sequence bundles
right after caching agent appearances, to avoid experiencing  
excessively stale sequence triggers.

~HS~
---------------
<previous email in thread>
To clarify my position:

Key difference is that there really is no good reason for a client on  
a fast link to wait
for the client on the slowest link before playing isochronous content  
locally.  Ever!

In 'live' performance situations there might be a need to
send trigger events for sequences.  (Trigger_sequence<UUID>) However,  
there is no
really good reason I can see to hold up the show because
some clients are on slow links.

Having clients signal readiness might be a useful *advisory*,
to indicate to a program director that enough clients are fully cached  
to start a performance.
This could be a simple broadcast to all clients requesting caching  
status
on a *particular* bundle of pre-cacheable content.   No automatic  
behavior, optional or
otherwise, is needed for feedback.  A scripted object might make such  
a broadcast and
simply report each responder's findings to the program director.

No client to client synchronization is needed
for isochronous content.   This is the key point I am making.
There's no reason for the complexity you described:
>>> * As soon as all viewers are ready (optionally, a parameter
>>> of the original request includes a timeout after which
>>> the delayed avatar is considered to have said "no"),
>>> the server informs all viewers to start playing the
>>> multi-gesture.

~HS~

On Dec 4, 2009, at 9:42 AM, Carlo Wood wrote:

> Bottomline is, there should be an option (preferably the default
> in the future) to wait a while until all needed assets are
> downloaded, and then there needds to be a way to tell the other
> clients (through the sim) that you are ready.
>
> I don't seem much difference there with your ideas :)
>
> Once all assets are ready (locally cached) then just playing
> them in order as described by the gesture info will work
> good enough: there is only one synchronization point needed:
> when to START.
>
> On Fri, Dec 04, 2009 at 09:01:51AM -0800, Han Sontse wrote:
>> I've been giving this a lot of thought recently.  It seems to me
>> that strict isochronous behavior between clients is not really
>> needed for gestures and synchronous animations (dances and hugs).
>>
>> It doesn't really matter how long it takes a client to acquire the
>> assets needed for a given interaction.  Each client is showing a
>> arbitrarily delayed rendition of the sim state anyway.
>>
>> It seems more useful to me to have a mechanism that describes a
>> isochronous sequence using arbitrary collection of animated content;
>> Audio, MIDI, animation, etc.  A sequence time-line, if you will.
>> The sequence lists the assets that will be played in the time line.
>> These are then requested/located as needed by the client.   Once all
>> are queued, the sequence is played.   From each client's perspective
>> the script plays in local real time, but without any synchronization
>> between clients.    Tighter synchronization between clients could be
>> achieved  simply by having a mechanism for clients to be notified
>> that sequenced objects are present in the local environment.  The
>> client can then choose to pre-cache these sequences subject to user
>> preference.
>>
>> As the gesture mechanism operates today in SL, some content will
>> begin playing as expected, while other content declared in the
>> sequence are still loading.  I do not know if this is as designed,
>> or it reflects some collection of bugs in the gesture handling
>> process.
>>
>> What is missing here is a robust method for notifying clients that
>> isochronous groupings of content are present in the local
>> environment.
>>
>> A more complex issue exists for situations where multiple clients
>> seek to create a live interactive performance.  The current state of
>> the internet as a whole simply cannot support hard isochronous
>> behavior.    However a performing group at a physical location might
>> cause a library of sequences to be cached in the clients, which can
>> then be triggered by a tokenized sequence embedded in a QuickTime
>> Stream.  This choreography would then appear isochronous at each
>> client individually.
>>
>> For an example of how difficult this problem is, try to watch a
>> streamed TV show with someone who is 3K miles away, while you have a
>> voice channel open so that you can comment on the show to each other
>> in real time.  SL or not this takes some insight and discipline to
>> achieve a "just like being there" experience for even for a party of
>> two!   This is even true if the content being viewed is already
>> local to both participants!   For a more typical example: try
>> listening to the same radio stream with a friend and try singing
>> along to it!
>>
>> ~HS~
>>
>>
>>
>> On Nov 10, 2009, at 12:41 PM, Carlo Wood wrote:
>>
>>> On Tue, Nov 10, 2009 at 09:11:57PM +0100, Carlo Wood wrote:
>>>> make animated hugs (or any couple animations) really work (with
>>>> effort).
>>>
>>> without*
>>>
>>>
>>> Here is an idea of how it could work:
>>>
>>> * Viewer A (or object) initiates Multi-gesture Request
>>> to N (other) viewers, this involves a list of the
>>> animations (UUID) to play for each of the avatars
>>> involved, and sounds to play etc, and the starting
>>> position in the world of each avatar.
>>> * Each viewer reacts with: Ok, will try; or Nope, I won't.
>>> If any viewer denies the request (or times out?)
>>> then the server informs all viewers that the multi-
>>> animation was aborted (and why). [ An optional flag
>>> could be to continue anyway, but without the avatar(s)
>>> that don't want to. ]
>>> * Once all avatars that want to play the multi-animation
>>> are known, the server sends them a message: "We have a go!"
>>> * All viewers that have all the needed information to
>>> continue (downloaded and cached the animations, sounds etc)
>>> send a message to the server: "ready!"
>>> * As soon as all viewers are ready (optionally, a parameter
>>> of the original request includes a timeout after which
>>> the delayed avatar is considered to have said "no"),
>>> the server informs all viewers to start playing the
>>> multi-gesture.
>>>
>>> And YAY! finally a completely synchronized couple (and
>>> multi) animation, even synced with sound!
>>>
>>> Options in case of delays (timeouts) should be:
>>> - Play as soon data available locally,
>>> - Wait till at least N, or at least those, avatars
>>> are ready.
>>> - Abort completely if any can't play in time.
>>> - If someone is delayed and the rest already plays
>>> the animation, then the viewer can decide to abort
>>> or play the animations locally anyway, as soon as
>>> they become avaiable, etc etc... All those possibilities
>>> are mere bits in already existing messages, so why not.
>>>
>>> As a side effect of this it should be possible to
>>> play a gesture involving animation and sound by ONE
>>> avatar (as are the gestures in SL now) but *synchronized*:
>>> everyone sees it at the same time, and the animations ARE
>>> synced with the sound once the start to play (unlike now,
>>> where the animation is played and 30 seconds later you
>>> hear the sound).
>>>
>>> -- 
>>> Carlo Wood <[email-masked]>
>>> _______________________________________________
>>> ogpx mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/ogpx
>>
>
> -- 
> Carlo Wood <[email-masked]>

On Fri, Dec 04, 2009 at 10:19:09AM -0800, Han Sontse wrote:
> To clarify my position:
>
> Key difference is that there really is no good reason for a client
> on a fast link to wait
> for the client on the slowest link before playing isochronous
> content locally.  Ever!

Um, not to you - but there is for me, and probably others.
No reason to dictate what is possible and what is not possible.

In my proposal it is possible to NOT wait and just play the
animation whenever you have the needed data AND it is possible
to wait until all or some clients have it too. Not all, not
forever, but configurable. Configurable is always better.

For example, if I'm dancing some romantic slow dance
then I DO want to be full synchronized with my partner,
so that right at the moment I lift him/her up I can type
something that refer to lifting him/her up.

I'd care less about some noob with a slow link at the
other edge of the dance floor - so I'd configure my
client to wait for my partner, and wait at most 4 seconds
for everyone else, and then start it.

> In 'live' performance situations there might be a need to
> send trigger events for sequences.  (Trigger_sequence<UUID>)
> However, there is no
> really good reason I can see to hold up the show because
> some clients are on slow links.
>
> Having clients signal readiness might be a useful *advisory*,
> to indicate to a program director that enough clients are fully
> cached to start a performance.

Well, clearly every viewer by itself is free to do what
they want - so it's automatically advisory in nature.

> This could be a simple broadcast to all clients requesting caching
> status
> on a *particular* bundle of pre-cacheable content.   No automatic
> behavior, optional or
> otherwise, is needed for feedback.  A scripted object might make
> such a broadcast and
> simply report each responder's findings to the program director.
>
> No client to client synchronization is needed
> for isochronous content.   This is the key point I am making.
> There's no reason for the complexity you described:
>>>> * As soon as all viewers are ready (optionally, a parameter
>>>> of the original request includes a timeout after which
>>>> the delayed avatar is considered to have said "no"),
>>>> the server informs all viewers to start playing the
>>>> multi-gesture.

Yes there is (although it's not complex imho).
The argument "we can do it simpler, there is no need for
this complexity" is exactly the reason that currently
all couple animations and all gestures with sound totally
suck.

If you "simplify" the proposed protocol you take away
possibilities, things will not be possible anymore,
things that some people might want or need.

For example, someone wants to jump in the air
shouting "Whoooohoooo!" and right after that run
for the door and bolt out.

Then that person NEEDS to know when everyone is ready,
so that they ALL see the jump and whoohoo at the same
time, otherwise some would see him run to the door
and jump when already being outside (or worse, zap
back inside, jump and then zap outside again).

I know I'm not diplomatic, but you'll have to admit
that in some cases and/or for some people there IS a
need to wait till the slower ones have the data too.

-- 
Carlo Wood <[email-masked]>

---

From: Carlo Wood <[email-masked]>
Date: Sat, 5 Dec 2009 13:40:52 +0100

On Fri, Dec 04, 2009 at 06:11:10PM -0800, Han Sontse wrote:
> With all due respect,  Carlo, I am finding that I disagree with your analysis.

There is no analysis.. it is something has been said before
on the list: if YOU don't want something, then that doesn't
mean there will be no need for it in the future by others.

You shouldn't take YOUR experience and desires and then say:
"we will never need this, NOBODY will EVER need this",
while it's relatively easy and simple to add the *support*
for it to the protocol!

By adding the *possibility* for viewers to wait for others,
you do not force anyone to do it different from what you
describe. But you also don't take away the possibility to
do it how other people WANT it (like me, I personally think
this is important - so thus far we have a 50% - 50% vote for
the NEED of this wait feature).

So, to be really undiplomatic blunt: who are you to deny
me this experience? While adding just a little to the protocol
we BOTH can be happy? You can do it your way, and I can do it
my way? But you say: "You don't need it" and want to remove
the possibility from the protocol so that you can do it your
way and I CANNOT do it the way I want ?!

THAT is the reasoning behind NOT leaving out rather trivial
options to the protocol. And really, this has been said on
the list before a few times imho (and no, I'm not going to
find that back).

> cacheing behavior.   I'm drawing on that experience for my assertions.

I am too, and I say I need this.

> I can't think of a single reasonable use case that would *require* clients

I can.

> don't require it to achieve the outcome you desire. 

??? Then you didn't read very well what I said.
If you want to do what I said then you *NEED* the possibility to wait
for other (some) clients (some limited time), and why not, have the
possibility to abort, or continue anyway (as you want).

Having those options you can synchronize the experience well within
lag-time, well within typing-delay time (ie, under 1 second). If you
don't wait, the playing time could run up till 30 seconds or more!

Having those options you can synchronize the experience well within
1 second AND you can do what you want: just state don't wait for anyone
and play stuff the second you get it.

[...]
> To synchronize a hug all that is needed is for the 

"all that is needed" to accomplish YOUR needs and views.
For me, I realllllly need to know when other clients are
ready too.

[...snip...]
> be longer for some clients than others.  The exact wall clock delay does not
> matter since the end user has no idea when the event was actually sent.

Yes we totally disagree here, and you are wrong: there are many other
shared experiences (typed text, and remaining movement of the avatar
before and after; not to mention voice chat) that need to be more or
less in sync with gestures; the user WILL have an idea when the event
was sent.

The wall clock DOES matter - certainly when the delay can run
up to a whole minute.

You are "right" for delays less than a second, and I argue
that we should try hard to get the shared experience synchronized
within one second IF THE USER(S) SO DESIRE.

I already gave you examples where it is notable when the delay
is larger, but I guess every example is personal. If YOU don't
care that your friends see you bolt for the door and once outside
you jump and shout "Whooohooo!", while you really intended to
first jump and shout and THEN run for the door, then well--
that is entirely your choice. Now let me, and others, have mine.

Another example, your hug: You initiate a hug request (your client
already has the needed assets or starts to download them immediately).
By the time that your partner clicks 'Ok', your viewer is ready
and immediately starts to play the hug. While hugging you say:
"Mmmmmmmmmmm, so nice". And YOU don't care if your partner sees
that text while they don't see you hugging yet; others do however.
Because others DO, we NEED support in the protocol to know when
other clients are ready (at least, have all the needed data
cached; any other delay is too short to be bothered with-- I'm
with you on that then).

---

From: Carlo Wood <[email-masked]>
Date: Sat, 5 Dec 2009 13:48:42 +0100

Blah!

Before, this was a private conversation...

I didn't notice that Han suddenly CC-ed this list (quoting
my private mail in the process, thank you very much), so
this post is really private.

Had I known it was now suddenly public, I would never have
"attacked" Han in an almost personal way, of course. I would
appologise but well, I'm not the one that suddenly started to
CC the ogpx list :/

Anyway, I guess I'll check the CC line better next time
I react to private mail, I guess.

Carlo

---

From: Lawson English <[email-masked]>
Date: Sat, 05 Dec 2009 07:00:27 -0700

All this goes back to what I just posted on the SLDEV list about taking 
initiative and coding something up. Having the ability for clients to 
talk to each other rather than some 3rd party service might blur the 
line concerning what VWRAP is supposed to be talking about, but, IMHO, 
the line is getting blurred anyway.

What's the difference between a plugin sitting on someone's desktop in 
order to augment the client in some way, and a plugin that actually 
allows clients to talk to each other directly in order to collectively 
augment them in some way? As far as I'm concerned, these discussions 
about clients talking to each other are just extensions totalks about 
what responsibilities client plugins might have.

My current project is based on this idea:

 http://wiki.secondlife.com/wiki/User:Saijanai_Kuhn/Plugins_discussion#Proposed_Extension_to_Media_Plugin

I'm writing a SL client- squeak bridge (starting off easy with a port of 
my  do-nothing Python client for practice) that will allow me to test 
new client plugin ideas. The Holy Grail for the project will be to 
integrate Cobalt/Croquet with SL in some way(s) and that is just the tip 
of an iceberg that is being held under water by a large outcropping of 
land. There's more potential in P2P-VWRAP hybrids than there is in  P2P 
(Croquet/Cobalt) OR VWRAP (clear-cut client-server worlds) taken 
separately. While VWRAP/OGP/whatever's charter may not consider the P2P 
approach as relevant, the fact is, P2P is potentially in ANY extension 
of services involving clients which aren't talking exclusively to an old 
school SL simulator.

Lawson

---

From: Carlo Wood <[email-masked]>
Date: Mon, 7 Dec 2009 01:37:15 +0100

On Sat, Dec 05, 2009 at 07:00:27AM -0700, Lawson English wrote:
> All this goes back to what I just posted on the SLDEV list about
> taking initiative and coding something up. Having the ability for
> clients to talk to each other rather than some 3rd party service
> might blur the line concerning what VWRAP is supposed to be talking
> about, but, IMHO, the line is getting blurred anyway.
> 
> What's the difference between a plugin sitting on someone's desktop
> in order to augment the client in some way, and a plugin that
> actually allows clients to talk to each other directly in order to
> collectively augment them in some way? As far as I'm concerned,
> these discussions about clients talking to each other are just
> extensions totalks about what responsibilities client plugins might
> have.

For the record, nowhere I proposed to let clients talk to eachother.

It was my opinion that it is better to use the sim as trusted
middle man for all of this.

Someone sends a message to the sim, the sim sends a message to
all clients (within relevant range), and then clients reacts
back to the sim, who controls the decision making (based on
the parameters of the initial message). Once the state becomes
as requested, the sim sends again a message to all clients
to inform that that the requested state has been reached (or
one of the requested timeouts has been reached).

This means, in the case of synchronized gestures, that the
VWRAP protocol (client-server is also VWRAP no?) needs to
support some parameters for this first message that allows
a client to request the wanted strategy: a timeout per
client at least.

I think it will be sufficient to make a difference between
the following groups of clients:

1) Those participating in the multi-animation
2) Those within X meters.
3) Those within chat distance
4) Those within shout distance
5) All others

For each group, one could request a timeout as function
of distance and parcel... etc. This would result in
relative short messages (even with 100 avatars in one
place) *)

However, since all this information is already known
to the client, it's possibly better to just send a
complete list to the server with a fixed timeout
for each client (of interest).

Ie:

GESTURE_REQUEST 20.0 uuid1 -1 uuid2 10.0 uuid2 10.0 uuid3 4.0 uuid4 2.5432 uuid5 1.5281

giving timeouts in seconds, -1 means wait forever (clients not mentioned would
not be waited for). If uuid1 isn't ready within 20.0 seconds (the first parameter)
the gesture is aborted.

Those messages might be relative short too, if most viewers consider only
a small number of clients relevant (ie: only mention "friends" within chat
range).

To those thinking that this "too" complex... Really...
A computer has no problems with this "complexity" whatsoever.
All that is needed is to think it over once, document it well,
and then sit back and enjoy the flexibility and powerful
possibilities for years and years that follow.

*) Ie, I would configure my client to wait 30 seconds for
   those participating in the animation, MIN(10, 40 / distance)
   seconds for those in the same parcel and within chat range.
   25.2 / distance + 0.74 seconds for those in the same parcel
   but outside the chat range. 1.0 seconds for those outside the
   same parcel but within shout range, and not at all for those
   outside the parcel and outside the shout range.
   I'd only abort an animation when anyone participating in
   the animation would not be ready within 30 seconds, or
   when it was a gesture just for me and less than half of
   my friends within a range of 10 meter would not be ready
   within a time depending on the situation (ie, I'd configure
   the viewer to wait normally the usual time that it takes
   for anyone to download the assets, and then wait until
   at least all but one friend have the data... oh wait,
   for that I'd need to be informed about who is ready,
   when they are ready...