Subject: [ogpx] My take on Teleports and protocol resilience

From: Carlo Wood <[email-masked]>
Date: Sun, 25 Oct 2009 13:15:47 +0100

Lets start with some brainstorming...

Things that might be relevant, in no particular order.

* State of avatar
* State of attachments
* Location of avatar
* Time of unresponsiveness of avatar (position) towards viewer
* Time of unresponsiveness of attachments (attaching/detaching/scripts)
* Perception of the viewer
* Perception of other viewers
* Region boundary crossing
* Teleport over larger distance
* User configurable parameters
* did I miss something important?

Real message starts here:

Something that protocol (or implementation)- wise leads to problems
are copies of the same data on different hosts; this always leads
to desynchronization of this data with heaps of untrackable problems
(bugs).

The ONLY way to keep a copy of some data reliably synchrononized is by
having a stream of state changes being sent from one host to another,
where the messages that contain that state changes always keep
the same order and the stream is never terminated/lost (in which case
a full-resynchronization would be necessary). Note that this means
that the source of the state changes has to be a single point of
origin, which automatically means that we can identify the REAL
(original) copy of the data. Thus:

  [original] ---> stream of state changes --> [copy]

Lets call this an "unidirectional state" by lack of a better word.

[PS I'm ignoring actual implementation details here. I'm NOT
    saying that currently anyone is using a TCP-stream of
    state changes, so don't quote JUST this part with the comment
    that this is not how it current works, Infinity :p
    Instead, this is an mathematical approach, the abstract
    equivalent of any possible deployment case].

Basically, we want to avoid copies of the same data.

The simulator contains a lot of state information that cannot
be moved away from the simulator though: because the state is
needed to calculate the interactions between all objects and
avatars in the region, which would become way too slow if,
for example, several agent domain services would have to be
queried all the time. I can imagine that this also holds for
the attachments on an avatar.

This means that if an avatar moves from one region to another,
all this state information has to be transfered too.

Thus, the origin sends state information to the new region.
If this fails, then we want the user to resume at it's old
location: we need to keep a copy of the state until we know
the teleport was successful. In other words, we will have
a copy of data at two different hosts, temporarily.

One way to make sure that this is not a problem is by
freezing all state; then copy it, and only once the copy
is successful, destroy the old data and resume the simulation
in the other region.

The problem we try to tackle now is the case were the old
region is not responsive...

In this case we have to notice that the AD can ALSO detect
that at least the new region is able to host the avatar:
the teleport can *partly* succeed. This can be detected
independent of copying all the state information.

Secondly, we have to realize that the 'location' of the
avatar is (state) data that does NOT have to be copied
(we're changing it anyway) and therefore is not part of
said state, and does not suffer from the copy-problem.

The same could be said about animations (which is currently
broken in Second Life: you must stop all animations
before teleporting or a desync happens): we could state
that after a teleport no animations are active, and leave
it to the viewer to re-initiate and required animations.
However, animations can be made unidirectional (meaning
that if anyone but the agent service wants to change the
animation, it has to send a request message to the AD,
which then grants it, so that the actual state change always
originates from the agent service (AS)).

As a result, we can think of the animation state as:

  [AS:animation state] --- message stream --> [simulator:animation state copy]

which means that we can teleport, keeping the correct
animation(s) without bothering with the source region.

The only clear case where this kind of 'trick' doesn't work
(I think it's not a trick, but the best way to implement this)
is for script states: those change way too frequent to host
them on the AS.

Conclusion

Thus, in the case of an unresponsive source region,
we CAN - without risks of desynchronization - immediately
transfer the avatar (location) and animations, and
any other INFREQUENTLY CHANGING, UNIDIRECTIONAL DATA
that can be stored on the AS (and buffered on the
simulator), like clothing UUIDs, and attachments.

However, the scripts in the attachments will not run
until their state is transfered from the old region.

Imho, this is hardly a problem: after user configurable
timeout we leave it to the user what he wants to do:
logout or reset the scripts :p... Ok, we just reset
the scripts after some time, where the timeout is
determined by the AD with possible input from the user.

-- 
Carlo Wood <[email-masked]>

---

From: Vaughn Deluca <[email-masked]>
Date: Sun, 25 Oct 2009 23:30:48 +0100

> [AS:animation state] --- message stream --> [simulator:animation state
copy]

What you are suggesting here is keeping a copy of the avatar state. We would
be increasing bandwith and storage space to gain flexibility. I do  not
think storage space is a real problem. I am not sure about bandwidth.

Is there anybody who can put some real numbers on this? How much extra work
would the region have to do? Is it still acceptable if the Agent service is
in Australia and the region in Europe?

-Vaughn

On Sun, Oct 25, 2009 at 1:15 PM, Carlo Wood <[email-masked]> wrote:

> _______________________________________________
> ogpx mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ogpx
>

---

From: Carlo Wood <[email-masked]>
Date: Mon, 26 Oct 2009 16:02:39 +0100

That's not how I'd put it Vaughn...

First of all, for the data proposed (for example, the currently active
animation(s)), the AS *is* the natural authority and origin to begin with.

Allow me to introduce some more theory / jargon. Consider some variable
that changes over time (lets call it 'data'). The value of the variable
plays a role in more than one host, and synchronization is done via a
network. Because networks are not reliable it must be defined who is
the authority, so that in case of failure, things can be re-synchronized
from the authority. This means automatically that the authority must
keep a copy of the data, of course.

Now consider the following:

  Authority		Remote
  value      -------->  value

I call this a 'downstream' synchronization: the data is always and only
first changed on the authority, and then propagated away from it.

'Upstream' messages that would change the value almost garanteed lead
to problems, simple example: assume some variable has value 'A',
the authority changes it to 'B', and some downstream host changes it
at the same time to 'C'. You'd then have:

  Authority                   Remote
  'B'       ---B-->  <--C---  'C'

ending in

  Authority                   Remote
  'C'                         'B'

Hence that it is much better to work with requests upstream:

  Authority                           Remote
  'B'       ---B-->  <--request:C---  'A'

  Authority                           Remote
  'C'       ---C-->                   'B'

  Authority                           Remote
  'C'                                 'C'

where there exists a race condition whether or not we end
up with 'C' or 'B', but at least things stay synchronized.

Ok, back to the animation state of an avatar.
If we state that the AS is the authority, then you can hardly
speak of 'keeping a copy', it's a trivial thing that both
the AS as well as the simulator need to know this value.

  AS                       Simulator
  'animation'    ----->    'animation'

If you'd want to avoid this (minimal!?) RAM usage on the AS
then you are forced to drop the AS as authority, and there
is no way you can recover gracefully in case of network
failure. Worse, in case of a teleport, you need to CHANGE
who the authority is, which gives heaps and heaps of potential
problems.

On Sun, Oct 25, 2009 at 11:30:48PM +0100, Vaughn Deluca wrote:
> What you are suggesting here is keeping a copy of the avatar state. We would be
> increasing bandwith and storage space to gain flexibility. I do  not think
> storage space is a real problem. I am not sure about bandwidth.

There would be NO increase in bandwidth imho, because the AS *is* the
natural authority to begin with: if the user controls the avatar, changes
it, then the source/origin is the AS (well, the viewer, but it's no
big deal to use the AS as authority; this DOES mean however that the
viewer basically only sends requests, and not real state changes).

This picture also shows that the most logical place for HUD scripts
is the AS as well, not the simulator.

> would the region have to do? Is it still acceptable if the Agent service is in
> Australia and the region in Europe?
> 
> -Vaughn
>

---

From: Vaughn Deluca <[email-masked]>
Date: Mon, 26 Oct 2009 20:22:26 +0100

> First of all, for the data proposed (for example, the currently active

LOL, I have the strong feeling we agree on almost all counts, I was merely
pointing out some side effects and trying to get a handle on the actual
costs your solution would carry. Your post explains again very clearly the
logic behind your proposal, but that was not my point. I fully agree with
you that it is a very logical and robust solution, my question related to
the efficiency costs.

At the end of your post you come to that point:

>then you are forced to drop the AS as authority,

First, my question was how much data we are *actually* talking about here.
You use the animation as an example, but there is much more in the Avatars
state than just the current animations. If we serialise the Avatars state to
send it over to the next region, how much data is that, and what if I am a
full size Dragon with a thousand active scripts and hundreds and hundreds of
attachments? And  what can we expect it to be in the future?
I *suspect* its on average not a critical amount, yet, i was hoping somebody
who *knows* would chime in.

Second, as i mentioned in my post i am not very concerned about the storage
space, but more about the *bandwidth*. In your proposal all changes pass
from viewer to AS to region instead of directly from viewer to region. No
matter how you call the info a copy, it is present at *both* locations i and
since you want to keep the AS in the loop, all updates pass trough the AS,
potentially crossing the whole real world. How much traffic is that, and
shopuld we be worried?

>and there is no way you can recover gracefully in case of
>network failure.

Right, that is the consequence of being more efficient.  I am  trying to get
some good estimates for the price of that robustness. But maybe i should
look at the sourcecode and get an estimation myself...

>Worse, in case of a teleport, you need to CHANGE

It depends what you are aiming for. You are paying in network traffic and
storage space for robustness and flexibility. In some use cases you will
want to optimise the robustness of the system yet, if i am running an Agent
Domain as a business, i might really not be pleased with he traffic your
solution makes me carry, just in case some region i do not own fails...
 Again, it might be all peanuts, but i would like to *know*

Finally, there might actually be another solution. After all, the *real*
"natural" authority is the viewer.  The viewer can keep a local copy of
avatar state. In case of a network failure the AS could ask the viewer to
send the authoritative info so it can re-create the state and send it over
to the new region. Would that work?

-Vaughn

> >      [AS:animation state] --- message stream --> [simulator:animation
> state
> >     copy]
> >

---

From: Carlo Wood <[email-masked]>
Date: Mon, 26 Oct 2009 23:03:44 +0100

On Mon, Oct 26, 2009 at 08:22:26PM +0100, Vaughn Deluca wrote:
> space, but more about the *bandwidth*. In your proposal all changes pass from
> viewer to AS to region instead of directly from viewer to region. No matter how

Nah, only the avatar state changes, which are really minimal...
The bulk, all traffic FROM the region TO the viewer can go directly
to the viewer for start (except avatar state change requests:
scripts changing the avatars animation - which is really minimal).
Also real data, like texture uploads will most likely go directly
to the simulator, but well... those are things that are not done
often:

* changing animation
* changing attachments
* changing clothes
* movement (even!)

the reason movement doesn't carry lots of data is simply because
it's generated by a few typed characters: human input.

Each of those are small messages (sone UUID's) and with a low
frequency.

In fact, I can't think of anything from viewer to simulator
that uses any significant bandwidth :/

Script state might be "significant", but 1) smaller than the
average texture, and 2) only needed when someone teleports.
Besides, scripts run on the simulator, not on the AS (although
I think HUD scripts should; but THOSE don't need their state
to be sent to the simulator).

Non-avatar state data can just go directly to the simulator
anyway, thus. For example, local chat.

---

From: Morgaine <[email-masked]>
Date: Tue, 27 Oct 2009 02:49:08 +0100

There are many different pathways and data semantics involved during a
teleport event.

Almost every permutation of one client, one AD, two regions, two RDs, and a
potentially unlimited number of asset services can yield a pathway that may
be exercised during the overall teleport avatar operation.  There is no
single rule that applies to them all and allows one to claim that it's
efficient.  And it gets even more complex when you consider that some
service pathways may be proxied by regions and others not, as a provider
design choice.

Furthermore,  more than one type of data can flow along each pathway, so
even a single pathway can involve many different semantics.  This makes
generalized answers very unhelpful.

This is why I decoupled the operation of *agent* teleport from the other *
avatar*-related operations that are triggered at the same time but which are
very distinct from agent teleport.  This allows us to make progress on TP
protocol resilience without having to consider the numerous pathways for
avatar data first.

Morgaine,

================================

On Mon, Oct 26, 2009 at 11:03 PM, Carlo Wood <[email-masked]> wrote:

> from

---

From: Vaughn Deluca <[email-masked]>
Date: Tue, 27 Oct 2009 22:32:39 +0100

OK, you convinced me regarding the bandwith. We are talking very low
frequency user generated changes, and a small amount of data is passed by
reference. Should be no problem.

I am still worried about the detour, but  i have no argument, just a gut
feeling  :)

I do not see the need to send the movement instructions via the AS. It will
ony generate unneeded delays, and that info is not part of the avatar state
anyhow.
Why "even" movment?