Subject: [ogpx] Teleports and protocol resilience

From: Morgaine <[email-masked]>
Date: Tue, 13 Oct 2009 06:13:19 +0100

One of the advantages we have in developing the VWRAP protocols is that we
are able to look back at legacy SL and Opensim protocols and recognize
design mistakes or limitations in them.  This allows us to avoid repeating
such mistakes or limitations in the next generation of systems.

One of the most common sources of frustration and dissatisfaction is
simulator non-responsiveness.  While this has many possible causes, in VWRAP
we are not interested in the internal implementation of simulators, but we *
ARE* interested in the ability of a protocol endpoint to perform its duty
within the protocol.  A jammed simulator host is in many cases quite unable
to perform its protocol duties, or in some cases only exceedingly slowly,
often timing out in a TP for example.  We have a huge amount of experience
of this happening in both SL and Opensim, so it is a practical reality.  On
occasion, simulators will be unable to fulfil their part in a protocol, and
this needs to be taken into account because it is *not uncommon*.

One key area in which the above is relevant is in teleports *OUT* of a
simulator that is under distress.  Quite often users wish nothing more than
to *leave* the region being run by a dying simulator, but when teleport-out
requires cooperation from the host that one is trying to leave then this is
often not possible at all.  In this situation, the only remedy in existing
systems is to forcibly terminate the client and relog in another region.  We
should avoid such out-of-protocol remedies being necessary through good
protocol design.

In VWRAP, we have both Rez Avatar and Derez Avatar capabilities, which lead
to corresponding protocol operations during teleport.  If R1 is a region
being run by a non-responsive simulator from which we want to escape, and R2
is another region to which we wish to go, if the protocol requires a Derez
in R1 to be completed before a Rez in R2 can commence then the user will
have difficulties.  Clearly we don't want this.

In http://tools.ietf.org/html/draft-hamrick-ogp-intro-00 , it is made clear
that "*The agent domain MUST also remove the avatar from it's current
location before placing the avatar in the destination location*."  This
suggests that the protocol will be sensitive to R1 non-responsiveness.
While we do not yet have an actual VWRAP Teleport draft, it seems likely
that its initial incarnation will have that same problem built in.

I suggest that the protocol define Derez and Rez as *concurrent* and *
non-dependent* operations to avoid this situation.  The AD can mark R1 as
disabled for all further agent state changes --- this will provide all the
protection needed to prevent brief double-presence anomalies from being
significant.  If a jammed R1 refuses to give up its hold on the avatar, then
at least the user will not suffer from it.  Reaping dead simulator sessions
then becomes a problem for the region operator alone, and not for the AD,
the user, and the region as happens now.

Morgaine.

---

From: Kari Lippert <[email-masked]>
Date: Tue, 13 Oct 2009 01:25:59 -0400

+1

Kari

On Tue, Oct 13, 2009 at 1:13 AM, Morgaine <[email-masked]>wrote:

> we are not interested in the internal implementation of simulators, but we
> *ARE* interested in the ability of a protocol endpoint to perform its duty
> to *leave* the region being run by a dying simulator, but when
> teleport-out requires cooperation from the host that one is trying to leave
> then this is often not possible at all.  In this situation, the only remedy
> in existing systems is to forcibly terminate the client and relog in another
> region.  We should avoid such out-of-protocol remedies being necessary
> through good protocol design.
>
> In http://tools.ietf.org/html/draft-hamrick-ogp-intro-00 , it is made
> clear that "*The agent domain MUST also remove the avatar from it's
> current location before placing the avatar in the destination location*."
> This suggests that the protocol will be sensitive to R1 non-responsiveness.
> _______________________________________________
> ogpx mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ogpx
>
>

---

From: "Infinity Linden (Meadhbh Hamrick)" <[email-masked]>
Date: Tue, 13 Oct 2009 05:20:15 -0700

the reason the teleport process removes a user from one region before
placing it in another is to ensure that the user's avatar exists in at
most one place. i understand that this is a non-issue with hypergrid
or tourist models, as there's no assumption of coherence between
regions. but, many of us want to provide a virtual world experience
that includes this concept.

thx
m/∞
--
   infinity linden (aka meadhbh hamrick)  *  it's pronounced "maeve"
         http://wiki.secondlife.com/wiki/User:Infinity_Linden

On Mon, Oct 12, 2009 at 22:25, Kari Lippert <[email-masked]> wrote:
> I suggest that the protocol define Derez and Rez
> as concurrent and non-dependent operations to avoid this situation.  The AD
> can mark R1 as disabled for all further agent state changes --- this will
> provide all the protection needed to prevent brief double-presence anomalies
> from being significant.  If a jammed R1 refuses to give up its hold on the
> avatar, then at least the user will not suffer from it.  Reaping dead
> simulator sessions then becomes a problem for the region operator alone, and
> not for the AD, the user, and the region as happens now.
>
> +1
>
> Kari
>
>
>> ARE interested in the ability of a protocol endpoint to perform its duty
>> this needs to be taken into account because it is not uncommon.
>>
>> One key area in which the above is relevant is in teleports OUT of a
>> to leave the region being run by a dying simulator, but when teleport-out
>> In VWRAP, we have both Rez Avatar and Derez Avatar capabilities, which
>> lead to corresponding protocol operations during teleport.  If R1 is a
>> region being run by a non-responsive simulator from which we want to escape,
>> and R2 is another region to which we wish to go, if the protocol requires a
>> Derez in R1 to be completed before a Rez in R2 can commence then the user
>> will have difficulties.  Clearly we don't want this.
>>
>> clear that "The agent domain MUST also remove the avatar from it's current
>> location before placing the avatar in the destination location."  This
>> I suggest that the protocol define Derez and Rez as concurrent and
>> non-dependent operations to avoid this situation.  The AD can mark R1 as

---

From: Joshua Bell <[email-masked]>
Date: Tue, 13 Oct 2009 08:53:15 -0700

> the user, and the region as happens now.

Unresponsive origin regions are an excellent use case to consider.

One of the reasons that this fails today is that the origin RD maintains
some amount of agent state (e.g. script state for attachments). For a
sim-to-sim handoff, the origin RD must successfully transmit information
either to the AD or the destination RD (or both), otherwise the user
experience is poor (information is lost).  Note that this needs to happen
during a simple logoff/login as well. It seems to me that agent state in the
AD should have ACID-ish characteristics, including the agent's current
location.

Given that we must be able to describe behavior of closing a client-AD
connection when the RD is unresponsive (e.g. the host has dropped off the
network, etc), I'd be inclined to tackle unresponsive origins in similar
fashion for logout/login, region crossing, and teleport. (Which in the
current drafts are indeed tackled together.)

So: I would agree that the current behavior of SL is less than ideal - if
the source region is unresponsive (for some TBD definition), a teleport
should probably not fail. But to preserve user data (and preserve the user
experience), the initial transport attempt must involve a transaction
between the AD, source RD and destination RD.

---

From: David W Levine <[email-masked]>
Date: Tue, 13 Oct 2009 12:25:16 -0400

I think this gets even messier when you look at multiple deployment 
patterns. With a range of possible ways people might build out the 
services, its going to be interesting to work out what one expects at a 
region That said, one assumes that the key services which matter for 
attempting to determine if a "region" is live are those which emit the 
agent's in world component's current state. Working out what's needed, and 
why, might well, help us put some bounds on this. 

- David

Joshua Bell <[email-masked]> 
Sent by: [email-masked]
10/13/2009 11:53 AM

To
[email-masked]
cc

Subject
Re: [ogpx] Teleports and protocol resilience

On Mon, Oct 12, 2009 at 10:13 PM, Morgaine <[email-masked]
> wrote:
I suggest that the protocol define Derez and Rez as concurrent and 
non-dependent operations to avoid this situation.  The AD can mark R1 as 
disabled for all further agent state changes --- this will provide all the 
protection needed to prevent brief double-presence anomalies from being 
significant.  If a jammed R1 refuses to give up its hold on the avatar, 
then at least the user will not suffer from it.  Reaping dead simulator 
sessions then becomes a problem for the region operator alone, and not for 
the AD, the user, and the region as happens now.

One of the reasons that this fails today is that the origin RD maintains 
some amount of agent state (e.g. script state for attachments). For a 
sim-to-sim handoff, the origin RD must successfully transmit information 
either to the AD or the destination RD (or both), otherwise the user 
experience is poor (information is lost).  Note that this needs to happen 
during a simple logoff/login as well. It seems to me that agent state in 
the AD should have ACID-ish characteristics, including the agent's current 
location.

---

From: Meadhbh Hamrick <[email-masked]>
Date: Tue, 13 Oct 2009 09:33:00 -0700

josh: two phase commit for teleport? i guess it _could_ work, but it would
make things more complicated.
david: this is why i want to limit the number of supported deployment
patterns. while it would be a great research project to work out a "design
language" for describing deployments, i think we should stick with the few
deployment patterns we have before us: Second Life, Second Life with client
mediated Services, Tourist and OpenSim standalone mode. (OpenSim UGAIM grid
mode would probably be something like "Second Life with client mediated
Services")

my primary concern is, i don't want to standardize a teleport protocol that
works ONLY with the tourist model.

On Tue, Oct 13, 2009 at 9:25 AM, David W Levine <[email-masked]> wrote:

> patterns. With a range of possible ways people might build out the services,
> its going to be interesting to work out what one expects at a region That
> said, one assumes that the key services which matter for attempting to
> determine if a "region" is live are those which emit the agent's in world
> component's current state. Working out what's needed, and why, might well,
> help us put some bounds on this.
>
> - David
>
>
>
>
>  *Joshua Bell <[email-masked]>*
> [email-masked]  cc
> On Mon, Oct 12, 2009 at 10:13 PM, Morgaine <*
> [email-masked]* <[email-masked]>> wrote:

---

From: Mike Dickson <[email-masked]>
Date: Tue, 13 Oct 2009 11:55:33 -0500

On Tue, 2009-10-13 at 15:53 +0000, Joshua Bell wrote:

> One of the reasons that this fails today is that the origin RD
> maintains some amount of agent state (e.g. script state for
> attachments). For a sim-to-sim handoff, the origin RD must
> successfully transmit information either to the AD or the destination
> RD (or both), otherwise the user experience is poor (information is
> lost).  Note that this needs to happen during a simple logoff/login as
> well. It seems to me that agent state in the AD should have ACID-ish
> characteristics, including the agent's current location.

The scripting comment is interesting re: interoperability also.  What
does that imply if I'm going from an OpenSim instance for example which
has additional scripting functions enabled that a SL region might not
support.  How does the destination region determine it can run the
scripts in attachments and that its safe to do so?

Mike

---

From: Charles Krinke <[email-masked]>
Date: Tue, 13 Oct 2009 10:09:10 -0700 (PDT)

Good point, Mike. I suspect the answer is going to be in how the protocol handles 'graceful' failure for teleports when the 'toRegion' has one or more settings that are different then the 'fromRegion'.

Charles

________________________________
From: Mike Dickson <[email-masked]>
To: Joshua Bell <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: Tuesday, October 13, 2009 9:55:33 AM
Subject: Re: [ogpx] Teleports and protocol resilience

Mike

---

From: Joshua Bell <[email-masked]>
Date: Tue, 13 Oct 2009 10:25:10 -0700

> scripts in attachments and that its safe to do so?
Awesome scenario to dive into.

FWIW, we've dodged the bullet on this so far with Second Life. While an SL
grid can operate in a heterogeneous state, we typically deploy new features
with forward/backward compat issues in a "disabled" state until all hosts
are updated to support the new feature, then flip a switch to enable it. In
the script case, this would mean that the ability to author/compile scripts
with new functionality would be disabled grid-wide until all hosts had been
updated. Obviously that doesn't work once agents and data are crossing
authoritative domains. :)

Can any implementers of OpenSim (or other non-SL systems using the
agent/region paradigm) describe how this is handled in those systems today?
Is there region-specific state to an agent (attachments, scripts, etc) and
how is this updated/maintained? How much effort falls on the content creator
vs. the system administrators, how aware of this is the user, etc?

---

From: Vaughn Deluca <[email-masked]>
Date: Tue, 13 Oct 2009 19:31:30 +0200

Regarding the serialisation and passing-off of avatar data, the present TP
draft keeps the source region in the communication path, even *after* a
successful Derez and serialisation. The result from the new Rez is passed
via the source region back to the agent domain.
There might be good reasons for that path (efficiency?), but for sure it
complicates things if the source region is dying.

@Joshua
RD is not the same as Region, in the TP drafs its the Region(service) that
has the interface.

@Infinity, the tourist use case is not very well defined as yet (working on
that) but in my interpretation if will for sure specify the same constraint
of "no duplication of avatars".

-Vaughn

---

From: Joshua Bell <[email-masked]>
Date: Tue, 13 Oct 2009 11:12:18 -0700

> RD is not the same as Region, in the TP drafs its the Region(service) that
Oops - you're right, of course. Everything we're talking about on this
thread can occur within a single RD or between two RDs. I was sloppily using
"RD" as shorthand for "collection of services that implement region
behavior" but in this case we need to be more precise.

---

From: Vaughn Deluca <[email-masked]>
Date: Tue, 13 Oct 2009 20:30:57 +0200

:) I fell in that trap many times.
But what about the the entity to serialise the avatar data to at derez? You
mentioned either agent domain or destination region.
the tp draft uses the region, with as a consequence a second pass trough the
source region.
What are the argument pro and contra this solution compared to serialising
to the AD?
I have the feeling the slightly non-obvious path in the draft was picked for
a good reason, but i do not know enough about the details to see why.

-Vaughn

---

From: Meadhbh Hamrick <[email-masked]>
Date: Tue, 13 Oct 2009 12:40:28 -0700

hmm.. there were several iterations of the teleport protocol. i'm not
really sure this is how it was deployed. and if it is, i would argue
it _shouldn't_ be deployed this way 'cause of the trust issue.

in the typical "Second Life" use case, the client trusts the agent
domain, and the agent domain trusts region domains 1 and 2. the two
region domains do not necessarily trust each other. or rather, there
may be utility to a future where all region domains do not need to
directly trust all other region domains. in the "tourist" use case,
the agent domain can be seen as an actor of the client, so it's really
the client who directly trusts the two region domains. but, you retain
the benefit when you do not depend on the two region domains trusting
each other.

man. we seriously need some diagrams.

-cheers
-meadhbh/infinity

On Tue, Oct 13, 2009 at 10:31 AM, Vaughn Deluca <[email-masked]> wrote:
>> <[email-masked]> wrote:
>>>

---

From: Dan Olivares <[email-masked]>
Date: Tue, 13 Oct 2009 15:55:04 -0400

One option here is to have a timeout within the agent domain for a
response from the source region with a cap for the destination region
and go over the source region's head to directly request it from the
destination region.      While this isn't always optimal (in the
script and attachment state), it would provide a 'graceful failure'
scenario.     This would also necessitate the need to describe some
kind of a priority 'state backup' mechanism that gets saved to
periodically for when failures occur. It's my guess that some state is
more important then others..   such as..    financial transaction
state..  is more important then say..   who's within 20m on a sensor.
Even though we've not discussed inventory itself yet, it would seem
like we're heading in the direction of having the Agent Domain manage
inventory, therefore the inventory transactions that take place
themselves wouldn't be affected..   just the state of any item that's
'rezzed' in a simulator and not instanced (Attachments).    Would
people be okay with having old data on their attachments if they
didn't have to log-off and log back on in a failure case?

Regards

Dan Olivares

---

From: Meadhbh Hamrick <[email-masked]>
Date: Tue, 13 Oct 2009 13:45:36 -0700

ack. this was supposed to go out to the list.

On Tue, Oct 13, 2009 at 1:36 PM, Meadhbh Hamrick
<[email-masked]> wrote:
> but before we go too far down the rabbit hole with solutions, can we
> make sure we understand the problem domain?
>
> if i understand it, there's a preference that a user's avatar be
> rezzable in the destination region before it is derez'd in the source
> region. so in other words, we don't want to derez the user's av from
> where it is until we know it can be rezzed where it's going.
>
> is this a fair discussion of the problem? i mean irrespective of the
> proposed solutions, this is a feature of teleport that we want, right?
>
> assuming there is consensus on this point, we may want to gingerly
> inject the requirement that the agent domain or the destination region
> domain MAY want to deny the user's teleport. (remember the fuss over
> adult content and underaged users?)
>
> it seems that this is a reasonable place to consider this topic.
> however, it doesn't seem like there's radical consensus over the
> question of whether it's the AD's responsibility to block underage rez
> attempts in adult sims or the destination RD's responsibility or both.
> so when discussing teleport, maybe we could add some verbiage like
> "the AD has the option to block the teleport request here" or "the RD
> has the option to block the teleport here" in flow diagrams.
>
> -cheers
> -meadhbh
>
> On Tue, Oct 13, 2009 at 12:40 PM, Meadhbh Hamrick

---

From: Vaughn Deluca <[email-masked]>
Date: Wed, 14 Oct 2009 00:17:05 +0200

On Tue, Oct 13, 2009 at 10:36 PM, Meadhbh Hamrick <[email-masked]

> proposed solutions, this is a feature of teleport that we want, right?

Yes, i definitely think so.

> attempts in adult sims or the destination RD's responsibility or both.

I think there was reasonable consensus that both can block the TP, simply
because its local policy that rules.

> has the option to block the teleport here" in flow diagrams.

Absolutely, and I did so in this example:
 http://www.ietf.org/mail-archive/web/ogpx/current/msg00570.html

:)
-Vaughn

> maintains

---

From: Morgaine <[email-masked]>
Date: Fri, 23 Oct 2009 17:47:12 +0100

Looking back at the replies in this thread, I think that the goal and the
means to achieve it didn't quite come across.

I was trying to address only a very specific issue, just protocol resilience
under source region non-responsiveness, since this is common enough that it
merits addressing.  I did not suggest that there be any perceivable change
of teleport semantics under normal operation (because no such change is
needed), only a change in service coupling.  The semantics we experience in
SL and in Opensim would remain completely unchanged, except in the single
case of source region non-responsiveness.  Under this single anomalous case
there *would* be a perceivable change, but that change would be a huge
improvement.

There would be no new decoherence introduced since exactly the same state
changes would occur on TP as before, with no possibiity of agent state
change in the source region once the AD accepts the TP.

All that's needed to achieve such resilience for teleport at the protocol
level is a slight revision of operation phasing to permit greater execution
overlap, as I outlined.  This is independent of anything else that happens
in the course of the overall teleport operation --- the change of phasing
would affect only the transfer of *agent location* alone, nothing else.

In particular, it should not be confused with the separate requirement for
instantiation of assets or objects at destination, nor with the matter of
serializing and deserializing script states.  The latter has not even been
defined for VWRAP, so it's hard to talk about changing it.  In any event,
this isn't about those aspects of teleport, and doesn't affect them --- they
would continue to work as before.

One of the central aspects of VWRAP is that the protocol is based on a
multiple services model, and one of the key approaches in highly scalable
systems design is to keep services decoupled to the largest extent
possible.  That's what I'm proposing here, a *partial decoupling* that has
no normal semantic change but which does have benefits in anomalous
situations.

Agent location change *can* be decoupled significantly from asset
instantiation change and script state transfer.  My suggestion referred to
this decoupled *agent location change* only, not to asset and simulation
services.  Those other two services undergo state transitions at the same
time as change of agent location does on TP, but services should never be
coupled together unnecessarily, and in this case the coupling can be left
very weak.  The three types of service operations can proceed each at their
own independent rates, coupled at TP initiation time and nowhere else.

It should be noted that the legacy protocols do some of this already, in
that the agent is already active in the destination region long before her
avatar or objects have appeared.  Furthermore, the avatar currently
continues to be visible in the source region for a while after the agent
becomes active in the destination region, because of normal operation
latencies, sim-side queueing, and client lag.  This is a normal part of
current operation, and is not considered an anomaly. What's important is
that no new state change to the agent is possible in the source region after
TP is initiated, and that would remain true.

The impact of this on the other parts of the puzzle needs to wait until
those other parts are examined.  We're not there yet, but I would hope that
improving teleport protocol resilience would be a desireable goal when the
only noticeable change in semantics occurs under fault conditions and
provides a major improvement on current behaviour.

Morgaine.

======================================

---

From: David W Levine <[email-masked]>
Date: Fri, 23 Oct 2009 14:09:50 -0400

This is a good point, and highlights a very tricky issue in any 
distributed system design. A protocol should behave gracefully in the face 
of failure. One of the definite challenges here is partition vs. failure. 
Its impossible to distinguish between a failed service and one which is 
lost behind a network partition. In the case of a partition, with an 
avatar continuing to run on momentarily isolated region, care needs to be 
taken to recover gracefully not only from the "can't get to the region 
service" to get it to release the agent, but also the "The region has 
returned to contact"  So,I think  when we talk about being resilient we 
need to think about the range of failure, from "the service I was using 
has crashed" to "the service I was using has become invisible due
to a network partition." Of course, they look identical. This rather 
strongly implies good semantics on what happens to a "Stub" session which 
becomes isolated as well as the "new" session
which we create by bypassing a failed service. (In the case of teleport, 
the old region's idea of agent state and the new ones, in other cases, 
possibly other isolated state) 

Food for thought

- David
~ Zha

Morgaine <[email-masked]> 
Sent by: [email-masked]
10/23/2009 12:47 PM

To
[email-masked]
cc

Subject

I was trying to address only a very specific issue, just protocol 
resilience under source region non-responsiveness, since this is common 
enough that it merits addressing.  I did not suggest that there be any 
perceivable change of teleport semantics under normal operation (because 
no such change is needed), only a change in service coupling.  The 
semantics we experience in SL and in Opensim would remain completely 
unchanged, except in the single case of source region non-responsiveness.  
Under this single anomalous case there would be a perceivable change, but 
that change would be a huge improvement.

All that's needed to achieve such resilience for teleport at the protocol 
level is a slight revision of operation phasing to permit greater 
execution overlap, as I outlined.  This is independent of anything else 
that happens in the course of the overall teleport operation --- the 
change of phasing would affect only the transfer of agent location alone, 
nothing else.

In particular, it should not be confused with the separate requirement for 
instantiation of assets or objects at destination, nor with the matter of 
serializing and deserializing script states.  The latter has not even been 
defined for VWRAP, so it's hard to talk about changing it.  In any event, 
this isn't about those aspects of teleport, and doesn't affect them --- 
they would continue to work as before.

possible.  That's what I'm proposing here, a partial decoupling that has 

Agent location change can be decoupled significantly from asset 
instantiation change and script state transfer.  My suggestion referred to 
this decoupled agent location change only, not to asset and simulation 
services.  Those other two services undergo state transitions at the same 
time as change of agent location does on TP, but services should never be 
coupled together unnecessarily, and in this case the coupling can be left 
very weak.  The three types of service operations can proceed each at 
their own independent rates, coupled at TP initiation time and nowhere 
else.

It should be noted that the legacy protocols do some of this already, in 
that the agent is already active in the destination region long before her 
avatar or objects have appeared.  Furthermore, the avatar currently 
continues to be visible in the source region for a while after the agent 
becomes active in the destination region, because of normal operation 
latencies, sim-side queueing, and client lag.  This is a normal part of 
current operation, and is not considered an anomaly. What's important is 
that no new state change to the agent is possible in the source region 
after TP is initiated, and that would remain true.

The impact of this on the other parts of the puzzle needs to wait until 
those other parts are examined.  We're not there yet, but I would hope 
that improving teleport protocol resilience would be a desireable goal 
when the only noticeable change in semantics occurs under fault conditions 
and provides a major improvement on current behaviour.

Morgaine.

On Tue, Oct 13, 2009 at 6:13 AM, Morgaine <[email-masked]> 
wrote:

One of the most common sources of frustration and dissatisfaction is 
simulator non-responsiveness.  While this has many possible causes, in 
VWRAP we are not interested in the internal implementation of simulators, 
but we ARE interested in the ability of a protocol endpoint to perform its 
duty within the protocol.  A jammed simulator host is in many cases quite 
unable to perform its protocol duties, or in some cases only exceedingly 
slowly, often timing out in a TP for example.  We have a huge amount of 
experience of this happening in both SL and Opensim, so it is a practical 
reality.  On occasion, simulators will be unable to fulfil their part in a 
protocol, and this needs to be taken into account because it is not 
uncommon.

One key area in which the above is relevant is in teleports OUT of a 
simulator that is under distress.  Quite often users wish nothing more 
than to leave the region being run by a dying simulator, but when 
teleport-out requires cooperation from the host that one is trying to 
leave then this is often not possible at all.  In this situation, the only 
remedy in existing systems is to forcibly terminate the client and relog 
in another region.  We should avoid such out-of-protocol remedies being 
necessary through good protocol design.

In VWRAP, we have both Rez Avatar and Derez Avatar capabilities, which 
lead to corresponding protocol operations during teleport.  If R1 is a 
region being run by a non-responsive simulator from which we want to 
escape, and R2 is another region to which we wish to go, if the protocol 
requires a Derez in R1 to be completed before a Rez in R2 can commence 
then the user will have difficulties.  Clearly we don't want this.

Morgaine.

---

From: Morgaine <[email-masked]>
Date: Sat, 24 Oct 2009 04:45:40 +0100

Excellent observation, David!

To use a different form of words, fault conditions can occur in all the
possible states of the system, and if we want the protocol to be robust then
we'll probably have to examine all its major states and ensure that there
are always state transitions available out of them.

In that light, being able to TP out of a stuck region is just one of those
recovery transitions without which the protocol would simply break as a
result of the exception.  There is of course no need to design VWRAP such
that exception handling always ends in session termination --- this would
result in a very poor user experience by default, and so it should be
avoided wherever possible.

My suggestion is therefore likely to be just one of many designed to avoid
session termination under conditions that, while not normal, are
nevertheless unfortunately rather common in practice.  The user's client
application should never be forced to disconnect completely (or even worse
to terminate) unless there is absolutely no alternative, and it's up to us
to provide such alternatives wherever we can.

Breaking the user's immersive experience through termination should only be
a solution of last resort.

Morgaine.

====================================

On Fri, Oct 23, 2009 at 7:09 PM, David W Levine <[email-masked]> wrote:

>
> This is a good point, and highlights a very tricky issue in any distributed
> system design. A protocol should behave gracefully in the face of failure.
> One of the definite challenges here is partition vs. failure. Its impossible
> to distinguish between a failed service and one which is lost behind a
> network partition. In the case of a partition, with an avatar continuing to
> run on momentarily isolated region, care needs to be taken to recover
> gracefully not only from the "can't get to the region service" to get it to
> release the agent, but also the "The region has returned to contact"  So,I
> think  when we talk about being resilient we need to think about the range
> of failure, from "the service I was using has crashed" to "the service I was
> using has become invisible due
>  *Morgaine <[email-masked]>*
> perceivable change of teleport semantics under normal operation (because no
> such change is needed), only a change in service coupling.  The semantics we
> experience in SL and in Opensim would remain completely unchanged, except in
> the single case of source region non-responsiveness.  Under this single
> anomalous case there *would* be a perceivable change, but that change
> would be a huge improvement.
>
> On Tue, Oct 13, 2009 at 6:13 AM, Morgaine <[email-masked]
> * <[email-masked]>> wrote:
> In *http://tools.ietf.org/html/draft-hamrick-ogp-intro-00*<http://tools.ietf.org/html/draft-hamrick-ogp-intro-00>, it is made clear that "
> *The agent domain MUST also remove the avatar from it's current location
> before** placing the avatar in the destination location*."  This suggests
> that the protocol will be sensitive to R1 non-responsiveness.  While we do
> not yet have an actual VWRAP Teleport draft, it seems likely that its
> initial incarnation will have that same problem built in.
>

---

From: Lawson English <[email-masked]>
Date: Fri, 23 Oct 2009 23:21:19 -0700

Morgaine wrote:
> To use a different form of words, fault conditions can occur in all 
> the possible states of the system, and if we want the protocol to be 
> robust then we'll probably have to examine all its major states and 
> ensure that there are always state transitions available out of them.
>
> In that light, being able to TP out of a stuck region is just one of 
> those recovery transitions without which the protocol would simply 
> break as a result of the exception.  There is of course no need to 
> design VWRAP such that exception handling always ends in session 
> termination --- this would result in a very poor user experience by 
> default, and so it should be avoided wherever possible.
>
> My suggestion is therefore likely to be just one of many designed to 
> avoid session termination under conditions that, while not normal, are 
> nevertheless unfortunately rather common in practice.  The user's 
> client application should never be forced to disconnect completely (or 
> even worse to terminate) unless there is absolutely no alternative, 
> and it's up to us to provide such alternatives wherever we can.
>
> Breaking the user's immersive experience through termination should 
> only be a solution of last resort.
>
>
> Morgaine.
>
>
>
>
>
> On Fri, Oct 23, 2009 at 7:09 PM, David W Levine <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     distributed system design. A protocol should behave gracefully in
>     the face of failure. One of the definite challenges here is
>     partition vs. failure. Its impossible to distinguish between a
>     failed service and one which is lost behind a network partition.
>     In the case of a partition, with an avatar continuing to run on
>     momentarily isolated region, care needs to be taken to recover
>     gracefully not only from the "can't get to the region service" to
>     get it to release the agent, but also the "The region has returned
>     to contact"  So,I think  when we talk about being resilient we
>     need to think about the range of failure, from "the service I was
>     using has crashed" to "the service I was using has become
>     invisible due
>     to a network partition." Of course, they look identical. This
>     rather strongly implies good semantics on what happens to a "Stub"
>     session which  becomes isolated as well as the "new" session
>     which we create by bypassing a failed service. (In the case of
>     teleport, the old region's idea of agent state and the new ones,
>     in other cases, possibly other isolated state)  
>
>     Food for thought
>
>     - David
>     ~ Zha
>
>
>
>     *Morgaine <[email-masked]
>     <mailto:[email-masked]>>*
>     Sent by: [email-masked] <mailto:[email-masked]>
>
>     10/23/2009 12:47 PM
>
>     	
>     To
>     	[email-masked] <mailto:[email-masked]>
>     cc
>     	
>     Subject
>     Looking back at the replies in this thread, I think that the goal
>     and the means to achieve it didn't quite come across.
>
>     resilience under source region non-responsiveness, since this is
>     common enough that it merits addressing.  I did not suggest that
>     there be any perceivable change of teleport semantics under normal
>     operation (because no such change is needed), only a change in
>     service coupling.  The semantics we experience in SL and in
>     Opensim would remain completely unchanged, except in the single
>     case of source region non-responsiveness.  Under this single
>     anomalous case there */would/* be a perceivable change, but that
>     change would be a huge improvement.
>
>     There would be no new decoherence introduced since exactly the
>     same state changes would occur on TP as before, with no possibiity
>     of agent state change in the source region once the AD accepts the TP.
>
>     All that's needed to achieve such resilience for teleport at the
>     protocol level is a slight revision of operation phasing to permit
>     greater execution overlap, as I outlined.  This is independent of
>     anything else that happens in the course of the overall teleport
>     operation --- the change of phasing would affect only the transfer
>     of /agent location/ alone, nothing else.
>
>     In particular, it should not be confused with the separate
>     requirement for instantiation of assets or objects at destination,
>     nor with the matter of serializing and deserializing script
>     states.  The latter has not even been defined for VWRAP, so it's
>     hard to talk about changing it.  In any event, this isn't about
>     those aspects of teleport, and doesn't affect them --- they would
>     continue to work as before.
>
>     One of the central aspects of VWRAP is that the protocol is based
>     on a multiple services model, and one of the key approaches in
>     highly scalable systems design is to keep services decoupled to
>     the largest extent possible.  That's what I'm proposing here, a
>     /partial decoupling/ that has no normal semantic change but which
>     does have benefits in anomalous situations.
>
>     Agent location change /can/ be decoupled significantly from asset
>     instantiation change and script state transfer.  My suggestion
>     referred to this decoupled /agent location change/ only, not to
>     asset and simulation services.  Those other two services undergo
>     state transitions at the same time as change of agent location
>     does on TP, but services should never be coupled together
>     unnecessarily, and in this case the coupling can be left very
>     weak.  The three types of service operations can proceed each at
>     their own independent rates, coupled at TP initiation time and
>     nowhere else.
>
>     It should be noted that the legacy protocols do some of this
>     already, in that the agent is already active in the destination
>     region long before her avatar or objects have appeared. 
>     Furthermore, the avatar currently continues to be visible in the
>     source region for a while after the agent becomes active in the
>     destination region, because of normal operation latencies,
>     sim-side queueing, and client lag.  This is a normal part of
>     current operation, and is not considered an anomaly. What's
>     important is that no new state change to the agent is possible in
>     the source region after TP is initiated, and that would remain true.
>
>     The impact of this on the other parts of the puzzle needs to wait
>     until those other parts are examined.  We're not there yet, but I
>     would hope that improving teleport protocol resilience would be a
>     desireable goal when the only noticeable change in semantics
>     occurs under fault conditions and provides a major improvement on
>     current behaviour.
>
>
>     Morgaine.
>
>
>
>
>
>
>     On Tue, Oct 13, 2009 at 6:13 AM, Morgaine
>     <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     One of the advantages we have in developing the VWRAP protocols is
>     that we are able to look back at legacy SL and Opensim protocols
>     and recognize design mistakes or limitations in them.  This allows
>     us to avoid repeating such mistakes or limitations in the next
>     generation of systems.
>
>     One of the most common sources of frustration and dissatisfaction
>     is simulator non-responsiveness.  While this has many possible
>     causes, in VWRAP we are not interested in the internal
>     implementation of simulators, but we *ARE* interested in the
>     ability of a protocol endpoint to perform its duty within the
>     protocol.  A jammed simulator host is in many cases quite unable
>     to perform its protocol duties, or in some cases only exceedingly
>     slowly, often timing out in a TP for example.  We have a huge
>     amount of experience of this happening in both SL and Opensim, so
>     it is a practical reality.  On occasion, simulators will be unable
>     to fulfil their part in a protocol, and this needs to be taken
>     into account because it is /not uncommon/.
>
>     One key area in which the above is relevant is in teleports *OUT*
>     of a simulator that is under distress.  Quite often users wish
>     nothing more than to /leave/ the region being run by a dying
>     simulator, but when teleport-out requires cooperation from the
>     host that one is trying to leave then this is often not possible
>     at all.  In this situation, the only remedy in existing systems is
>     to forcibly terminate the client and relog in another region.  We
>     should avoid such out-of-protocol remedies being necessary through
>     good protocol design.
>
>     In VWRAP, we have both Rez Avatar and Derez Avatar capabilities,
>     which lead to corresponding protocol operations during teleport. 
>     If R1 is a region being run by a non-responsive simulator from
>     which we want to escape, and R2 is another region to which we wish
>     to go, if the protocol requires a Derez in R1 to be completed
>     before a Rez in R2 can commence then the user will have
>     difficulties.  Clearly we don't want this.
>
>     In _http://tools.ietf.org/html/draft-hamrick-ogp-intro-00_ , it is
>     made clear that "/The agent domain MUST also remove the avatar
>     from it's current location *before*// placing the avatar in the
>     destination location/."  This suggests that the protocol will be
>     sensitive to R1 non-responsiveness.  While we do not yet have an
>     actual VWRAP Teleport draft, it seems likely that its initial
>     incarnation will have that same problem built in.
>
>     I suggest that the protocol define Derez and Rez as */concurrent/*
>     and */non-dependent/* operations to avoid this situation.  The AD
>     can mark R1 as disabled for all further agent state changes ---
>     this will provide all the protection needed to prevent brief
>     double-presence anomalies from being significant.  If a jammed R1
>     refuses to give up its hold on the avatar, then at least the user
>     will not suffer from it.  Reaping dead simulator sessions then
>     becomes a problem for the region operator alone, and not for the
>     AD, the user, and the region as happens now.
>
Beware of one-size-fits-all requirements. I can easily imagine there 
will be users who wouldn't want to have a degraded/alternate/unexpected 
experience on TP failure and who WOULD want the system just to reset. 
And no doubt other scenarios will make sense as well. After all, if 
someone uses VWRAP as the foundation for a gaming platform, they may 
have completely different rules that they expect users participating to 
follow as long as they are participating.

Lawson

Lawson

---

From: Morgaine <[email-masked]>
Date: Sat, 24 Oct 2009 11:08:33 +0100

Indeed!   Not only are extra user options always a good thing, but a TP from
a region in one world could be to a region in a highly different world with
very specific requirements.  As you point out, "one size fits all" is
definitely inappropriate.

In this particular case, that could be handled by the AD sending the client
an event notification to the effect that the source region did not respond
within the timeout period.

The client would then be free to implement disconnection if this is the
user's configured choice.  I think it's fair to expect that most people
would not choose that option, but it should certainly be available to them
if needed.

The agent would probably already be located in the destination region by
this time from the AD's perspective (depending on timout length), but not
necessarily from the client's perspective --- that's a matter of client
implementation and user options.  Some Second Life clients already implement
"Don't blank screen on TP", "Don't show TP progress bar" and "Display avatar
before wearables are available" as options, so they're already handling the
component parts of the overall TP operation quite flexibly.

The AD of a world may itself have a policy of not allowing TPs to complete
when the source region is unavailable, but this should not be hardwired into
the protocol.  As you say, one size does not fit all, particularly in the
case of choices that provide a bad user experience.

Morgaine.

===========================================

On Sat, Oct 24, 2009 at 7:21 AM, Lawson English <[email-masked]> wrote:

>> Breaking the user's immersive experience through termination should only
>> be a solution of last resort.
>>
>>
>> Morgaine.
>>
>>
>>
>>
>>
>> On Fri, Oct 23, 2009 at 7:09 PM, David W Levine <[email-masked] <mailto:
>> [email-masked]>> wrote:
>>
>>
>>    region long before her avatar or objects have appeared.    Furthermore,
>> the avatar currently continues to be visible in the
>>    which lead to corresponding protocol operations during teleport.    If
>> R1 is a region being run by a non-responsive simulator from
>>  Beware of one-size-fits-all requirements. I can easily imagine there will
> be users who wouldn't want to have a degraded/alternate/unexpected
> experience on TP failure and who WOULD want the system just to reset. And no
> doubt other scenarios will make sense as well. After all, if someone uses
> VWRAP as the foundation for a gaming platform, they may have completely
> different rules that they expect users participating to follow as long as
> they are participating.
>
>
> Lawson
>
> Lawson
>

---

From: Vaughn Deluca <[email-masked]>
Date: Sat, 24 Oct 2009 15:59:18 +0200

Morgaine,

If I understand you right, you are suggesting the following:

- The agent domain is able to complete a teleport (after a timeout period)
even if the original region completley fails to respond, so *even* if there
is no guarantee the invocation of the derez cap has been processed.

- To prevent true duplication of avatars, the again domain keeps some record
of the old connection, to be able to deal gracefully with the  case that the
old region comes back to life. This implies that in some cases two copies of
the avatar might be visible, one "dead body" in the old region, and the live
version in the new region. This might in some cases be preferable to a
stranded avatar, yet in other situation could be unacceptable (e.g. in a
combat game, were mere visual presence might be enough to influence the
game).

-Therefore, in spite of the freedom of the agent domain to complete a tp
from a non-responding region, its left up to the region to decide if it
wants to accept the rezzing of an avatar that has not  completed the
de-rezzing in the old region.

- Since there is no way to get the most recent agent state from the old
region, this special TP without conformation from the originating region
would bring back the agent in some default state in the new region. This
process would lead to a graceful degradation of the system. With some clever
setting of defaults it might not be too disturbing for the avatar.  It is
left up to the client to instruct the agent service about the way to deal
with these types of anomalies.

-Finally, your suggestions imply that the Agent  service MAY acts as an
intermediate in the TP, rather than the two regions communicating directly
as the current TP draft specifies.

At first sight is all looks reasonably realistic, yet it is for sure more
complicated than the current TP draft, and I am not really able to judge
what we are actually bringing onto our selfs here.

- Vaughn
.

On Sat, Oct 24, 2009 at 12:08 PM, Morgaine
<[email-masked]>wrote:

> Indeed!   Not only are extra user options always a good thing, but a TP
> from a region in one world could be to a region in a highly different world
> with very specific requirements.  As you point out, "one size fits all" is
>>> those recovery transitions without which the protocol would simply break as
>>> a result of the exception.  There is of course no need to design VWRAP such

---

From: Morgaine <[email-masked]>
Date: Sun, 25 Oct 2009 03:47:39 +0100

> is no guarantee the invocation of the derez cap has been processed.

By "complete a teleport" I mean "*succeed in placing the agent in the
destination region independent of the state of its avatar*".  I'm making
this distinction because that's precisely the semantic we have right now in
SL and Opensim --- the avatar wearables and attachments arrive much later,
asynchronously.  The *avatar* does *not* obey a semantic of "one place only"
but has parallel existence for a short while in both regions even under
normal operation.  (Under abnormal operation of the source region, that
overlap can last a very long time.)  It's the *active agent* that has unique
existence in only one region at a time, and that's the semantic which I want
to preserve as well.

(It's worth mentioning that the avatar and its equiped objects are left to
appear/disappear at their own sweet rate for a very good pragmatic reason.
Teleport would take an eternity if it were defined as a fully synchronous
operation for all relevant parts of agent and avatar together.)

> - To prevent true duplication of avatars, the again domain keeps some
> record of the old connection, to be able to deal gracefully with the  case
> that the old region comes back to life. This implies that in some cases two
> copies of the avatar might be visible, one "dead body" in the old region,
> and the live version in the new region. This might in some cases be
> preferable to a stranded avatar, yet in other situation could be
> unacceptable (e.g. in a combat game, were mere visual presence might be
> enough to influence the game).
>

Note that this dual presence of avatars is exactly the situation that we
have currently, as I outlined above.  It's important to be clear about the
distinction between agents and avatars in this area.  The key semantic is
that, even if an avatar is still visible in the source region after a
teleport, there be *no active agent* associated with it.  In other words,
any events which the source region may send the agent after the AD has
accepted the TP are discarded.  Under normal operation there should be no
such inappropriate events arriving because the source region will have
processed the Derez, but under abnormal operation anything can happen so one
has to handle it defensively.

As concurrent programming becomes more common in this emerging "age of
multicore", we're going to be seeing these kinds of anomalies arise with
ever greater frequency, particularly as regions become virtualized or
implemented scalably on clusters.

> de-rezzing in the old region.

Yes.  As Lawson highlighted, "one size fits all" is not appropriate.

> with these types of anomalies.

Fortunately it's a very easy anomaly to handle, since the TP simply acquires
the semantic of a first-time rez in the destination region at the point of
source region timeout.

> as the current TP draft specifies.

The AD carries the "first say" responsibility for agent teleport policy,
that's its key role in teleports, and therefore it must always be an
intermediary in teleports otherwise it cannot effect such agent policy.
Region-to-region communication should really be seen as a behind-the-scenes
optimization or an implementation detail for *avatar*  asset/object/state
transfer --- notice clearly it's *avatar*  here, not agent.  And since it
breaks resilient teleport under source region anomalies, it needs to be
optional.

> what we are actually bringing onto our selfs here.

These are pre-draft discussions for VWRAP --- I've purposely avoided
focussing on the original OGP Teleport drafts since many things are changing
for VWRAP anyway as a result of the introduction of RD-controlled services
that do not feature in the original drafts.  We've got to get the VWRAP
teleport semantics well understood in our minds first before churning out a
VWRAP Teleport draft spec for it.  There are still many unknowns in this
area. :-)

Morgaine.

========================================

On Sat, Oct 24, 2009 at 2:59 PM, Vaughn Deluca <[email-masked]>wrote:

> - To prevent true duplication of avatars, the again domain keeps some
> record of the old connection, to be able to deal gracefully with the  case
> that the old region comes back to life. This implies that in some cases two
> copies of the avatar might be visible, one "dead body" in the old region,
> and the live version in the new region. This might in some cases be
> preferable to a stranded avatar, yet in other situation could be
> unacceptable (e.g. in a combat game, were mere visual presence might be
> enough to influence the game).
>
> On Sat, Oct 24, 2009 at 12:08 PM, Morgaine <[email-masked]
> > wrote:
>
>> In this particular case, that could be handled by the AD sending the
>> client an event notification to the effect that the source region did not
>> respond within the timeout period.
>>
>> On Sat, Oct 24, 2009 at 7:21 AM, Lawson English <[email-masked]>wrote:
>>
>>> Morgaine wrote:
>>>
>>>> On Fri, Oct 23, 2009 at 7:09 PM, David W Levine <[email-masked]<mailto:
>>>> [email-masked]>> wrote:
>>>>
>>>>

---

From: Carlo Wood <[email-masked]>
Date: Sun, 25 Oct 2009 11:48:26 +0100

> that includes this concept.

That is hardly a good argument.

What is the *reason* you want this?

The reason users want to TP away from a region is usually to start
over somewhere else as soon as possible. I'm pretty sure that most
users don't like having to wait very long.

Surely there is a good technical solution to this problem, so
lets just solve it.

-- 
Carlo Wood <[email-masked]>

PS "many of us" is a bit vague... personally I have the feeling that
   most users will agree with Morgaine that tp-ing away from a region
   should never be impossible because that region is non-responsive.