Subject: [P2PSIP] Client operation in RELOAD

From: "Narayanan, Vidya" <[email-masked]>
Date: Wed, 1 Jul 2009 12:18:13 -0700

The client operation in RELOAD is currently underspecified and cannot be implemented based on the spec.  Here are the procedures that need to be specified to support client behavior with attachment at the identity owner.  For attachment at any other arbitrary point, a lot more thought needs to be put in to ensure support for mobility and stability of operation.  My suggestion is that it is better left outside the scope of this draft and defined separately.  In this case, RELOAD will only support attachment at the identity owner, which seems very reasonable.  

Message changes necessary to support clients: 
---------------------------------------------
We need to introduce a "Node Type" in the JOIN Request message that indicates that the joining node wishes to do so as a client. Also, Update messages need to have a "case clients" and the "case full" needs to include client information.  

I suggest adding a section on clients - suggested text below: 

Clients: 
--------
Clients are nodes that may provide and receive services in an overlay without participating in overlay routing or incurring the same amount of maintenance overhead as the overlay peers.  Clients are modeled as close to peers as possible in terms of overlay functionality.  In order to allow these nodes to participate in the overlay without inherently being a routing member of the overlay, some changes are required to the join process and the manner in which neighbor relationships are handled for such nodes.  However, all other procedures (e.g., data management using STORE, FETCH, etc.) are handled in exactly the same way as for any other peer in the overlay.  

We denote the point of attachment of the Non-Routing Peer on to the overlay as Overlay Attachment Point (OAP).  The peer responsible for the node ID of the client provides the OAP functionality.  This allows the integration of clients into the architecture with the least deviation from the peer.  Hence, any node that wishes to be a client finds the node that owns its identity space and attaches to it.  By definition, any node serving as a peer in the overlay MUST provide the OAP functionality as well.  The mode of using a different node as the OAP may be allowed as an extension to this protocol - this can be done for availability or load balancing reasons, for instance.  
 
Clients obtain overlay node identities just like peers do.  Hence, the enrollment server may be completely unaware of the distinction between a peer and a client.  By virtue of this, the enrollment procedure is identical for peers and clients.  The join procedure is somewhat simplified for a client and the section below describes that process.  All data management operations to/from a client are indistinguishable from those of a peer.   

The main difference between a client and a peer is that a client only has a single neighbor in the overlay, i.e., the OAP.  The client routes all messages through the OAP - i.e., it has a default route with the OAP as next hop that is used for all messages.  The OAP has the client as a neighbor and has it in its routing table.  None of the nodes in the overlay except the OAP are aware of the presence of the client. The messages to/from a client are natively routed on the overlay.  All messages that are sent to the client arrive at the OAP by regular overlay routing (since it owns the identity space that includes that of the client).  The OAP then looks up its routing table to route the message to the client.  

The OAP may make use of the presence of a client to offload some of its storage responsibilities or to replicate data stored at it.  This may be mutually beneficial in some cases - e.g., the client may be limited in terms of bandwidth and battery, while the OAP may be limited in memory. However, this is out of scope for this document itself and any such storage sharing that occurs between a client and OAP is not visible to the rest of the overlay.  Hence, any such thing must be managed out of band between the OAP and the client.  

Client Bootstrap Process: 
-------------------------
A client obtains a list of bootstrap peers from the enrollment server as any other node in the overlay.  The client triggers the join process the same way as a peer does.  But, the process is a bit more abridged than for a normal peer.  It is described below: 

1.  Client connects to its chosen bootstrap node. The client MUST form a TLS session with the bootstrap node, using the certificate or PSK required for overlay membership.  
2.  The client sends a Join request message to its own Node-ID c, through the BP.  The Join Request message indicates that the joining entity is a client (with Node Type being set to Client).  This is routed to the admitting peer (AP).  The AP sends a response to the Join, after noting that it must serve as the OAP.  
3.  The client also sends an Attach request message to c through the BP.  This may be sent in parallel with the Join message.  The AP sends an Attach response, resulting in connection establishment between the client and AP.  At this point, the AP has the client included in its routing and connection tables and the client has established a default route through the AP.  The join process is now complete.   

The OAP MUST NOT send any Update messages to other peers in the overlay as a result of a client joining the overlay.  The OAP simply updates its routing and connection tables with the client information.  The client does the same.  The OAP may, however, follow up the join process with a series of STORE request messages to transfer some data it stores to the client. 

Impact of Churn on Clients: 
---------------------------- 
As the ownership of the overlay identity space changes, the clients may be affected.  For instance, if the OAP to which a client is attached leaves the overlay or fails, the client needs to find the appropriate OAP to attach to.  Also, when the identity ownership changes due to a new peer attaching between the OAP and the client in the identity space, the client needs to use the new peer as its OAP.   Essentially, the need to change OAP may arise from one of the following three conditions: 

* A new peer joined the overlay with an identity that lies between the identities of a client and its OAP

* The OAP of a client performs a graceful leave from the overlay

* The OAP of a client abruptly leaves the overlay or fails

These three cases are explained below. 

Handling New Peers: 
-------------------

Consider the case where a client (C1) is attached to an OAP (OAP1).  A new peer, P, attaches to the overlay, with an identity that falls between C1's and OAP1's identity.  P is now responsible for the identity that belongs to C1 and hence needs to serve as the OAP to C1 (say, OAP2).  When P completes its Join process, OAP1 detects that it no longer owns the identity corresponding to C1.  OAP1 now sends an Update message to C1, indicating that it has a new neighbor.  The Update message provides information about P to C1.  Upon receiving the Update message, C1 sends a Join Request via OAP1 to P.  The difference between this JOIN process and the regular one is that C1 can perform this Join via OAP1 instead of finding a bootstrap peer to perform the join through.  Since P is part of the overlay and is OAP1's neighbor, OAP1 can reach P directly, thereby greatly reducing the number of hops the Join messages need to traverse.  Once the new Join process completes, P takes over as C1's OAP (OAP2).  C1 may send a Leave Request to OAP1 subsequent to its attachment to OAP2 - this will trigger cleaning up of state corresponding to C1 at OAP1.  

If OAP1 stored any of its data at C1, C1 must transfer that back to OAP1 before joining again with OAP2.  The mechanism to handle this is out of scope of this specification.  

When a change in the identity space ownership causes the OAP of a particular node to change, the present OAP MUST send an Update message to the client, indicating that it has a new neighbor.  Upon receiving the update message, the client MUST send a Join message to the new OAP - this JOIN message can be routed through its current OAP.  The client SHOULD send a Leave request to its old OAP, once it has successfully attached to the new OAP.    

Graceful Leave of an OAP:
-------------------------
When an OAP gracefully leaves an overlay, it typically sends a LEAVE request message to all its neighbors.  In addition, the OAP SHOULD send a Leave Request message to the client.  Further, the OAP SHOULD also send an Update message to the client, containing information about its successor, which will be the new OAP for the client.  When the client receives such an Update Request, it MUST send a Join Request message to its new OAP.  If the old OAP is able to assist in the Join process, the Join messages can traverse through it to set up the connection between the client and its new OAP.  However, if the old OAP leaves the overlay prior to that process, the client needs to find a bootstrap peer and route its Join Request message through that peer to its new OAP, in which case, the procedure becomes similar to the failure of an OAP case described below.   

Failure of an OAP: 
------------------
An abrupt leave or a failure of an OAP implies that there is no active messaging from the OAP to its clients indicating that an OAP change is required.  In this case, the client will detect the absence of its OAP due to a timeout indicating that the OAP is unreachable.  The client then needs to perform the Join process via a bootstrap peer in the overlay to find its new OAP.  
 
Impact on attach messages: 
---------------------------
There is an issue that nodes may send attach messages to the client's node id when looking for a finger.  The client needs to reject such Attach requests coming to it.  However, the client will be unable to distinguish Attach messages that are being sent to it to really set up an end-to-end connection for other purposes.  We may want to consider introducing a "purpose" in the Attach request messages that indicate the reason for the Attach.  This will make it possible for the client to reject a request that is sent to establish a finger.  The node sending the Attach request can then slightly alter the resource id to issue another request.  

Thanks,
Vidya

---

From: "Xiao, Lin (NSN - CN/Beijing)" <[email-masked]>
Date: Thu, 2 Jul 2009 11:42:37 +0800

Hi vidyan,

I also agree with you to give a deeper discussion and specification on
RELOAD clients. IMO, all client related specification could be taken out
side of BASE draft. RELOAD BASE just need give a general rule for
overlay construction and common behavior of both peers and clients. 

In your email below, you defined clients as nodes without routing
responsibility. However, in RELOAD Base, clients was defined as nodes
which have Node-ID but do not participate in routing or storage. That
means, They could be nodes with routing but without storage. Three kinds
of clients has been analysed in draft
[draft-xiao-p2psip-client-routing-00.txt],
http://www.ietf.org/internet-drafts/draft-xiao-p2psip-client-routing-00.
txt. This small difference may need to be negotiate with RELOAD Base. No
matter what the result will be, at least, we should discuss something
basing on a same terminology, right?

You recommend only a single neighbour for each client. IMO, a client can
maintain more than one neighbours, but only keeps one entry in its
Routing Table at a time. It not only ensures all messages from/to the
client are through the OAP, but also make sure the client could be
quickly hand over to a Direct Attachment Point (DAP), which can be
selected directly from the neighbours, when necessary.  It offers a
seamless transfer from client to eclient.

By the way. in my draft [draft-xiao-p2psip-client-routing-00.txt], the
Routing Client and Pure Client defined there are in the scope of your
client discussion with similar concern. We do need a unified protocol
for Reload clients.

Best Regards
Lin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of ext Narayanan, Vidya
Sent: Thursday, July 02, 2009 3:18 AM
To: [email-masked]
Subject: [P2PSIP] Client operation in RELOAD

The client operation in RELOAD is currently underspecified and cannot be
implemented based on the spec.  Here are the procedures that need to be
specified to support client behavior with attachment at the identity
owner.  For attachment at any other arbitrary point, a lot more thought
needs to be put in to ensure support for mobility and stability of
operation.  My suggestion is that it is better left outside the scope of
this draft and defined separately.  In this case, RELOAD will only
support attachment at the identity owner, which seems very reasonable.  

Message changes necessary to support clients: 
---------------------------------------------
We need to introduce a "Node Type" in the JOIN Request message that
indicates that the joining node wishes to do so as a client. Also,
Update messages need to have a "case clients" and the "case full" needs
to include client information.  

Clients: 
--------
Clients are nodes that may provide and receive services in an overlay
without participating in overlay routing or incurring the same amount of
maintenance overhead as the overlay peers.  Clients are modeled as close
to peers as possible in terms of overlay functionality.  In order to
allow these nodes to participate in the overlay without inherently being
a routing member of the overlay, some changes are required to the join
process and the manner in which neighbor relationships are handled for
such nodes.  However, all other procedures (e.g., data management using
STORE, FETCH, etc.) are handled in exactly the same way as for any other
peer in the overlay.  

We denote the point of attachment of the Non-Routing Peer on to the
overlay as Overlay Attachment Point (OAP).  The peer responsible for the
node ID of the client provides the OAP functionality.  This allows the
integration of clients into the architecture with the least deviation
from the peer.  Hence, any node that wishes to be a client finds the
node that owns its identity space and attaches to it.  By definition,
any node serving as a peer in the overlay MUST provide the OAP
functionality as well.  The mode of using a different node as the OAP
may be allowed as an extension to this protocol - this can be done for
availability or load balancing reasons, for instance.  
 
Clients obtain overlay node identities just like peers do.  Hence, the
enrollment server may be completely unaware of the distinction between a
peer and a client.  By virtue of this, the enrollment procedure is
identical for peers and clients.  The join procedure is somewhat
simplified for a client and the section below describes that process.
All data management operations to/from a client are indistinguishable
from those of a peer.   

The main difference between a client and a peer is that a client only
has a single neighbor in the overlay, i.e., the OAP.  The client routes
all messages through the OAP - i.e., it has a default route with the OAP
as next hop that is used for all messages.  The OAP has the client as a
neighbor and has it in its routing table.  None of the nodes in the
overlay except the OAP are aware of the presence of the client. The
messages to/from a client are natively routed on the overlay.  All
messages that are sent to the client arrive at the OAP by regular
overlay routing (since it owns the identity space that includes that of
the client).  The OAP then looks up its routing table to route the
message to the client.  

The OAP may make use of the presence of a client to offload some of its
storage responsibilities or to replicate data stored at it.  This may be
mutually beneficial in some cases - e.g., the client may be limited in
terms of bandwidth and battery, while the OAP may be limited in memory.
However, this is out of scope for this document itself and any such
storage sharing that occurs between a client and OAP is not visible to
the rest of the overlay.  Hence, any such thing must be managed out of
band between the OAP and the client.  

Client Bootstrap Process: 
-------------------------
A client obtains a list of bootstrap peers from the enrollment server as
any other node in the overlay.  The client triggers the join process the
same way as a peer does.  But, the process is a bit more abridged than
for a normal peer.  It is described below: 

1.  Client connects to its chosen bootstrap node. The client MUST form a
TLS session with the bootstrap node, using the certificate or PSK
required for overlay membership.  
2.  The client sends a Join request message to its own Node-ID c,
through the BP.  The Join Request message indicates that the joining
entity is a client (with Node Type being set to Client).  This is routed
to the admitting peer (AP).  The AP sends a response to the Join, after
noting that it must serve as the OAP.  
3.  The client also sends an Attach request message to c through the BP.
This may be sent in parallel with the Join message.  The AP sends an
Attach response, resulting in connection establishment between the
client and AP.  At this point, the AP has the client included in its
routing and connection tables and the client has established a default
route through the AP.  The join process is now complete.   

The OAP MUST NOT send any Update messages to other peers in the overlay
as a result of a client joining the overlay.  The OAP simply updates its
routing and connection tables with the client information.  The client
does the same.  The OAP may, however, follow up the join process with a
series of STORE request messages to transfer some data it stores to the
client. 

Impact of Churn on Clients: 
---------------------------- 
As the ownership of the overlay identity space changes, the clients may
be affected.  For instance, if the OAP to which a client is attached
leaves the overlay or fails, the client needs to find the appropriate
OAP to attach to.  Also, when the identity ownership changes due to a
new peer attaching between the OAP and the client in the identity space,
the client needs to use the new peer as its OAP.   Essentially, the need
to change OAP may arise from one of the following three conditions: 

* A new peer joined the overlay with an identity that lies between the
identities of a client and its OAP

Handling New Peers: 
-------------------

Consider the case where a client (C1) is attached to an OAP (OAP1).  A
new peer, P, attaches to the overlay, with an identity that falls
between C1's and OAP1's identity.  P is now responsible for the identity
that belongs to C1 and hence needs to serve as the OAP to C1 (say,
OAP2).  When P completes its Join process, OAP1 detects that it no
longer owns the identity corresponding to C1.  OAP1 now sends an Update
message to C1, indicating that it has a new neighbor.  The Update
message provides information about P to C1.  Upon receiving the Update
message, C1 sends a Join Request via OAP1 to P.  The difference between
this JOIN process and the regular one is that C1 can perform this Join
via OAP1 instead of finding a bootstrap peer to perform the join
through.  Since P is part of the overlay and is OAP1's neighbor, OAP1
can reach P directly, thereby greatly reducing the number of hops the
Join messages need to traverse.  Once the new Join process completes, P
takes over as C1'
 s OAP (OAP2).  C1 may send a Leave Request to OAP1 subsequent to its
attachment to OAP2 - this will trigger cleaning up of state
corresponding to C1 at OAP1.  

If OAP1 stored any of its data at C1, C1 must transfer that back to OAP1
before joining again with OAP2.  The mechanism to handle this is out of
scope of this specification.  

When a change in the identity space ownership causes the OAP of a
particular node to change, the present OAP MUST send an Update message
to the client, indicating that it has a new neighbor.  Upon receiving
the update message, the client MUST send a Join message to the new OAP -
this JOIN message can be routed through its current OAP.  The client
SHOULD send a Leave request to its old OAP, once it has successfully
attached to the new OAP.    

Graceful Leave of an OAP:
-------------------------
When an OAP gracefully leaves an overlay, it typically sends a LEAVE
request message to all its neighbors.  In addition, the OAP SHOULD send
a Leave Request message to the client.  Further, the OAP SHOULD also
send an Update message to the client, containing information about its
successor, which will be the new OAP for the client.  When the client
receives such an Update Request, it MUST send a Join Request message to
its new OAP.  If the old OAP is able to assist in the Join process, the
Join messages can traverse through it to set up the connection between
the client and its new OAP.  However, if the old OAP leaves the overlay
prior to that process, the client needs to find a bootstrap peer and
route its Join Request message through that peer to its new OAP, in
which case, the procedure becomes similar to the failure of an OAP case
described below.   

Failure of an OAP: 
------------------
An abrupt leave or a failure of an OAP implies that there is no active
messaging from the OAP to its clients indicating that an OAP change is
required.  In this case, the client will detect the absence of its OAP
due to a timeout indicating that the OAP is unreachable.  The client
then needs to perform the Join process via a bootstrap peer in the
overlay to find its new OAP.  
 
Impact on attach messages: 
---------------------------
There is an issue that nodes may send attach messages to the client's
node id when looking for a finger.  The client needs to reject such
Attach requests coming to it.  However, the client will be unable to
distinguish Attach messages that are being sent to it to really set up
an end-to-end connection for other purposes.  We may want to consider
introducing a "purpose" in the Attach request messages that indicate the
reason for the Attach.  This will make it possible for the client to
reject a request that is sent to establish a finger.  The node sending
the Attach request can then slightly alter the resource id to issue
another request.  

Thanks,
Vidya
_______________________________________________
P2PSIP mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2psip

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 08 Jul 2009 10:43:19 -0700

At Wed, 1 Jul 2009 12:18:13 -0700,
Narayanan, Vidya wrote:
> The client operation in RELOAD is currently underspecified and
> cannot be implemented based on the spec. 

I'm certainly prepared to believe that's correct, but I'm not really
getting that from your message. From my perspective, the major
feature of the RELOAD client design is that there *isn't* any
special support for clients, but that it just drops out of
the rest of the specified protocol.

> Update messages need to have a "case clients" and the "case full"
> needs to include client information.

In the current design, clients simply don't send Join or Update, so this
isn't an issue. Because they don't really participate in either the
storage or routing fabric, as far as I can tell they don't need
to do either.

> Clients are nodes that may provide and receive services in an
> overlay without participating in overlay routing or incurring the
> same amount of maintenance overhead as the overlay peers.  Clients
> are modeled as close to peers as possible in terms of overlay
> functionality.  In order to allow these nodes to participate in the
> overlay without inherently being a routing member of the overlay,
> some changes are required to the join process and the manner in
> which neighbor relationships are handled for such nodes.  However,
> all other procedures (e.g., data management using STORE, FETCH,
> etc.) are handled in exactly the same way as for any other peer in
> the overlay.
> 
> overlay as Overlay Attachment Point (OAP).  The peer responsible for
> the node ID of the client provides the OAP functionality.  This
> allows the integration of clients into the architecture with the
> least deviation from the peer.  Hence, any node that wishes to be a
> client finds the node that owns its identity space and attaches to
> it.  By definition, any node serving as a peer in the overlay MUST
> provide the OAP functionality as well.  The mode of using a
> different node as the OAP may be allowed as an extension to this
> protocol - this can be done for availability or load balancing
> reasons, for instance.
>  
> Clients obtain overlay node identities just like peers do.  Hence,
> the enrollment server may be completely unaware of the distinction
> between a peer and a client.  By virtue of this, the enrollment
> procedure is identical for peers and clients.  The join procedure is
> somewhat simplified for a client and the section below describes
> that process.  All data management operations to/from a client are
> indistinguishable from those of a peer.
> 
> The main difference between a client and a peer is that a client
> only has a single neighbor in the overlay, i.e., the OAP.  The
> client routes all messages through the OAP - i.e., it has a default
> route with the OAP as next hop that is used for all messages.  The
> OAP has the client as a neighbor and has it in its routing table.
> None of the nodes in the overlay except the OAP are aware of the
> presence of the client. The messages to/from a client are natively
> routed on the overlay.  All messages that are sent to the client
> arrive at the OAP by regular overlay routing (since it owns the
> identity space that includes that of the client).  The OAP then
> looks up its routing table to route the message to the client.
>
> The OAP may make use of the presence of a client to offload some of
> its storage responsibilities or to replicate data stored at it.
> This may be mutually beneficial in some cases - e.g., the client may
> be limited in terms of bandwidth and battery, while the OAP may be
> limited in memory. However, this is out of scope for this document
> itself and any such storage sharing that occurs between a client and
> OAP is not visible to the rest of the overlay.  Hence, any such
> thing must be managed out of band between the OAP and the client.

I agree with almost all of this text, but it's not normative
and I'm not sure really clarifies things that much. As far as
clients storing data for OAPs goes, IMO that should be out
of scope of the protocol. If clients and OAPs want to design
some new protocol that handles that great, but it's not RELOADs
business.

> A client obtains a list of bootstrap peers from the enrollment
> server as any other node in the overlay.  The client triggers the
> join process the same way as a peer does.  But, the process is a bit
> more abridged than for a normal peer.  It is described below:
> 
> 1.  Client connects to its chosen bootstrap node. The client MUST
> form a TLS session with the bootstrap node, using the certificate or
> PSK required for overlay membership.  2.  The client sends a Join
> request message to its own Node-ID c, through the BP.  The Join
> Request message indicates that the joining entity is a client (with
> Node Type being set to Client).  This is routed to the admitting
> peer (AP).  The AP sends a response to the Join, after noting that
> it must serve as the OAP.  3.  The client also sends an Attach
> request message to c through the BP.  This may be sent in parallel
> with the Join message.  The AP sends an Attach response, resulting
> in connection establishment between the client and AP.  At this
> point, the AP has the client included in its routing and connection
> tables and the client has established a default route through the
> AP.  The join process is now complete.
> 
> The OAP MUST NOT send any Update messages to other peers in the
> overlay as a result of a client joining the overlay.  The OAP simply
> updates its routing and connection tables with the client
> information.  The client does the same.  The OAP may, however,
> follow up the join process with a series of STORE request messages
> to transfer some data it stores to the client.

I disaree with this: the client should not be sending a Join because
that indicates the desire to be a peer. There is no need for the
join as mere attachment enables direct routing to the client
from the OAP.

> As the ownership of the overlay identity space changes, the clients
> may be affected.  For instance, if the OAP to which a client is
> attached leaves the overlay or fails, the client needs to find the
> appropriate OAP to attach to.  Also, when the identity ownership
> changes due to a new peer attaching between the OAP and the client
> in the identity space, the client needs to use the new peer as its
> OAP.  Essentially, the need to change OAP may arise from one of the
> following three conditions:
> 
> * A new peer joined the overlay with an identity that lies between
> * the identities of a client and its OAP
> 
> Consider the case where a client (C1) is attached to an OAP (OAP1).
>  A new peer, P, attaches to the overlay, with an identity that falls
>  between C1's and OAP1's identity.  P is now responsible for the
>  identity that belongs to C1 and hence needs to serve as the OAP to
>  C1 (say, OAP2).  When P completes its Join process, OAP1 detects
>  that it no longer owns the identity corresponding to C1.  OAP1 now
>  sends an Update message to C1, indicating that it has a new
>  neighbor.  The Update message provides information about P to C1.
>  Upon receiving the Update message, C1 sends a Join Request via OAP1
>  to P.  The difference between this JOIN process and the regular one
>  is that C1 can perform this Join via OAP1 instead of finding a
>  bootstrap peer to perform the join through.  Since P is part of the
>  overlay and is OAP1's neighbor, OAP1 can reach P directly, thereby
>  greatly reducing the number of hops the Join messages need to
>  traverse.  Once the new Join process completes, P takes over as C1'
>  s OAP (OAP2).  C1 may send a Leave Request to OAP1 subsequent to
>  its attachment to OAP2 - this will trigger cleaning up of state
> If OAP1 stored any of its data at C1, C1 must transfer that back to
> OAP1 before joining again with OAP2.  The mechanism to handle this
> is out of scope of this specification.
> 
> particular node to change, the present OAP MUST send an Update
> message to the client, indicating that it has a new neighbor.  Upon
> receiving the update message, the client MUST send a Join message to
> the new OAP - this JOIN message can be routed through its current
> OAP.  The client SHOULD send a Leave request to its old OAP, once it
> has successfully attached to the new OAP.

In general, whenever a peer's view of its neighbor changes it
should Update all its neighbors. As "client" isn't a first-class
term in RELOAD, this implies clients as well. I agree we should
explicitly state that clients need to disconnect and reconnect
to the new responsible peer. As indicated above, I believe the
discussion of Join is incorrect.

> There is an issue that nodes may send attach messages to the
> client's node id when looking for a finger.  The client needs to
> reject such Attach requests coming to it.  However, the client will
> be unable to distinguish Attach messages that are being sent to it
> to really set up an end-to-end connection for other purposes.  We
> may want to consider introducing a "purpose" in the Attach request
> messages that indicate the reason for the Attach.  This will make it
> possible for the client to reject a request that is sent to
> establish a finger.  The node sending the Attach request can then
> slightly alter the resource id to issue another request.

There is supposed to be an AppAttach message that makes this
clear. It just somehow didn't make it into the draft. It will
be in the next revision.

-Ekr

---

From: "Narayanan, Vidya" <[email-masked]>
Date: Thu, 9 Jul 2009 14:03:02 -0700

Hi Lin,

> From: Xiao, Lin (NSN - CN/Beijing) [mailto:[email-masked]]
> Sent: Wednesday, July 01, 2009 8:43 PM
> To: Narayanan, Vidya; [email-masked]
> Subject: RE: [P2PSIP] Client operation in RELOAD
> 
> Hi vidyan,
> 

This may be okay, although supporting client attachment at the identity owner seems fundamental and hence, I suggested that be part of the base draft.  

> responsibility. 

It should also say no storage responsibility.  Any storage that a client may offer is outside the scope of RELOAD itself and should be managed out of band.  So, from the overlay's perspective, the client has no storage responsibility. 

> However, in RELOAD Base, clients was defined as nodes

Using the same terminology would be good.  I do, however, think that we should not get into storage on clients or routing without storage, etc.  It will fundamentally complicate the DHT and is best kept simple with clients having no routing or storage from the overlay's perspective.   

> seamless transfer from client to eclient.

Sure, I didn't mean to exclude this - more often than not, however, having more connections may not be a necessary burden on the client.  But, a client may optionally choose to do this as you say.  

> client discussion with similar concern. We do need a unified protocol

We do need to define how clients work in detail.  Especially for attachment at arbitrary points, I think it is best if mobility is handled natively without needing the client to re-store all data with a new destination list upon movement.  This will be similar to the annoyance of embedding IP addresses in applications and all the issues that has caused over the years and I'd really like to avoid that here. 

Regards,
Vidya

> the
> identity
> OAP1

---

From: "Narayanan, Vidya" <[email-masked]>
Date: Thu, 9 Jul 2009 14:03:08 -0700

I'm not sure I understand what you are saying.  Are you saying that an Attach request without a preceding Join is indicative of joining as a client?  If so, a client finds a bootstrap peer and sends an Attach request to its own node id, in order to find the node that it is supposed to attach to?  AFAICT, this doesn't seem to be described in the draft.  Even if it is, I don't understand how a node differentiates a client from another neighbor, because treating the client as a neighbor and sending updates to all other neighbors will create an incorrect view of the DHT for the other nodes.  If you are saying that all of this is implicit in the fact that an Attach was received without a Join, it seems setting up for issues.  I think it is better off to explicitly join as a client to avoid any ambiguity. 

It is possible that I misunderstood what you are saying, in which case, please explain a bit further. 

Thanks,
Vidya 

> From: Eric Rescorla [mailto:[email-masked]]
> Sent: Wednesday, July 08, 2009 10:43 AM
> To: Narayanan, Vidya
> Cc: [email-masked]
> Subject: Re: [P2PSIP] Client operation in RELOAD
>

---

From: Julian Cain <[email-masked]>
Date: Thu, 9 Jul 2009 21:14:30 -0700

On Jul 9, 2009, at 2:03 PM, Narayanan, Vidya wrote:

> I'm not sure I understand what you are saying.  Are you saying that  
> an Attach request without a preceding Join is indicative of joining  
> as a client?  If so, a client finds a bootstrap peer and sends an  
> Attach request to its own node id, in order to find the node that it  
> is supposed to attach to?  AFAICT, this doesn't seem to be described  
> in the draft.  Even if it is, I don't understand how a node  
> differentiates a client from another neighbor, because treating the  
> client as a neighbor and sending updates to all other neighbors will  
> create an incorrect view of the DHT for the other nodes.

None of  this is defined in the draft, only indications of "Client  
Operation" thus the controversy of Client Join vs. Peer Join will  
continue to be debated for now.

> If you are saying that all of this is implicit in the fact that an  
> Attach was received without a Join, it seems setting up for issues.   
> I think it is better off to explicitly join as a client to avoid any  
> ambiguity.

IMHO: You SHOULD perform a Join as a Client and then if need be Join  
as a Peer. You'd have to first Join and Lookup where you belong in the  
ring and then re-join at that node-id to avoid ring imbalance.

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 10 Jul 2009 13:23:07 -0700

At Thu, 9 Jul 2009 14:03:08 -0700,
Narayanan, Vidya wrote:
> as a client?  

No, I'm saying that "client" isn't a first class concept in RELOAD.
The generic procedures defined in RELOAD make peers handle client
nodes correctly automatically. So, all that's required is that
nodes which wish not to be part of the routing fabric and
storage fabric not send the messages that would make them
so.

> If so, a client finds a bootstrap peer and sends an
> is supposed to attach to?

Yes.

>  AFAICT, this doesn't seem to be described
> in the draft.

That's because there is no new normative behavior. Clients simply 
don't send Join or Update.

>  Even if it is, I don't understand how a node
> create an incorrect view of the DHT for the other nodes. If you are
> saying that all of this is implicit in the fact that an Attach was
> received without a Join, it seems setting up for issues.  I think it
> is better off to explicitly join as a client to avoid any ambiguity.

A node doesn't know that an adjacency is a client or not. What it
knows is that it's a node that hasn't sent any Updates and therefore
shouldn't be advertised to other nodes in the overlay. It could
be a client or it could be a node which just hasn't sent any
Updates or Joins yet (but will eventually). It doesn't matter. This is
a feature of the design, not a bug.

-Ekr

---

From: Ted Hardie <[email-masked]>
Date: Fri, 10 Jul 2009 15:04:01 -0700

>storage fabric not send the messages that would make them

So, my personal preference is for "client" to be closer to a first
class concept, so that this behavior is explicit.  That might take
the form of a different message after attach.  That's largely a
style, thing, though, and I agree that the behavior you describe
works.  But see below.

>a feature of the design, not a bug.

My concern here is that without an explicit step, some set of
implementations may treat long post-Attach periods without
joins or updates as indicative that the relationship to the
attached node is not healthy.  That might take the form of timeouts
or something similar. 

You can retain the current theory and avoid that ambiguity by
making the theory you've articulated in this email explicit in the
text.  That would require some additional verbiage, but it should
not require anyone to actually change code.  For the folks who
implement post-WG discussion, that kind of breadcrumb will be
pretty important.

My two cents,
				Ted

---

From: Bruce Lowekamp <[email-masked]>
Date: Fri, 10 Jul 2009 18:05:23 -0400

> a feature of the design, not a bug.

I agree that a feature of the design is that Attach doesn't require a
specification of what it's being used for---which allows an
implementation to Attach to the responsible peer as an optimization
for resource refreshes, for example, without a change in the spec to
allow it.

The same feature can also be used by clients to support their operation.

But I think it's also true that there is relatively little support
should any of these Attaches need to migrate to another peer,
regardless of whether the Attaching node is a client or peer.  I think
that making sure we have this feature (or ensuring that it works
appropriately via sending Updates to all Attached nodes) would help
both client and peer operation.  That seems like it's useful in the
base spec.

More details of how clients might work I think should be in another
draft.  For example, client might achieve reliable connections via
maintaining a link with their responsible peer and its two successors,
then conduct appropriate maintenance on these in the event of topology
changes.  I would like for the base draft to allow the features to
implement this, but would rather not make the base draft any more
complicated by adding details of how to manage such support.  I could
see an argument for different ways of handling clients using the same
basic mechanisms.

Bruce

---

From: Bruce Lowekamp <[email-masked]>
Date: Fri, 10 Jul 2009 19:10:20 -0400

ekr already mentioned that AppAttach is supposed to come into the next draft.

What if we have a 16 bit field in Attach and AttachLite that specifies
role/purpose of the connection?  No normative purpose in the base
draft, but for use with future extensions.  Specify maybe five values
for neighbor, routing table, peer optimization, client responsible
peer attachment, and client optimization, and allow for future code
points.

That would allow future extensions that want to specify/control client
behavior more carefully to do so, but not make the base any more
complicated or restrict its behavior.

Possibly rather than a simple 16 bit field, this should be more like a
Forwarding Option so that if additional data is required, it can be
included.

Bruce

---

From: Ted Hardie <[email-masked]>
Date: Fri, 10 Jul 2009 16:25:28 -0700

>peer attachment, and client optimization, and allow for future code

In general, this sounds like a useful way forward. 

>Forwarding Option so that if additional data is required, it can be
 
I don't have a strong opinion on whether to go with a fix-length
field or a Forwarding Option, and I think the general idea could
work either way.
			regards,
				Ted

---

From: Julian Cain <[email-masked]>
Date: Fri, 10 Jul 2009 19:51:53 -0700

Why would a node send the "purpose of the connection" in it's Attach?  
Attach is low level and hardly indicative of anything Overlay related.  
The Attach clearly states the purpose of the connection outright, what  
we do not is the reason for the Join and what role the joining node  
wishes to assume.

jc

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 10 Jul 2009 20:01:42 -0700

At Fri, 10 Jul 2009 19:51:53 -0700,
> we do not is the reason for the Join and what role the joining node  

What do you mean? Only nodes which are going to store data send
joins. That's what Join means.

-Ekr

---

From: Julian Cain <[email-masked]>
Date: Fri, 10 Jul 2009 21:01:14 -0700

> joins. That's what Join means.

Technically Join doesn't mean or indicate nodes that "are going to  
store data", instead Join indicates a node that WANT's to store and  
route. If the join_req fails the Client is still a Client otherwise it  
sends a series of Store's along with Update's and only then is the  
node considered a Peer inside of the ring vs a Client outside of the  
ring.

I assume this AppAttach performs an Attach and then the AppAttach  
instead of a Join?

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 10 Jul 2009 21:26:44 -0700

At Fri, 10 Jul 2009 21:01:14 -0700,
Julian Cain wrote:
> route.

Sure, but "store", not "route". If you don't send updates, you
won't be routing.

> If the join_req fails the Client is still a Client otherwise it  

Actually, the node (formerly Client) does not send Stores. Rather,
the peer that it's taking over space from sends Stores. The
node can of course send its own Stores to store data at any
time.

> I assume this AppAttach performs an Attach and then the AppAttach  

AppAttach (when it exists) doesn't have anything to do with clients.
It's what you use to initiate connections for traffic other
than RELOAD. RELOAD technically supports this already with hte
application field, but our experience with implementatons was
that people found using the same method for both purposes was
really confusing.

-Ekr

---

From: jc <[email-masked]>
Date: Fri, 10 Jul 2009 23:12:22 -0700

> Sure, but "store", not "route". If you don't send updates, you

That's incorrect, Update is required to Join thus you MUST always  
perform them given the default Chord algorithm. The only time you  
needn't send Updates is Client vs Peer operation mode which is vaguely  
outlined in the current draft.

jc

---

From: "Narayanan, Vidya" <[email-masked]>
Date: Sat, 11 Jul 2009 23:10:07 -0700

I'd support this as well. I think this would be quite useful. 

Vidya

> From: Hardie, Ted
> Sent: Friday, July 10, 2009 4:25 PM
> To: Bruce Lowekamp
> Cc: Eric Rescorla; Narayanan, Vidya; [email-masked]

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 12 Jul 2009 11:33:20 -0700

At Sat, 11 Jul 2009 23:10:07 -0700,
> > In general, this sounds like a useful way forward.

I don't see that this change adds any value. It's just additional cruft.

-Ekr

---

From: "David A. Bryan" <[email-masked]>
Date: Sun, 12 Jul 2009 19:51:52 +0000

At least as the discussion has gone so far, I actually like the idea of this myself, particularly using an option structure of some sort.

There is already a bit of cruft here and there in the protocol (my personal take is that it is a bit too heavy in current form), but this actually looks useful to me, so for now, I'm behind the idea of adding it.

David (as individual)
Sent from my mobile device

-----Original Message-----
From: Eric Rescorla <[email-masked]>

Date: Sun, 12 Jul 2009 11:33:20 
To: Narayanan, Vidya<[email-masked]>
Cc: [email-masked]<[email-masked]>
Subject: Re: [P2PSIP] Client operation in RELOAD

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 12 Jul 2009 13:38:37 -0700

At Sun, 12 Jul 2009 19:51:52 +0000,
David A. Bryan wrote:
> There is already a bit of cruft here and there in the protocol (my
> personal take is that it is a bit too heavy in current form), but
> this actually looks useful to me, so for now, I'm behind the idea of
> adding it.

Not to put too fine a point on it, but this is really a not very
useful complaint. If you think the protocol has too much cruft,
how about identifying some bloat and explaining how to remove
it?

-Ekr

---

From: "David A. Bryan" <[email-masked]>
Date: Sun, 12 Jul 2009 20:54:14 +0000

Actually this is more of a general comment that it is big enough that using it as an excuse to not include something isn't really a valid reason. (Most of the things I personally view as cruft (mandatory TLS, for example) were, unfortunately, WG decisions, so no point in my complaining again)

My point is more that you saying "you don't like cruft" is not a very well articulated reason to not include a 16 bit field in an already 141 page document. I think we have included enough that, this looks to me as useful as many other things in the draft, and I personally support it.

David (as individual)

-----Original Message-----

Date: Sun, 12 Jul 2009 13:38:37 
To: <[email-masked]>
Cc: Eric Rescorla<[email-masked]>; <[email-masked]>; Narayanan, Vidya<[email-masked]>; [email-masked]<[email-masked]>
Subject: Re: [P2PSIP] Client operation in RELOAD

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 12 Jul 2009 14:08:09 -0700

At Sun, 12 Jul 2009 20:54:14 +0000,
David A. Bryan wrote:
> 
> Actually this is more of a general comment that it is big enough
> that using it as an excuse to not include something isn't really a
> valid reason.

So you'll totally support it when I suggest we incorporate a feature
to carry around BGP route tables? After all, we might need that
someday and if the protocol is bloated already, no harm no 
foul, right? 

> My point is more that you saying "you don't like cruft" is not a
> very well articulated reason to not include a 16 bit field in an
> already 141 page document. I think we have included enough that,
> this looks to me as useful as many other things in the draft, and I
> personally support it.

Of course, you're just totally mischaracterizing what I said, which
was:

   I don't see that this change adds any value. It's just additional
   cruft.

To be clear, my objection isn't just that I don't like cruft, but
rather that this feature is useless. Actually, now that I look at it,
it's worse than useless, it's actively harmful; RELOAD doesn't include
clients as a first-class concept for a good reason, it's not necessary
and it interferes with the orthogonality of the protocol, which is
almost never a good architectural decision.

It's particularly bad in this case because we're adding an ad hoc
indicator that receivers don't process but that categorizes nodes in
an ill-defined way. In fact, of the five possible settings Bruce
listed, I don't know what two of them ("peer optimization" and "client
optimization") mean, and one distinction (neighbor versus routing
table) only makes sense with specific overlays, and so properly
belongs at the Topology Plugin Layer.

-Ekr

---

From: Bruce Lowekamp <[email-masked]>
Date: Sun, 12 Jul 2009 17:15:48 -0400

> belongs at the Topology Plugin Layer.

The terms were explained in the email I sent out prior to that one.
Were we to add such a field, I think the exact values that it should
have would be a matter that required some thought.  (I at first
considered adding an overlay-specific bit, but since clients don't
need to implement the full overlay algorithm decided it would be
better to focus on more general concepts.)

Regardless, after further thought I believe that something more like
ForwardingOption with comprehension required/optional would be far
more useful.

Bruce

---

From: Song Haibin <[email-masked]>
Date: Mon, 13 Jul 2009 14:45:53 +0800

Bruce,

I followed this thread and generally I like this option. It would be
explicitly useful to identify client and peer operations. I'm a little
confused why we should depend on the Update message to determine whether an
attached node is a client or a peer.

Thanks!
Haibin

>From: [email-masked] [mailto:[email-masked]] 
>On Behalf Of Bruce Lowekamp
>Sent: Monday, July 13, 2009 5:16 AM
>To: Eric Rescorla
>Cc: [email-masked]; [email-masked]
>On Sun, Jul 12, 2009 at 5:08 PM, Eric 
>Rescorla<[email-masked]> wrote:
>>> Actually this is more of a general comment that it is big 
>enough that 
>>> using it as an excuse to not include something isn't really a valid 
>>> reason.
>>
>> someday and if the protocol is bloated already, no harm no foul, 
>> right?
>>
>>
>>> My point is more that you saying "you don't like cruft" is 
>not a very 
>>> well articulated reason to not include a 16 bit field in an already 
>>> 141 page document. I think we have included enough that, this looks 
>>> to me as useful as many other things in the draft, and I personally 
>>> support it.
>>
>> ? I don't see that this change adds any value. It's just 
>additional ? 
>> cruft.
>>
>Were we to add such a field, I think the exact values that it 
>should have would be a matter that required some thought.  (I 
>at first considered adding an overlay-specific bit, but since 
>clients don't need to implement the full overlay algorithm 
>decided it would be better to focus on more general concepts.)
>
>Regardless, after further thought I believe that something 
>more like ForwardingOption with comprehension 
>required/optional would be far more useful.
>
>Bruce

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 13 Jul 2009 07:37:14 -0700

> attached node is a client or a peer.

This is a category error: we don't depend on the Update message to determine
whether an attached node is a client or peer because client isn't
an important concept in RELOAD. Rather, the rules cause peers to treat
adjacencies
correctly whether they are peers, clients, or peers who haven't
completed the join
process yet. There is no need for a peer to detect or remember whether an
adjacency is a client or peer.

-Ekr

---

From: Ted Hardie <[email-masked]>
Date: Mon, 13 Jul 2009 11:47:26 -0700

>adjacency is a client or peer.

Speaking personally, I think there is a fair chance that a peer might want
to treat an adjacent node differently if it is a client and if it is a peer that intends
to but hasn't completed the join process yet.   I understand the
"a stranger is a friend you haven't met" theory behind  the current approach
which makes them equivalent.  But that understanding is based largely on
the explanation given on the list.   I think that some  small promotion in
the concept of client from  our current stance would be useful. 
We already know that people will do this,  and having explicit text on how
strikes me as a good idea. 

As I said before I'd be okay with just text in the document that
expounded the current theory.  That would help later readers better understand
why long-term adjacencies without complete join processes shouldn't be
treated as errors.  My personal style preference would be for a more
explicit set of distinguishing characteristics, but I recognize that may be
largely a style thing.

			regards,
				Ted

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 13 Jul 2009 11:56:18 -0700

> explicit set of distinguishing characteristics, but I recognize that  
I have added additional explanatory material in the forthcoming draft

Ekr