Subject: Re: I-D ACTION:draft-khosravi-forces-tcptml-01.txt

From: "Chawla, Shuchi" <[email-masked]>
Date: Mon, 28 Feb 2005 08:48:39 -0800

Jamal,

Thanks for the feedback on the draft.  Comments below:

TML Header (or lack thereof)
----------------------------
I concur that it would simplify the protocol to not require a TML
header.  However, before we make that decision, I think we would want to
make sure we have covered ground with respect to all of the
communication that may be required between TML peers.  We had identified
4 messages so far in this version of the draft: 
- 2 multicast support related messages
- 1 heartbeat related message
- 1 channel closure related message

Of these, the first three have been addressed in some form in these
discussions.  As we detail the protocol further, there may be other
messages that may come up - so we would need to ensure that all of these
messages can either be eliminated or dealt with in some other manner.
So, I just feel that this needs some more analysis.

Separate Peer TML Control Channel 
---------------------------------
If we do require peer TML communication, in my opinion the use of a
separate communication channel for TML messaging is likely to complicate
the situation - in effect, this implies the setup/use of a "control
channel for a control channel".  I have not thought about all of the
implications, but here are some areas that the protocol would likely
need to address:
- What if one control channel comes up and the other doesn't?
- Guaranteeing message sequencing between these two control channels may
become more problematic
- The TML Layer even with this separate channel needs the ability to
handle two types of messages:
	o Those with a TML layer header received over the TML-TML
control channel
	o Those without a TML layer header received over the PL-PL
control channel

TML Versioning
--------------
Not having a TML Layer header would imply that the TML Layer does not
have a version associated with it.  Is there any other place the TML
version may be embedded?  If not, it would imply that changes may not be
made to the TML Layer separate from the PL Layer since any change that
might impact interoperability or modify the layer's behavior would need
to be reflected in the protocol version.  Since there are multiple TMLs
that are likely to exist, but a single PL, it would seem prudent that
TML changes be versioned independent of the PL Layer.  How would this be
achieved?

PL Layer knowledge of (Two) Channels
------------------------------------
I haven't thought about this deeply, but here are my initial thoughts:
As long as the TML Layer is only responsible for "transport" and is not
a decision-maker, I think the PL Layer (or someone other than the TML
Layer) should have knowledge of the channels being setup.  This will
especially be required to handle failure cases.  If the TML Layer is
indeed the only entity aware of this, then it seems that in order to
deal with failure cases, the TML Layer would need to take charge, and in
that case, there may actually be more messaging required between peer
TMLs.

Also, we haven't addressed HA in detail this draft.  I think some of the
decisions we make will need to take into account HA requirements - it is
hard to architect HA in at a later stage.

Regards,
Shuchi

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Jamal Hadi Salim
Sent: Friday, February 25, 2005 10:57 AM
To: [email-masked]
Subject: Re: I-D ACTION:draft-khosravi-forces-tcptml-01.txt

I finaly had the chance to scan the draft. 

Some comments - also sharing our implementation experience of a TCP TML.

1) TML<->TML interface.

You dont need a protocol other than TCP or any additional headers - the
TCP headers should be sufficient - the TCP header is sufficient.

We actually started with the scheme described in the draft i.e a little
shim header and an additional control protocol. 
Eventually we found the only reason we needed the headers or the little
protocol was for multicast/replicast joining/leaving given TCP is
unicast only. Any other need can be achieved via standard TCP features. 
To simplify, we evolved towards having a separate inter-TML control
channel. This channel never carries PL messages. All it does at the
moment is carry the replicast join/leave state setup/teardown.
This is where our implementation sits right now.

We have now reached a conclusion that theres no need even for this extra
channel for multicast [1]. 

I think there maybe other TMLs that may require coordination and their
own protocol (such as TIPC which runs over ethernet or directly over a
fabric). In the case you are running over IP and another transport such
as TCP, it makes sense to reuse the services they offer.

2)PL<->TML interface.

The fact that a TCP TML has two channels being passed and that such
knowledge is controlled by and embedded to the PL will complicate
things. The configuration of the two channels is up to the C/FEM IMO.
This knowledge should be oblivious to the PL -  if you want simplicty
that is.

3) I am begining to worry a little about the redirection channel. 

Enforcing that all redirected messages have to have a PL header attached
to them would become problematic. Theres a gazillion ways that packets
could be redirected and most of the time dependent on hardware
capability. For example our existing hardware could redirect packets and
put extra VLAN tags on them; other much simpler schemes would simply
redirect raw packets (OSPF for example) based on the addresses.
I think we may need to rediscuss this in the context of the protocol
(dont wanna drag on this context because i will end up distracting the
topic). 

cheers,
jamal

[1] If we assume that the FE multicast PIDs belong to the FE Object
(theres a very good fit) then subscription can happen at the PL FE
Object level. This needs discusssion; however, upon receiving such a
message, the FE PL layer makes a local API call to add the multicast
group to the TML. All this local API call does is to tell the TML
"Add this to my list of legit IDs". Like i said above this still needs
discussion.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 28 Feb 2005 12:43:55 -0500

Hi Shuchi,

On Mon, 2005-02-28 at 11:48, Chawla, Shuchi wrote:
> communication that may be required between TML peers. 

A reasonable thing to do

>  We had identified

I am gonna assume TCP - which is in the context of this draft.

> - 2 multicast support related messages

I described the scheme in earlier email in response to Furquan - involve
the FE Object and have the multicast be a PL/FE Object
config/subscription issue.

> - 1 heartbeat related message

use TCP KEEPALIVE; 

> - 1 channel closure related message

EOF is returned when a TCP connection is closed

> So, I just feel that this needs some more analysis.

Refer to what i said above. We are lucky to have implemented along the
lines of what you propose (refer to the history i gave earlier)- that
helped shape my views.

> Separate Peer TML Control Channel 

In context of TCP - I would say kill this peer TML relationship. I
probably wouldnt say the same for TIPC.

> TML Versioning

I see versioning as useful in the case where you have a shim header.
Any other case - the versioning is how code is implemented; and if not
I would say this is a C/FEM config interface issue. If someone wants to
have CEM<->FEM communication they could do that; i would till think of
it mostly as a config of the TML. 

> PL Layer knowledge of (Two) Channels

My view is that the PL should only worry about Forces Level messaging
i.e treat the TML as a black box; a wire if you may. It just sends
things down and they get to its peers.
The PL may "boot" the TML - but thats about as much control it has over
it.
Remember: in a PL<->TML interface, open() does not necessarily equate to
a TCP socket open() - it doesnt in our case. It just means you are
connected to the TML.

> hard to architect HA in at a later stage.

I think we are there - whats missing is the details.

Two levels of HA:
1) Transport - this is where TML is responsible. Its one layer below the
PL. If a physical link goes down or a socket dies then TML takes
necessary action to recover. If absolutely no possibility of such
recovery (eg when a CE or FE dies) then the PL layer needs to be
involved.
If you remove the relationship of knowledge of how many transport
connections actually exist from the PL level - this becomes more a 
TML module worry and becomes specific to the hardware in which the TCP
TML was implemented on.

2) a FE or CE PL. connection to the peer level is down. This is the
responsibility of the PL. If a CE disappears, then its clear the FE 
needs to kick in the HA rules it has and either choose another CE or 
hibernate until current one comes back (we discuss a few of these
schemes in the draft - may need some reviewing).
This is clearly a PL module worry.

cheers,
jamal