Subject: Re: I-D ACTION:draft-khosravi-forces-tcptml-01.txt

From: "Chawla, Shuchi" <[email-masked]>
Date: Thu, 24 Feb 2005 09:06:46 -0800

Hello all,

I joined the ForCES group recently, and I'm a co-author of the TCP/IP
TML draft that was submitted recently.  

With respect to the API we have defined in the draft, we did intend for
it to be purely informative to demonstrate realization of the services
provided by the TML layer in the context of the ForCES protocol.  We
included in the TCP/IP TML draft since there is no other home for it at
the moment, with the knowledge, that it may have to be moved to a
separate informative draft.  

In my opinion, having gone through the process of defining the API
(although not as a complete API spec.) helped us ensure that the model
being proposed for services to be provided by the TML layer was indeed
legitimate. I believe having such an informative draft would be
beneficial to anyone wanting to implement the TML, and would likely
minimize interoperability issues that may arise between different TML
implementations.

Thanks.

Regards,
Shuchi Chawla

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Putzolu, David
Sent: Thursday, February 24, 2005 5:51 AM
To: [email-masked]
Subject: Re: I-D ACTION:draft-khosravi-forces-tcptml-01.txt

(hat off)
I agree on the value of defining a generic SAP,
and think it would be a natural fit as an appendix 
of the PL draft as it should be common to all TMLs.

In terms of the API draft, it is more challenging (but not
unprecedented) to do APIs in the IETF.  However, an
informative draft that documents how the SAP is realized as
an API (either generically or in Linux) would probably be
valuable and acceptable.

Cheers,
David

> -----Original Message-----
> From: Forwarding and Control Element Separation 
> [mailto:[email-masked]] On Behalf Of Joel M. Halpern
> Sent: Tuesday, February 22, 2005 10:04 AM
> To: [email-masked]
> Subject: Re: [FORCES] I-D ACTION:draft-khosravi-forces-tcptml-01.txt
> 
> My thanks to the authors for this revision of their work.
> This is a clear and useful description of a TML.
> 
> Two minor, related comments not so much about this document 
> but about how 
> the working group can handle one aspect of this.
> 
> It seems to me that for multiple TMLs to be useful, the upper 
> interface 
> provided by the different TMLs needs to be the same.  This 
> prompts two 
> thoughts:
>      Can the authors of other TML proposals please look at 
> the API defined 
> here, and let the working group know if that looks like the 
> interface they 
> would like to provide?
>      Once we get agreement, can we move that into the PL 
> document (since 
> the alternative, as far as I can tell, would be to try to 
> duplicate it 
> without change into each TML document.)
> 
> Next, could we possibly define this as a service interface 
> rather than an 
> API.  I don't think that we want to mandate that a PL must 
> invoke these 
> exact subroutine calls.  We do want to say that the PL will use these 
> services from the TML, resulting in a specific and 
> predictable on-the-wire 
> sequence.
> 
> Thank you,
> Joel
> 
> At 03:45 PM 2/21/2005, [email-masked] wrote:
> >A New Internet-Draft is available from the on-line Internet-Drafts 
> >directories.
> >
> >
> >         Title           : TCP/IP based TML (Transport 
> Mapping Layer) for 
> > ForCES protocol
> >         Author(s)       : H. Khosravi, et al.
> >         Filename        : draft-khosravi-forces-tcptml-01.txt
> >         Pages           : 25
> >         Date            : 2005-2-21
> >
> >This document defines the TCP/IP based TML (Transport Mapping Layer)
> >    for the ForCES protocol. It explains the rationale for 
> choosing the
> >    transport protocols and also describes how this TML addresses all
> >    the requirements described in the Forces [3] 
> requirements and ForCES
> >    protocol [7] document.
> >
> >A URL for this Internet-Draft is:
> >http://www.ietf.org/internet-drafts/draft-khosravi-forces-tcp
tml-01.txt

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 25 Feb 2005 13:56:47 -0500

I finaly had the chance to scan the draft. 

Some comments - also sharing our implementation experience of a TCP TML.

1) TML<->TML interface.

You dont need a protocol other than TCP or any additional headers - the
TCP headers should be sufficient - the TCP header is sufficient.

We actually started with the scheme described in the draft i.e a little
shim header and an additional control protocol. 
Eventually we found the only reason we needed the headers or the little
protocol was for multicast/replicast joining/leaving given TCP is
unicast only. Any other need can be achieved via standard TCP features. 
To simplify, we evolved towards having a separate inter-TML control
channel. This channel never carries PL messages. All it does at the
moment is carry the replicast join/leave state setup/teardown.
This is where our implementation sits right now.

We have now reached a conclusion that theres no need even for this extra
channel for multicast [1]. 

I think there maybe other TMLs that may require coordination and their
own protocol (such as TIPC which runs over ethernet or directly over a
fabric). In the case you are running over IP and another transport such
as TCP, it makes sense to reuse the services they offer.

2)PL<->TML interface.

The fact that a TCP TML has two channels being passed and that such
knowledge is controlled by and embedded to the PL will complicate
things. The configuration of the two channels is up to the C/FEM IMO.
This knowledge should be oblivious to the PL -  if you want simplicty
that is.

3) I am begining to worry a little about the redirection channel. 

Enforcing that all redirected messages have to have a PL header attached
to them would become problematic. Theres a gazillion ways that packets
could be redirected and most of the time dependent on hardware
capability. For example our existing hardware could redirect packets and
put extra VLAN tags on them; other much simpler schemes would simply
redirect raw packets (OSPF for example) based on the addresses.
I think we may need to rediscuss this in the context of the protocol
(dont wanna drag on this context because i will end up distracting the
topic). 

cheers,
jamal

[1] If we assume that the FE multicast PIDs belong to the FE Object
(theres a very good fit) then subscription can happen at the PL FE
Object level. This needs discusssion; however, upon receiving such a
message, the FE PL layer makes a local API call to add the multicast
group to the TML. All this local API call does is to tell the TML
"Add this to my list of legit IDs". Like i said above this still needs
discussion.

---

From: Furquan Ansari <[email-masked]>
Date: Fri, 25 Feb 2005 15:19:13 -0500

Jamal,
    Thanks for the comments. See inline comments....

>This is where our implementation sits right now.
Yes, we didn't really like the idea of having a separate TML shim header 
either and would love to avoid using it,
if possible. Like you mentioned, the reason this even exists in the 
draft is because there are certain
messages that need to be exchanged between the two TMLs - multicast 
join/leave being an example. And, yes
this can be avoided by using a completely separate channel for TML 
control - but it adds yet another channel
between the FE and CE.

....See more detials in comment (3)

>This knowledge should be oblivious to the PL -  if you want simplicty
My thinking is similar with respect to second part of your comment. As 
for the first part, not sure if the knowledge
at PL will add any significant complications. In fact, its possible that 
not having this information may make things
more complicated - because, if one of the channels (either the control 
or the data) goes down, the TML layer
has to automatically re-establish the channel - and this requires state 
maintenance and additional triggering mechanism. If the PL has the 
knowledge of it that the channel has closed down, it can trigger the 
open() call to re-establish again - simple.

>(dont wanna drag on this context because i will end up distracting the
(This comment and first comment are related)

Agreed - we don't necessarily need the TML header on redirected messages 
- as long as these messages are
carried over the separate data channel. However, if at some point there 
is a potential possibility of mixing/multiplexing
these messages with other control messages (e.g. if the PL control and 
data over single channel due to F/CEM conf.)
then one needs to have the shim header to appropriately de-mux the messages.

So basically one can imagine the following scenarios:

(1) 3 channels:
    - TML-TML, PL control, PL data - No TML Shim Header is needed
(2) 2 Channels
    (a) TML-TML/PL Control,  PL Data - TML header needed for Demux
    (b)  TML-TML,  PL Control/PL Data  - No TML header needed
(3) 1 Channel
    - TML-TML/PL Control/PL Data all together - TML Header needed

The draft of course is still preliminary and talks from the perspective 
of using Option 2(a) and potentially Option 3.
Of course, we can have further discussion on which would be the best choice.

>"Add this to my list of legit IDs". Like i said above this still needs
Hmm...not sure I fully understand where you are going with this. Are you 
saying that no join/leave group "message"
is needed between the CE and FE - and only a local API call to add 
multicast is sufficient?
If you see the draft, there is in fact a local API call from the PL to 
the TML to add the multicast group to the TML. However, as a result of 
this call, the CE needs to be informed of the new multicast receiver 
that joined the group and hence we need a Join (or Leave) "message" 
between the FE and CE TMLs.

Thanks,

-- 
Furquan Ansari
Bell Labs Research			Ph : 732-949-5249
Holmdel, NJ				Fax: 732-949-4556

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 25 Feb 2005 15:31:47 -0500

At least for the redirected packets (which folks are arguing should not 
have a shim header) I can not see how to avoid such a header.   Although 
the IP Header has a total length field, one of the reasons for sending the 
packet to CE may be that the amount of data received and the total length 
actually received may not match.
We can argue that in such a case we should never send that packet to the 
CE, but that seems an odd restriction.

Yours,
Joel

PS: This also mandates that the PL header have a total length field.  Is it 
really the job of the TML to tell the PL how to do its job?

---

From: Furquan Ansari <[email-masked]>
Date: Fri, 25 Feb 2005 16:16:55 -0500

Joel,
    But couldn't this be deduced from the Outer IP header? - if the 
transport mechanism is TCP/IP between the
FE and CE. I pressume that you mean that the FE receives an IP packet 
from a neighbor which could have
length mismatch and has to be redirected to the CE. This IP packet would 
be encapsulated within the TCP/IP
channel as payload between the CE and FE. In such as case, the outer IP 
header should contain the "actual" length
received.

Thanks,

Joel M. Halpern wrote:

> At least for the redirected packets (which folks are arguing should 
> not have a shim header) I can not see how to avoid such a header.   
> Although the IP Header has a total length field, one of the reasons 
> for sending the packet to CE may be that the amount of data received 
> and the total length actually received may not match.
> We can argue that in such a case we should never send that packet to 
> the CE, but that seems an odd restriction.
>
> Yours,
> Joel
>
> PS: This also mandates that the PL header have a total length field.  
> Is it really the job of the TML to tell the PL how to do its job?

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 25 Feb 2005 16:33:33 -0500

Remember that TCP is a stream protocol, not a packet protocol.  There is no 
"outer IP packet" to look at.

Yours,
Joel

At 04:16 PM 2/25/2005, Furquan Ansari wrote:
>FE and CE. I pressume that you mean that the FE receives an IP packet from 
>a neighbor which could have
>>have a shim header) I can not see how to avoid such a header.
>>Although the IP Header has a total length field, one of the reasons for 
>>sending the packet to CE may be that the amount of data received and the 
>>total length actually received may not match.

---

From: Furquan Ansari <[email-masked]>
Date: Fri, 25 Feb 2005 17:07:30 -0500

Unless you are using TCP over some other network protocol, I have to 
assume that the TCP stream
is carried within IP datagrams. So, the way I am seeing this is:

Headers involved:
------------
Orig Received at FE: [[Payload]IP]

Redirect Data from FE to CE: [[[{[Payload]IP}PL Hdr]TCP]IP]

So, what am I missing?

Joel M. Halpern wrote:

> Remember that TCP is a stream protocol, not a packet protocol.  There 
> is no "outer IP packet" to look at.
>
> Yours,
> Joel
>
>> length mismatch and has to be redirected to the CE. This IP packet 
>> would be encapsulated within the TCP/IP
>> channel as payload between the CE and FE. In such as case, the outer 
>> IP header should contain the "actual" length
>> received.
>>
>> Thanks,
>>
>> Joel M. Halpern wrote:
>>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 25 Feb 2005 19:57:26 -0500

> >Some comments - also sharing our implementation experience of a TCP TML.
[..]

> control - but it adds yet another channel

The extra TML-TML control channel thing is the way we have things at the
moment in our implementation - but it should die too if we can reach a
conclusion on how to do PL level multicast; more after (3) below

[..]

> for the first part, not sure if the knowledge at PL will add any significant 
> complications. In fact, its possible that 
> not having this information may make things more complicated - because, 
> if one of the channels (either the control or the data) goes down, the TML layer

I would see the TML as a blackbox to the PL at least. Whether it has
11 connections or 1 should be opaque to the TL - whether it actually
multicasts or replicasts should also not be something the PL should need
worry about. The setup of these parameters is one of the reasons we have
the xEM interface.
On the example you provide above: I can see a scenario where HA
decisions may involve the PL layer; but thats a slightly different issue
i.e all the TML would say is "btw, I just lost connection to CE1" and
the PL would say "ok, connect to CE2" - this, however, does not put into
the equation that 1 or 10 sockets went down. 

[In the case where you have underlying transport HA (ethernet channel
bonding, IP interfaces using VRRP or even SCTP multiple associations via
multihoming), these details of a transport channel going down could be
resolved by the TML layer without any involvement of the PL. But like i
said earlier this is a different issue].

> then one needs to have the shim header to appropriately de-mux the messages.

Agreed. What i was worried about is enforcing one scheme of "always add the PL
shim header for redirect". It may not be practical. Infact i suspect it will be 
the least used in the real world.

> Of course, we can have further discussion on which would be the best choice.

At the moment i am only referencing TCP TML. I dont think i can say the same for
TIPC running over PCI-Express AS switching or over ethernet. 
I think we need to just stay put with PL over TCP and no intermediate between 
protocol between TCP and the PL layer. In other words there is no TML-TML
(if we migrate multicast activity to FE Object). So it would be #2 above
with a twist:
PL Control, PL data, possibly other PL data channel(s).

> is needed between the CE and FE - and only a local API call to add 

Not exactly. Let me explain a join (leave will be equivalent). Note, we have not implemented
this, so it may have holes.

CE:
1 a). Join issued by CE to a select FEs for Mcast ID X. The join is sent to the FE Object 
at some path. Path could be to attribute or event subscription.
1 b). On succesfull response from all FEs, "add multicast" service call to local TML.
1c). This means we have now successfully added state at the CE and all FEs of interest.

FE: 
2a). On successfully processing 1a), the FE PL issues a local "add Multicast" for ID x.
2b)  response to config from 1a) to CE
2c) FE now has all necessary state to process these messages.

> between the FE and CE TMLs.

Yes, this is what we do as well today - but we want to get rid of it so it doesnt get 
propagated across.
You get rid of this and you dont need the TML protocol or any shim header.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 25 Feb 2005 20:04:12 -0500

In the case of redirection to a specific VLAN:
received example: [[Payload]IPx] matching a filter
Redirect Data from FE to CE: [[[{[Payload]IPx}MAChdr+vlan tag].
You can demux this easily and figure out the different lengths.
With the right hardware you can use different tags for different
LFB redirects.

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 25 Feb 2005 20:53:22 -0500

Assuming you knew where the packet started (because you knew where the 
previous one ended) how would you know where this packet ended?  If you 
simply ship the bytes over the TCP connection, TCP will not tell you where 
the boundary was.  And if you ship two such packets back to back, you may 
get handed both of them on a single read from TCP.

Yours,
Joel