Subject: Re: FE-CE data path

From: "Putzolu, David" <[email-masked]>
Date: Tue, 15 Mar 2005 14:59:05 -0800

[hat off...]

Hi Jamal - comment inline.

-David 

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Jamal Hadi Salim
Sent: Monday, March 14, 2005 2:32 AM
To: [email-masked]
Subject: Re: [FORCES] FE-CE data path

Joel,

I missed one subtle thing from your earlier email which became obvious
reading Furquans last email - in that we are only talking about data
"channel".

Yes i was suggesting SCTP-PR instead of DCCP. I wouldnt call it
non-retransmiting; however it is tunable to remove HOL.

However, upon further reflection, even i would not advocate the
mandatory use of SCTP-PR in this case. 

My reasons are slightly different than what Furquan and Shuchi: I do
think issues they faced of "reliability tunings" such as timers mismatch
etc that result in HOL, can be overcome with SCTP-PR/UDP/DCCP (but not
TCP).  
However, I dont think that we can force an implementation scheme.
A lot of commodity hardware exists that can encapsulate the data
channel packets in some header such as GRE, VLAN etc. Enforcing that
such packets MUST be encapsulated in SCTP is going to be a perfomance
problem and not very practical. 

David> In order to achieve true interop between FEs and CEs
David> don't we have to specify a particular protocol to use?
David> Put another way, if we do not specify the protocol, 
David> what guarantee is there that FooCorp FE can communicate
David> with BarCorp CE?
David> 
David> Also, is there strong evidence that SCTP, TCP, DCCP, 
David> etc. are performance problems?   The processing of the
David> app layer payload bytes for the protocols a router
David> engages in is at least two orders of magnitude more
David> complex than transport layer processing.  I.e. for
David> every byte of OSPF traffic a router receives, it may
David> spend 10 cycles to encapsulate it in TCP,  but it 
David> will probably spend 1,000 cycles for the associated 
David> SPF calculation.  Reducing the transport overhead
David> from 10 cycles to 1 cycle won't actually affect the
David> router performance in any way...

cheers,
jamal

On Tue, 2005-03-15 at 16:04, Joel M. Halpern wrote:
> I think that there is a different issue.
> TCP and SCTOP will both choose to provide retransmission.
> For the data channel, we want adaptive congestion control (rate
limiting) 
> with dropping of old packets.  We do not want ever want data channel 
> packets retransmited.
> 
> In contrast, the control channel actually requires retransmission for 
> reliability.
> 
> Are you suggesting a non-retransmitting mode of SCTP as an alternative
to 
> DCCP for the data channel?
> If so, I would be interested in the details of what that mode does.
(Also, 
> since most of us associated SCTP with the reliable mode, I at least
jumped 
> to the conclusion you meant reliable SCTP when you initially suggested
SCTP 
> without a qualifier.)
> 
> Yours,
> Joel
> 
> At 04:26 AM 3/14/2005, Jamal Hadi Salim wrote:
> >The real problem is not _reliability_ as provided by TCP - rather the
> >Head of Line Blocking presented by TCP (which is clearly visible when
> >you run a reliable protocol on top of it such as mentioned OSPF).
> >
> >PR-SCTP resolves this. Look at RFC 3758.
> >
> >cheers,
> >jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 14 Mar 2005 08:08:52 -0500

David,

> David> what guarantee is there that FooCorp FE can communicate

IMO, this falls into a general config issue:
You dont necessarily need a protocol. But you need some LFB which both
understand to make sure they interop.
In the example above if BarCorp CE knows that FooCorp FE supports
hardware level packet redirection (because it discovered this in the
capability phase) and that the FE can police and can attach VLAN headers
to said redirected packets then it can tell it, "when you see OSPF
packets with following signature .... meter not to exceed 1Mbps, if it
does drop it; if it doesnt, attach VLAN tag X, srcMACY, dstMACx,
priority value X and then send to your port Z". 
The CE maps that VLAN with those parameters to whatever s.ware (or
instance of) that is executing OSPF. 
I am gonna claim this feature above is commodity or close to. 
Infact i see this as being very common with newer and upcoming
architectures like ATCA where theres a clear demarcation between
base (control) links and fabric(data) links.
Of course to facilitate the interop someone is gonna have to 
write a draft describing these schemes - maybe i should whip one.

> David> router performance in any way...

I agree, the implications maybe highly exagerated - however, if you
were to look at the paths the two approaches take it is clear
they will perform relatively differently.
i.e
In the VLAN example i described above, the packet is shunted within
hardware never seeing the s/ware.
In the case you encapsulate with TCP etc, you have to cross fabric
to CPU and depending on OS you cross from one protection domain to
another and then reverse again (userspace->kernel->fabric --> CE)

I am actually not against using a protocol _but_ as i mentioned 
earlier i am not for mandating it either, for practical reasons.
If you gave me a choice between encapsulating on some IP transport
protocol and using VLANs when i have hardware that can do it i would do
the later. I have a feeling that most will.

cheers,
jamal