Subject: Re: FE-CE data path

From: "Chawla, Shuchi" <[email-masked]>
Date: Tue, 15 Mar 2005 13:13:26 -0800

Based on the discussions Furquan and I had after the ForCES meeting, and
some discussions with Joel, I am also leaning in the direction of not
using any kind of transport protocol for the data channel.  So, in that
sense, whether it's TCP or SCTP etc. would be immaterial.  It seems that
minimal intervention by ForCES (bypass PL/TML) on the data channel would
be appropriate: 
1) for performance reasons and 
2) this is data that ForCES shouldn't need to mess with

Based on the above, tunneling might be appropriate with the use of a
minimal header with enough information to process the packet once
received on the CE or the FE.  With respect to this data channel
overwhelming the control channel, as Furquan has mentioned, some sort of
rate limits may be implemented;  what may be problematic is to how to
guarantee these rate limits across multiple FEs connecting to a single
CE.  Presumably, there is some maximum bandwidth available per FE-CE
connection in an NE, and the limit of the data channel would be some
fraction of the per FE-CE connection?

If anyone is aware of data/experimentation available on the use of the
suggested options, that might be helpful.

Regards,
Shuchi
-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Furquan A. Ansari
Sent: Tuesday, March 15, 2005 12:43 PM
To: [email-masked]
Subject: Re: FE-CE data path

Folks,

As Joel pointed out, we chatted with him after the meeting (on the TCP 
TML) and discussed some of the interesting issues with using TCP for the

data channel. I have various reservations with respect to using TCP or 
any other
transport layer protocol (e.g. SCTP, UDP etc.) for the data channel. It 
is now my belief that a tunnel between the
CE and FE would be better suited for forwarding the data packets without

involving the transport layer protocols.
If congestion is an issue, it should be dealt with with rate limiting 
packets at the FE (e.g. 90% of the CE-FE b/w
for data packets so that control packets will always have enough spare 
capacity to go through).

I would like to bring out some of the issues for discussion for the data

channel. As authors of the draft, we ourselves
have not converged on what the ideal solution would be for the "data" 
channel - though we all agree that TCP would
be ideal for the control channel. [So, the comments below are based on 
my views and may not necessarily reflect
those of the other author's on the draft - though they are welcome to 
comment on it :) ]

PS: As for Jamal's comment regarding SCTP - I don't agree with him that 
SCTP would be better suited and
it falls in the same category as TCP and the issue are exactly the same 
as TCP.

---- list of things that I was discussing with other authors and one of 
our WG chair.

1) First, I don't think there should be a requirement for the "data" 
traffic to be congestion aware to begin with. It didn't
exist in current architectures in single chassis systems - so why should

it exist now.
    - not having a congestion aware "data" channel DOES NOT introduce 
"NEW" DDoS attack capabilities.
        It is just as good or as bad as existing systems. So it is fully

satisfying the requirements (and I don't think that the
        requirements RFC specifically mentions that the data channel has

to be congestion aware as well, which is correct         - let me know 
if  I am mistaken here).

2) I don't agree on how just having a congestion aware channel would 
"protect" against DDoS attacks. I just don't believe that using TCP or 
DCCP for the data channel would protect against DDoS attacks - since the

attacks can be more than just "bandwidth attacks" (SYN flooding etc.). 
All known solutions for DDoS bandwidth attacks are based on rate 
limiting only.

3) All protocol traffic that is destined to the CE as "data" traffic is 
essentailly non-forces control traffic (e.g.
OSPF, BGP, ISIS, LDP, RSVP etc.). Every protocol has in-built congestion

control. ICMP flooding may be an issue, but sending it over 
TCP/dccp/sctp  is not the solution.
    - If there are bogus protocol packets sent, they will still be 
limited to the rates allowed by the protocols themselves -
    unless the software running on the neighbors has been compromised - 
in which case you have much larger problems than congestion control on 
the FE-CE link (you basically have compromised software on routers)

4) Having TCP between FE and CE leads to severe performance degradation.

Data packets on incoming FE
interfaces can be switched to gre (or any other tunnel) at wire speed in

hardware. Using TCP or UDP ( read DCCP) - as would  be the case if DCCP 
is used - cannot achieve this. It will have to be taken out, processed, 
a PL header added and finally a TCP or UDP header added and sent over 
the session.
    - At the other end, you will have to terminate the outer transport 
session first and have to re-inject the packet
back into the kernel for terminating the inner transport session - be it

TCP or UDP. It's just plain ugly!
    - We essentially have the case of TCP in TCP or TCP in UDP (DCCP) or

TCP in SCTP etc. - all having the
    same disastrous performance implications and unknown behavior (the 
result of multiple control loops within
    a session)

    If a tunnel is used between the FE and CE, the packets arriving on 
the FE will be switched at wire speed with just
the tunnel header added - no TCP/UDP header, no PL header etc. and no 
processing by the forces module.

5) I am essentially saying that in some sense the TML header is the 
tunnel protocol header. I really don't see
the need for the "data" packets to be going through the ForCES PL layer 
for de-muxing to appropriate application (OSPF, BGP etc.) - Put it 
another way, I am basically saying that there is no PL or TML header for

the data packets - since tunneling will be transparent to the
applications.

6) The effects of reliability of TCP or SCTP on the FE-CE link are not 
clearly known and will be very difficult to
quantify the results. There will be multiple cases of duplicate packets 
being delivered to the applications (protocols)
that are not only unnecessary but also may potentially lead to incorrect

behavior. I can cite examples of how it can
potentially lead to incorrect operation for many protocols, but will 
defer for later.

7) I am not saying that the PL "redirect" header should be eliminated. 
There may still be some instances where
some data packets have to be sent from the FE to the CE encasulated in 
the PL message. Such messages can
be treated in a special way and sent over the TCP control channel itself

- with the PL redirect header added
to such packets.

8) Packet ordering is not an issue on the data channel - each individual

application will deal in its own way.

---------

So, the pitfalls of using any kind of transport layer protocol are 
similar - TCP, SCTP, DCCP (UDP). So, I propose
using some kind of tunneling mechanism for the data channel.

Thanks,

Joel M. Halpern wrote:

> In the TML document there is a very clear discussion of the tradeoffs 
> in using various protocols for forwarding data packets between the FE 
> and the CE (in either direction.)
> After the meeting, I chatted with the document authors a bit, as they 
> had found some interesting issues that were not mentioned in the
draft.
>
> For example, they quite correctly pointed out that forwarding IP 
> packets which are part of an external reliability scheme (TCP or OSPF 
> retransmission, or  ...) over TCP can produce some quite interesting 
> problems.
>
> Later, I was chatting with one of our chairs who also pointed out that

> forwarding TCP over TCP is not a good idea.
>
> However, the data channel can induce congestion.  And that congestion 
> can have very severe impacts on the control channel.  Particularly 
> since the control channel will slow down in the presence of
congestion.
>
> We could try to craft a multi-part scheme with configured rates, some 
> data traffic over TCP and some over UDP, and some additional feedback 
> to try to keep the whole thing in balance.  The odds are it won't work

> right.
>
> Given these choices, and given other external developments, I would 
> like to raises a question that has been discussed and agreed in the 
> past.  (The chairs can tell me there is not enough external change and

> I should shut up.)  DCCP is nearing IESG approval.  I am told that 
> there are DCCP stacks available.  Would it make more sense for us to 
> simply use DCCP for our mandatory-to-implement FE-CE data channel 
> protocol?  (This does not prohibit other protocols / encapsulations 
> for suitable environments.)
>
> Thank you,
> Joel M. Halpern

-- 
Furquan Ansari
Bell Labs Research			Ph : 732-949-5249
Holmdel, NJ				Fax: 732-949-4556

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Tue, 15 Mar 2005 16:09:42 -0800

Good point, Joel. This is one of the reasons we would need a some/PL
header for data packets.

I am not sure this would be solved with tunnels, unless you setup a
tunnel per external FE interface and that could be several hundred
tunnels if there are as many interfaces on the router.
You need this information (on which port/interface a packet arrived on
the FE), for multicast forwarding.

regards
Hormuzd

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Joel M. Halpern

>5) I am essentially saying that in some sense the TML header is the
tunnel 
>protocol header. I really don't see

>for de-muxing to appropriate application (OSPF, BGP etc.) - Put it
another 
>way, I am basically saying that there is no PL or TML header for the
data 
>packets - since tunneling will be transparent to the applications.

The data packets need some sort of header since we must know what
interface 
they arrived on, what FE sent them to the CE, and why the CE sent 
them.  That information can not reliably be determined purely from the 
underlying packet.

Yours,
Joel M. Halpern

---

From: "Putzolu, David" <[email-masked]>
Date: Wed, 16 Mar 2005 09:17:18 -0800

Hi Tolga.

I agree on the fast retransmit case.  The latencies involved
in a CE-FE connection are probably pretty small.

In regards to the computational cost of TCP vs. the computational
cost of the CE/FE processing the data itself,  can you provide
some examples?  Most of the L2, L3, and L4+ protocols I can think
of require at least an order of magnitude more processing for
the data than for transporting the data in TCP.  Off the top
of my head:

SNMP: Encoding alone is probably 2+ orders > TCP
Telnet: String processing at least 1+ order
SSH: Encryption at least 1+ order of magnitude
Routing protocols: SPF calculations will be 2+
Simple lookup protocols: 1+ order for validating and updating
  things like DHCP or ARP
Firewall rule manipulation: 1-2 orders of magnitude
  to validate and cook a new ACL entry

Of course, all of the above computation occurs on the CE.  Each
FE would have to pay the price of the TCP encap on its relatively
limited processor.  However, if a FE has the CPU horsepower to
parse the ForCES PL then it still feels to me like TCP is
trivial.

-David (hat off as usual)

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Tolga Asveren
Sent: Wednesday, March 16, 2005 7:37 AM
To: [email-masked]
Subject: Re: [FORCES] FE-CE data path

Just 2 cents from an outsider:

-Considering Fast Retransmit mechanism, will HOL a real issue even for
TCP,
for FE-CE communication -hopefully they don't communicate over big-I
Internet :-)-?

-Related with a former comment questioning the importance of computation
associated with connection oriented protocols with redundancy, e.g. TCP,
SCTP: People might want to use FORCES to build things except routers,
and
for certain cases the computation associated with redundancy might
matter.

   Tolga

> [mailto:[email-masked]]On Behalf Of Jamal Hadi Salim
> Sent: Tuesday, March 15, 2005 8:23 PM
> True SCTP isnt as widely deployed as say TCP or UDP;
> I dont think SCTP has the same technical problems as TCP as i was
> pointing out earlier if you use SCTP-PR.
>
> cheers,
> jamal
>
> On Tue, 2005-03-15 at 20:08, Deleganes, Ellen M wrote:
> > Even if there are good and stable DCCP implementations soon, this
> > doesn't mean it will become production soon. Won't that be an issue?
> > Even SCTP isn't all that widely deployed and it still has some of
the
> > technical problems people were raising regarding TCP.
> >
> > Regards,
> > Ellen

---

From: Furquan Ansari <[email-masked]>
Date: Wed, 16 Mar 2005 15:11:11 -0500

Hmm...Not sure if there is a problem with the list server of my outgoing
email - my mails to this list are not going. This is the 4th time I am
trying, but this time posting it directly using the web interface!
Hope it works...

-----------

Looks like that didn't work either - let me try it again....!!!

Subject:
Re: FE-CE data path
From:
"Furquan A. Ansari" <[email-masked]>
Date:
Wed, 16 Mar 2005 14:08:52 -0500
To:
[email-masked]

Ok.... I tried sending this long time back - but looks like something
happened and hasn't been received by the list.
So, here goes - again.....

----
Yes, good point indeed, Joel. I misinterpreted your question regarding "what
interface they arrive on" to mean at the CE (de-mux diff tunnel interfaces).
And, no  Hormuzd, I am not advocating having one tunnel per external FE
interface - that would be a pretty foolish thing to do. There will be a
single interface from an FE to a CE.

As a matter of fact, I have the CE/FE combination running as a normal router
- with OSPF etc. and should have looked at my own implementation more
carefully. One would indeed require/need the arriving interface information
- e.g.  if the packet is destined to a multicast address like OSPF packets
(for unicast, one can derive it from the destination address). Having said
that, the discussion of appending/providing some information to obtain this
result (intf info)  is different from needing a transport layer protocol.
There's nothing preventing one from still using a tunnel with the
information added - e.g. as Jamal pointed out in the VLAN example.

My motivation for tunnelling is to allow a packet to be forwarded at
wire-speed (ASICs) without it seeing the light of day at the processor of
the FE.  However, if some additional header/tag requires it to be pulled out
anyway, the performace advantages may not be that significant.

On a side note - I should have brought out this question long time back with
respect to the protocol draft (headers) but ended up simply using my own
propietary mechanism to ship the interface arrival information. The current
redirect header (PL header) does not contain any place on the header to
provide such information at all.

Thanks,

Khosravi, Hormuzd M wrote:

regards
Hormuzd

-----Original Message-----

tunnel

another

data

Yours,
Joel M. Halpern

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Wed, 16 Mar 2005 12:43:03 -0800

Furquan,

Just saw your email. To answer your question, this information is in the
Redirect message, not the common header. Specifically, it would be
conveyed by the LFB ID, since LFBs will represent the ports. See section
6.8 for more details.

Hope my message will post soon. We have had problems with the mailserver
in the past...it a good sign...shows the mailing list is busy :-))

regards
Hormuzd

-----Original Message-----
From: Forwarding and Control Element Separation
[mailto:[email-masked]] On Behalf Of Furquan Ansari
Sent: Wednesday, March 16, 2005 12:11 PM
To: [email-masked]
Subject: Re: [FORCES] FE-CE data path

Yes, good point indeed, Joel. I misinterpreted your question regarding
"what
interface they arrive on" to mean at the CE (de-mux diff tunnel
interfaces).
And, no  Hormuzd, I am not advocating having one tunnel per external FE
interface - that would be a pretty foolish thing to do. There will be a
single interface from an FE to a CE.

As a matter of fact, I have the CE/FE combination running as a normal
router
- with OSPF etc. and should have looked at my own implementation more
carefully. One would indeed require/need the arriving interface
information
- e.g.  if the packet is destined to a multicast address like OSPF
packets
(for unicast, one can derive it from the destination address). Having
said
that, the discussion of appending/providing some information to obtain
this
result (intf info)  is different from needing a transport layer
protocol.
There's nothing preventing one from still using a tunnel with the
information added - e.g. as Jamal pointed out in the VLAN example.

My motivation for tunnelling is to allow a packet to be forwarded at
wire-speed (ASICs) without it seeing the light of day at the processor
of
the FE.  However, if some additional header/tag requires it to be pulled
out
anyway, the performace advantages may not be that significant.

On a side note - I should have brought out this question long time back
with
respect to the protocol draft (headers) but ended up simply using my own
propietary mechanism to ship the interface arrival information. The
current
redirect header (PL header) does not contain any place on the header to
provide such information at all.

Thanks,

regards
Hormuzd

-----Original Message-----

tunnel

another

data

Yours,
Joel M. Halpern