Subject: FE-CE data path

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 15 Mar 2005 11:37:46 -0500

In the TML document there is a very clear discussion of the tradeoffs in 
using various protocols for forwarding data packets between the FE and the 
CE (in either direction.)
After the meeting, I chatted with the document authors a bit, as they had 
found some interesting issues that were not mentioned in the draft.

For example, they quite correctly pointed out that forwarding IP packets 
which are part of an external reliability scheme (TCP or OSPF 
retransmission, or  ...) over TCP can produce some quite interesting problems.

Later, I was chatting with one of our chairs who also pointed out that 
forwarding TCP over TCP is not a good idea.

However, the data channel can induce congestion.  And that congestion can 
have very severe impacts on the control channel.  Particularly since the 
control channel will slow down in the presence of congestion.

We could try to craft a multi-part scheme with configured rates, some data 
traffic over TCP and some over UDP, and some additional feedback to try to 
keep the whole thing in balance.  The odds are it won't work right.

Given these choices, and given other external developments, I would like to 
raises a question that has been discussed and agreed in the past.  (The 
chairs can tell me there is not enough external change and I should shut 
up.)  DCCP is nearing IESG approval.  I am told that there are DCCP stacks 
available.  Would it make more sense for us to simply use DCCP for our 
mandatory-to-implement FE-CE data channel protocol?  (This does not 
prohibit other protocols / encapsulations for suitable environments.)

Thank you,
Joel M. Halpern

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 14 Mar 2005 03:16:04 -0500

> forwarding TCP over TCP is not a good idea.

Glad this is being understood. 
[At least I hope not to be chased around anymore with baseball bats for
being against TCP].

> prohibit other protocols / encapsulations for suitable environments.)

Ive mentioned SCTP a few times so far as an alternative.

We (Weiming, Robert, Alex and myself) are working on a SCTP based TML
to replace the old draft we had. We will be putting out a draft soon.

Both SCTP and DCCP are still newish. However, SCTP is far more widely
deployed. The only missing "feature" from the SCTP swiss knife is
multicast. There maybe need for the TML to deploy UDP in addition when
multicast is needed.

cheers,
jamal

---

From: Tolga Asveren <[email-masked]>
Date: Tue, 15 Mar 2005 14:34:39 -0500

one question inline...
> -----Original Message-----
> From: Forwarding and Control Element Separation
> [mailto:[email-masked]]On Behalf Of Jamal Hadi Salim
> Sent: Monday, March 14, 2005 3:16 AM
> To: [email-masked]
> Subject: Re: FE-CE data path
>
>
[TOLGA]If reliability is provided externally -by running TCP/SCTP etc... on
service endpoints, and I think this is assumed in the context of this
discussion-, wouldn't SCTP also an overkill, afterall it also would provide
an additonal layer of reliability, which would consume computing resources?
For such a setup, the only problem I see with UDP is lack of congestion
control, which is the reason for DCCP.

---

From: "Furquan A. Ansari" <[email-masked]>
Date: Tue, 15 Mar 2005 15:42:51 -0500

Folks,

As Joel pointed out, we chatted with him after the meeting (on the TCP 
TML) and discussed some of the interesting issues with using TCP for the 
data channel. I have various reservations with respect to using TCP or 
any other
transport layer protocol (e.g. SCTP, UDP etc.) for the data channel. It 
is now my belief that a tunnel between the
CE and FE would be better suited for forwarding the data packets without 
involving the transport layer protocols.
If congestion is an issue, it should be dealt with with rate limiting 
packets at the FE (e.g. 90% of the CE-FE b/w
for data packets so that control packets will always have enough spare 
capacity to go through).

I would like to bring out some of the issues for discussion for the data 
channel. As authors of the draft, we ourselves
have not converged on what the ideal solution would be for the "data" 
channel - though we all agree that TCP would
be ideal for the control channel. [So, the comments below are based on 
my views and may not necessarily reflect
those of the other author's on the draft - though they are welcome to 
comment on it :) ]

PS: As for Jamal's comment regarding SCTP - I don't agree with him that 
SCTP would be better suited and
it falls in the same category as TCP and the issue are exactly the same 
as TCP.

---- list of things that I was discussing with other authors and one of 
our WG chair.

1) First, I don't think there should be a requirement for the "data" 
traffic to be congestion aware to begin with. It didn't
exist in current architectures in single chassis systems - so why should 
it exist now.
    - not having a congestion aware "data" channel DOES NOT introduce 
"NEW" DDoS attack capabilities.
        It is just as good or as bad as existing systems. So it is fully 
satisfying the requirements (and I don't think that the
        requirements RFC specifically mentions that the data channel has 
to be congestion aware as well, which is correct         - let me know 
if  I am mistaken here).

2) I don't agree on how just having a congestion aware channel would 
"protect" against DDoS attacks. I just don't believe that using TCP or 
DCCP for the data channel would protect against DDoS attacks - since the 
attacks can be more than just "bandwidth attacks" (SYN flooding etc.). 
All known solutions for DDoS bandwidth attacks are based on rate 
limiting only.

3) All protocol traffic that is destined to the CE as "data" traffic is 
essentailly non-forces control traffic (e.g.
OSPF, BGP, ISIS, LDP, RSVP etc.). Every protocol has in-built congestion 
control. ICMP flooding may be an issue, but sending it over 
TCP/dccp/sctp  is not the solution.
    - If there are bogus protocol packets sent, they will still be 
limited to the rates allowed by the protocols themselves -
    unless the software running on the neighbors has been compromised - 
in which case you have much larger problems than congestion control on 
the FE-CE link (you basically have compromised software on routers)

4) Having TCP between FE and CE leads to severe performance degradation. 
Data packets on incoming FE
interfaces can be switched to gre (or any other tunnel) at wire speed in 
hardware. Using TCP or UDP ( read DCCP) - as would  be the case if DCCP 
is used - cannot achieve this. It will have to be taken out, processed, 
a PL header added and finally a TCP or UDP header added and sent over 
the session.
    - At the other end, you will have to terminate the outer transport 
session first and have to re-inject the packet
back into the kernel for terminating the inner transport session - be it 
TCP or UDP. It's just plain ugly!
    - We essentially have the case of TCP in TCP or TCP in UDP (DCCP) or 
TCP in SCTP etc. - all having the
    same disastrous performance implications and unknown behavior (the 
result of multiple control loops within
    a session)

    If a tunnel is used between the FE and CE, the packets arriving on 
the FE will be switched at wire speed with just
the tunnel header added - no TCP/UDP header, no PL header etc. and no 
processing by the forces module.

5) I am essentially saying that in some sense the TML header is the 
tunnel protocol header. I really don't see
the need for the "data" packets to be going through the ForCES PL layer 
for de-muxing to appropriate application (OSPF, BGP etc.) - Put it 
another way, I am basically saying that there is no PL or TML header for 
the data packets - since tunneling will be transparent to the applications.

6) The effects of reliability of TCP or SCTP on the FE-CE link are not 
clearly known and will be very difficult to
quantify the results. There will be multiple cases of duplicate packets 
being delivered to the applications (protocols)
that are not only unnecessary but also may potentially lead to incorrect 
behavior. I can cite examples of how it can
potentially lead to incorrect operation for many protocols, but will 
defer for later.

7) I am not saying that the PL "redirect" header should be eliminated. 
There may still be some instances where
some data packets have to be sent from the FE to the CE encasulated in 
the PL message. Such messages can
be treated in a special way and sent over the TCP control channel itself 
- with the PL redirect header added
to such packets.

8) Packet ordering is not an issue on the data channel - each individual 
application will deal in its own way.

---------

So, the pitfalls of using any kind of transport layer protocol are 
similar - TCP, SCTP, DCCP (UDP). So, I propose
using some kind of tunneling mechanism for the data channel.

Thanks,

Joel M. Halpern wrote:

> In the TML document there is a very clear discussion of the tradeoffs 
> in using various protocols for forwarding data packets between the FE 
> and the CE (in either direction.)
> After the meeting, I chatted with the document authors a bit, as they 
> had found some interesting issues that were not mentioned in the draft.
>
> For example, they quite correctly pointed out that forwarding IP 
> packets which are part of an external reliability scheme (TCP or OSPF 
> However, the data channel can induce congestion.  And that congestion 
> can have very severe impacts on the control channel.  Particularly 
> since the control channel will slow down in the presence of congestion.
>
> We could try to craft a multi-part scheme with configured rates, some 
> data traffic over TCP and some over UDP, and some additional feedback 
> to try to keep the whole thing in balance.  The odds are it won't work 
> right.
>
> Given these choices, and given other external developments, I would 
> like to raises a question that has been discussed and agreed in the 
> past.  (The chairs can tell me there is not enough external change and 
> I should shut up.)  DCCP is nearing IESG approval.  I am told that 
> there are DCCP stacks available.  Would it make more sense for us to 
> simply use DCCP for our mandatory-to-implement FE-CE data channel 
> protocol?  (This does not prohibit other protocols / encapsulations 
> for suitable environments.)
>
> Thank you,
> Joel M. Halpern

-- 
Furquan Ansari
Bell Labs Research			Ph : 732-949-5249
Holmdel, NJ				Fax: 732-949-4556

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 15 Mar 2005 15:54:35 -0500

Responses in line, since there is much good context from Furquan.

At 03:42 PM 3/15/2005, Furquan A. Ansari wrote:
>...

Given that there are multiple FEs (and possibly even multiple NEs) sharing 
the interconnect, it is very hard to create the proper configuration for 
each FE to rate limit its data traffic.  In fact, if one assumes that new 
FEs (and possibly new NEs) can be added to the interconnect, it is almost 
impossible to properly set the rate limits.

> to be congestion aware as well, which is correct         - let me know 

Note that congestion management within routers is actually a very important 
issue.  Almost all routers carefully prioritize between routing control 
packets and "other" packets.  Routers with internal control exchanges 
prioritize those ahead of routing.  There is a new BCP coming out that 
talks about prioritizing within OSPF.  In our environment, overload of data 
packets will block the control channel.  That would be "really bad".  This 
is not really about DDoS.  There are perfectly normal things that can cause 
transient overloads of packets needing CE attention.

Hormuz raised the question of wether there are available DCCP stacks.  If 
there are, I think we should use it.  If there are not, I think we are in a 
very bad situation.  Ignoring congestion is not a good way out.

>...
>    - If there are bogus protocol packets sent, they will still be limited 
> to the rates allowed by the protocols themselves -
>    unless the software running on the neighbors has been compromised - in 
> which case you have much larger problems than congestion control on the 
> FE-CE link (you basically have compromised software on routers)

There are known cases where these protocols do not back off very well.

>hardware. Using TCP or UDP ( read DCCP) - as would  be the case if DCCP is 
>used - cannot achieve this. It will have to be taken out, processed, a PL 
>header added and finally a TCP or UDP header added and sent over the session.
>    If a tunnel is used between the FE and CE, the packets arriving on the 
> FE will be switched at wire speed with just

I believe this issue is one of implementation.  Some amount of driver 
modification is going to be required.  Some amount of packet processing of 
data packets is necessary (even if just to check that things are arriving 
over the proper interfaces.)  Yes, making use of a kernel DCCP stack and 
then shoving the result through TCP may be a bit tricky.  But there are 
ways to make it work.
If someone can show me how a GRE tunnelled packet will adapt properly to 
congestion and not displace the TCP traffic, maybe we could go with a GRE 
encapsulated frame instead of DCCP. (We would still have to deal with the 
problem of the frame being sent, with its encapsulation, being bigger than 
the interconnect medium size limit.)

>5) I am essentially saying that in some sense the TML header is the tunnel 
>protocol header. I really don't see
>for de-muxing to appropriate application (OSPF, BGP etc.) - Put it another 
>way, I am basically saying that there is no PL or TML header for the data 
>packets - since tunneling will be transparent to the applications.

The data packets need some sort of header since we must know what interface 
they arrived on, what FE sent them to the CE, and why the CE sent 
them.  That information can not reliably be determined purely from the 
underlying packet.

Yours,
Joel M. Halpern

---

From: "Furquan A. Ansari" <[email-masked]>
Date: Tue, 15 Mar 2005 16:56:14 -0500

Joel,
    Thanks for the comments. More comments inlime....

Joel M. Halpern wrote:

>> for data packets so that control packets will always have enough 
>> spare capacity to go through).
>
>
> Given that there are multiple FEs (and possibly even multiple NEs) 
> sharing the interconnect, it is very hard to create the proper 
> configuration for each FE to rate limit its data traffic.  In fact, if 
> one assumes that new FEs (and possibly new NEs) can be added to the 
> interconnect, it is almost impossible to properly set the rate limits.

[Furquan] Yes, coming up with a right solution for rate limiting when 
multiple FEs are sharing an interconnect would be
difficult, but not impossible - we will have to work on it.  Just as an 
example, every interface going out of the FE
towards the CE should be rate limited and the aggregation device should 
have the uplink capacity that is in fact
the aggregate value of all the FE-CE b/w links.

                                     CE
                                        |
                                        | B/w=(A+B+C)
                                        |
                    -------------                 
                    |                    |                |
             A   |              B   |             C|
                    |                    |                |  
                    FE1            FE2            FE3
                 
    B/W capacities between FE1-CE = A, FE2-CE = B, FE3-CE = C
    Rate limited Data channel for FE1 = 0.9A, FE2=0.9B, FE3=0.9C

    This will always ensure that there is enough capacity for the 
control packets between the CE and FEs. I'm
sure there are other mechanisms, but just wanted to point one possible 
solution.  

>
>> ---- list of things that I was discussing with other authors and one 
>> of our WG chair.
>>
>> exist in current architectures in single chassis systems - so why 
>> should it exist now.
>>        It is just as good or as bad as existing systems. So it is 
>> fully satisfying the requirements (and I don't think that the
>>        requirements RFC specifically mentions that the data channel 
>> has to be congestion aware as well, which is correct         - let me 
>> know if  I am mistaken here).
>
>
> Note that congestion management within routers is actually a very 
> important issue.  Almost all routers carefully prioritize between 
> routing control packets and "other" packets.  Routers with internal 
> control exchanges prioritize those ahead of routing.  There is a new 
> BCP coming out that talks about prioritizing within OSPF.  In our 
> environment, overload of data packets will block the control channel.  
> That would be "really bad".  This is not really about DDoS.  There are 
> perfectly normal things that can cause transient overloads of packets 
> needing CE attention.

[Furquan] No doubt that congestion is an important issue and  like I 
mentioned below, most protocols have inbuilt congestion control 
mechanims. You are absolutely right that there can be transient overload 
conditions that may cause congestion and I believe that rate limiting is 
sufficient for such cases - without having to resort to a congestion 
aware protocol. Certainly, one can maitain different queues in the 
router (FE) for control and data packets to realise priorities - with 
the control packets added to the higher priority queue.

>
> Hormuz raised the question of wether there are available DCCP stacks.  
> If there are, I think we should use it.  If there are not, I think we 
> are in a very bad situation.  Ignoring congestion is not a good way out.
>
>> ...
>> 3) All protocol traffic that is destined to the CE as "data" traffic 
>> is essentailly non-forces control traffic (e.g.
>> OSPF, BGP, ISIS, LDP, RSVP etc.). Every protocol has in-built 
>> congestion control. ICMP flooding may be an issue, but sending it 
>> over TCP/dccp/sctp  is not the solution.
>>    unless the software running on the neighbors has been compromised 
>> - in which case you have much larger problems than congestion control 
>> on the FE-CE link (you basically have compromised software on routers)
>
>
>> 4) Having TCP between FE and CE leads to severe performance 
>> degradation. Data packets on incoming FE
>> interfaces can be switched to gre (or any other tunnel) at wire speed 
>> in hardware. Using TCP or UDP ( read DCCP) - as would  be the case if 
>> DCCP is used - cannot achieve this. It will have to be taken out, 
>> processed, a PL header added and finally a TCP or UDP header added 
>> and sent over the session.
>> back into the kernel for terminating the inner transport session - be 
>> it TCP or UDP. It's just plain ugly!
>>    - We essentially have the case of TCP in TCP or TCP in UDP (DCCP) 
>> or TCP in SCTP etc. - all having the
> modification is going to be required.  Some amount of packet 
> processing of data packets is necessary (even if just to check that 
> things are arriving over the proper interfaces.)  Yes, making use of a 
> kernel DCCP stack and then shoving the result through TCP may be a bit 
> tricky.  But there are ways to make it work.
> If someone can show me how a GRE tunnelled packet will adapt properly 
> to congestion and not displace the TCP traffic, maybe we could go with 
> a GRE encapsulated frame instead of DCCP. (We would still have to deal 
> with the problem of the frame being sent, with its encapsulation, 
> being bigger than the interconnect medium size limit.)
>
[Furquan]  Please see the example above with the figure for a possible 
solution that can potentially ensure that
the GRE tunnelled packets will not displace TCP control traffic between 
the FE and the CE.
As for the the frame being larger than the packet, you are right  that 
it can potentially lead to IP fragmentation is
cases where the packet size may cross the MTU size of the network 
between the FE and CE. I believe that all
IP devices will "have" to necessarily handle the IP frag cases anyway - 
or they won't be fully IPv4 compliant.
Second, even TCP and DCCP will add additional headers on top of the 
packet received (in fact many more headers
than for the GRE tunnel header case). This will just as well lead to IP 
fragmentation. If the data channel is TCP,
one can argue that TCP negotiates MSS with the other end that may 
prevent IP frag to some extent - but it's no
guarantee. With UDP(DCCP) case, there is no such negotiation and will 
almost certainly undergo fragmentation
more often that the gre case.

Anyway, I don't think IP frag should be a show stopper.

>> the need for the "data" packets to be going through the ForCES PL 
>> layer for de-muxing to appropriate application (OSPF, BGP etc.) - Put 
>> it another way, I am basically saying that there is no PL or TML 
>> header for the data packets - since tunneling will be transparent to 
>> the applications.
>
>
> The data packets need some sort of header since we must know what 
> interface they arrived on, what FE sent them to the CE, and why the CE 
> sent them.  That information can not reliably be determined purely 
> from the underlying packet.

[Furquan] The tunnel provides this information since it's a 
point-to-point interface. It will give you information
regarding which physical interface it came on  - since the GRE tunnel 
was created by applying it to some specific
physical interface. Since it's p-to-p, you have information on who the 
other end is as well - i.e. which FE sent it.
Don't know why you would need the last case - but in rare situtations 
that this is needed, redirecting such data packets
over the control channel with the appropriate "redirect" PL header added 
would be the thing to do (Point 7 in my
previous email).

Thanks,

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 15 Mar 2005 17:03:12 -0500

I have a basic problem with this.  We are trying to design a congestion 
controlling datagram transfer protocol.  We are not the right place to do 
that.  Congestion control is known to be very tricky, and obvious answers 
often have very non-obvious implications.  (Just getting the delay bounded 
DiffServ stuff to work right took some interesting effort, and I had some 
very good folks to work with.)

it is possible that DCCP is too complex, but there is really no difference 
between an encapsulation driver (GRE) and a small protocol stack (DCCP I hope.)

I will endeavor in the near future to take a good look at the DCCP 
documents to see what they really give us and what it will take to use 
it.  (The last protocol I wanted to use did not have the capabilities I was 
looking for.)

If DCCP is not usable, we are in for a lot of very tricky work to get 
congestion control right, particularly since we need it to interact 
properly with TCP on the same link.

Yours,
Joel

At 04:56 PM 3/15/2005, Furquan A. Ansari wrote:
>    This will always ensure that there is enough capacity for the control 
> packets between the CE and FEs. I'm

---

From: "Furquan A. Ansari" <[email-masked]>
Date: Tue, 15 Mar 2005 17:35:06 -0500

Joel,
    You probably mistook my example as a design/need for a congestion 
controlling datagram transfer protocol - not true. I was just pointing 
out that the case of rate limiting the data channel is sufficient to 
achieve our objectives - that the control channel packets can be 
communicated during transient overload conditions. The case where you 
have individual interfaces from FE to CE, it is very trivial - clamp the 
high threshold to some percentage for data traffic (e.g. 90%). In the 
case where you end up sharing the interconnect with multiple FEs, 
exactly the same mechanism would suffice as long as the aggregated 
uplink capacity is greater than or equal to the sum of indivual 
capacities. If someone is foolish enough to have an uplink that is of 
same capacity as other (access) links - then you can potentially have 
congestion and the solution should be that he/she should upgrade the 
link. We don't need a specific congestion control "transport" protocol 
to achieve this.

E.g.
                                         CE
                                            |
                                            |    1G Ethernet
                                            |
                                        ---Eth Switch
                            ---  |        |   -----
                            |          ---                  |
                            |                |                    |
         10 Mbps  |                | 100Mbps |   100Mbps
                            |                |                    |
                         FE1         FE2           FE3

    You can add/remove more elements to the NE (aggregate up to 1G) and 
use the same policy of rate limiting on the new FE and it would work 
fine. No complexities here.

Thanks,

Joel M. Halpern wrote:

> I have a basic problem with this.  We are trying to design a 
> congestion controlling datagram transfer protocol.  We are not the 
> right place to do that.  Congestion control is known to be very 
> tricky, and obvious answers often have very non-obvious implications.  
> (Just getting the delay bounded DiffServ stuff to work right took some 
> interesting effort, and I had some very good folks to work with.)
>
> it is possible that DCCP is too complex, but there is really no 
> difference between an encapsulation driver (GRE) and a small protocol 
> stack (DCCP I hope.)
>
> it.  (The last protocol I wanted to use did not have the capabilities 
> I was looking for.)
>
>> configuration for each FE to rate limit its data traffic.  In fact, 
>> if one assumes that new FEs (and possibly new NEs) can be added to 
>> the interconnect, it is almost impossible to properly set the rate 
>> limits.
>>
>> difficult, but not impossible - we will have to work on it.  Just as 
>> an example, every interface going out of the FE
>> towards the CE should be rate limited and the aggregation device 
>> should have the uplink capacity that is in fact
>> sure there are other mechanisms, but just wanted to point one 
>> possible solution.
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 15 Mar 2005 17:58:45 -0500

Actually, rate limiting will. in general, not do what we want.
Suppose that we have several cards, each limitted to a certain rate (over 
time).
1) In order to support many cards, we would have to set that rate limit per 
card very low.  (Large numbers of FEs is a stated design target.)
2) If there is very little "data" traffic, TCP will have crept up very 
nicely.  If you then slam the link hard with the first couple of packets 
from many sources (such synchrony can and does happen in the real world) 
you will cause TCP too back off really hard.  It will take a long time to 
recover.

These are just two examples.  And I am not an expert on congestion control 
and the complexities thereof.  This is very tricky, very subtle, stuff.

Yours,
Joel M. Halpern

PS: Let me phrase this differently.  I would not want to be the chairs who 
had to explain to the IESG that we chose to invent a whole new congestion 
control mechanism.  It would be very unpleasant.

At 05:35 PM 3/15/2005, Furquan A. Ansari wrote:
> case where you end up sharing the interconnect with multiple FEs, exactly 
> the same mechanism would suffice as long as the aggregated uplink 
> capacity is greater than or equal to the sum of indivual capacities. If 
> someone is foolish enough to have an uplink that is of same capacity as 
> other (access) links - then you can potentially have congestion and the 
> solution should be that he/she should upgrade the link. We don't need a 
> specific congestion control "transport" protocol to achieve this.
>
>E.g.
>                                         CE
>                                            |
>                                            |    1G Ethernet
>                                            |
>                                        ---Eth Switch
>                            ---  |        |   -----
>                            |          ---                  |
>                            |                |                    |
>>often have very non-obvious implications.
>>it.  (The last protocol I wanted to use did not have the capabilities I 
>>was looking for.)
>>