Subject: [Pce] Question / issue PCEP port restriction - multiple PCEP adjacencies

From: Ramon Casellas <[email-masked]>
Date: Tue, 15 Feb 2011 09:55:20 +0100

Dear PCErs,

During some interoperability tests, we found an issue with the PCEP 
protocol and RFC5440 which manifests when using specific (although 
common) operating systems and we would appreciate your feedback to 
evaluate our options.

RFC5440 states (section 5)  that PCEP operates over TCP using a 
registered TCP port (4189) and all PCEP messages MUST be sent using the 
registered TCP port for the source and destination TCP port. Moreover, 
only one active connection between to given hosts/peers may be active at 
a time.
Although this is conceptually valid, I fail to see the reasoning behind 
the first restriction, and I see the second one as a direct consequence 
of the first.

The use case is as follows: assume a PCE host, with a PCEid address (in 
line with the "NodeID" "RouterID", or Loopback address, lo:0, defined as 
an IP address such that it is valid provided there is some connectivity, 
etc.)
The PCE creates a socket which is listening for TCP/PCEP connections 
("binding" to either the INADDR_ANY, 0.0.0.0 or restrict it to such 
PCEid address, but in any case to the port 4189).  At the same time it 
should also be able to have/initiate persistent TCP/PCEP connections to 
other peers (say, as a client), ideally keeping the same loopback 
address (NodeID) as the source IP address, while forcing the source port 
to what the RFC requires.

The main problem is that using that single PCEid address, it is not 
possible, or at least not possible in a portable way, to "bind" or 
associate several sockets to the same (local address, local port) pair. 
Of course,it should be theoretically possible, given that TCP 
connections would still be different considering the tuple (srcaddr, 
4189, dstaddr, 4189) but in practice it isn't:

After the first "listening" socket, the PCE would, for each persistent 
peer,
(i) create a socket SOCK_STREAM,
(ii) bind it to nodeid/4189 and
(iii) connect to remote peer/4189.

The first bind system call may fail depending on how the server is 
listening for connections (even worse, it some systems the bind system 
call may hijack the listening socket) or the second peer bind would fail 
since it would try to  bind to the same local address/port. For example, 
the GNU/Linux man page for bind is clear: ip(7) "Only one IP socket may 
be bound to any given local (address, port) pair". In technical terms, 
PCEP requires duplicate bindings at the socket level. Some systems may 
implement SO_REUSEPORT (1) socket option which could allow that, 
although it main use is to bind multicast addresses, or enable some sort 
of load sharing amongst TCP listeners in several threads. Unfortunately 
this is not implemented for example in GNU/Linux, although some 
non-official patches have been proposed (2). It is not clear whether it 
will be integrated or not.

Our workarounds seem to be either: allow clients to bind to port 0 
(ephemeral port) and use such allocated (dynamic) port (which is not 
compliant with RFC), or set IP aliases so each bind uses a different 
local address (we may need as many local addresses as peers, and some 
"mapping" to uniquely identify a PCE).

For what is worth, in my humble opinion the standard RFC is too strict 
requiring that all (client) sockets bind to port 4189

In summary we would appreciate your views on this:

a) what's the reasoning to force clients to bind on 4189 rather than say 
like in http? -- sometimes it is argued that this simplifies management 
such as firewall rules, etc. but I do not agree --

b) why limit to a single PCEP connection between two peers (other than a 
direct consequence of a) ), rather than allowing several parallel 
connections, provided that hosts use a single IP address and thus they 
can be identified?

c) As potential PCE implementers, have you encountered this issue, did 
you work around it? Are we missing something obvious?

Any feedback is appreciated

Best regards,

Ramon

(1) http://wiki.treck.com/Socket_Options

(1) http://marc.info/?l=linux-netdev&m=127165882728524&w=2 
<http://marc.info/?l=linux-netdev&m=127165882728524&w=2>

-- 
Ramon Casellas, Ph.D.
Research Associate - Optical Networking Area -- http://wikiona.cttc.es
CTTC - Centre Tecnològic de Telecomunicacions de Catalunya, PMT Ed B4
Av. Carl Friedrich Gauss 7 08860 Castelldefels (Barcelona) - Spain
Tel.: +34 93 645 29 16 -- Fax. +34 93 645 29 01

---

From: <[email-masked]>
Date: Tue, 15 Feb 2011 11:49:21 +0200

Just my input (worth about 2 cents).

Although not explicitly stated in RFC 5440, the PCEP protocol has copied
a lot of features from BGP (OPEN, KEEPALIVE, and CLOSE messages
being the most obvious).  BGP uses TCP port 179, but only when 
listening; the BGP peer taking the initiative to connect to another BGP
peer may use any BGP port number.  I agree, there is no obvious
reason for forcing a PCEP client to use port 4189 as source port.

Christian 
________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Ramon Casellas [[email-masked]]
Sent: Tuesday, February 15, 2011 10:55 AM
To: [email-masked]; [email-masked]
Subject: [Pce] Question / issue PCEP port restriction - multiple PCEP   adjacencies

Dear PCErs,

Best regards,

Ramon

_______________________________________________
Pce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pce

---

From: Julien Meuric <[email-masked]>
Date: Tue, 15 Feb 2011 11:06:34 +0100

Hi Ramon.

Thank you for sharing your implementation experience. This reminds me of 
an old thread on this list: 
http://www.ietf.org/mail-archive/web/pce/current/msg02096.html

That really looks like RFC 5440 is facing a lack in the implementation 
of the Linux OS. If compliance to the RFC requires to implement the 
patch you mention, as immediate solution I suggest to do so and to post 
any bug report/implementation feedback that may advocate including it 
for further system releases.

To all *implementers* of PCEP, a feedback on your own experience, 
associated to the OS you use, would be very appreciated. Experience over 
BSD and any Unix system would be much valuable. You may contact the 
chairs privately if you like.

Thanks,

Julien

Le 15/02/2011 09:55, Ramon Casellas a écrit :
> registered TCP port (4189) and all PCEP messages MUST be sent using 
> the registered TCP port for the source and destination TCP port. 
> Moreover, only one active connection between to given hosts/peers may 
> be active at a time.
> Although this is conceptually valid, I fail to see the reasoning 
> behind the first restriction, and I see the second one as a direct 
> consequence of the first.
>
> The use case is as follows: assume a PCE host, with a PCEid address 
> (in line with the "NodeID" "RouterID", or Loopback address, lo:0, 
> defined as an IP address such that it is valid provided there is some 
> connectivity, etc.)
> should also be able to have/initiate persistent TCP/PCEP connections 
> to other peers (say, as a client), ideally keeping the same loopback 
> address (NodeID) as the source IP address, while forcing the source 
> port to what the RFC requires.
>
> associate several sockets to the same (local address, local port) 
> pair. Of course,it should be theoretically possible, given that TCP 
> call may hijack the listening socket) or the second peer bind would 
> fail since it would try to  bind to the same local address/port. For 
> example, the GNU/Linux man page for bind is clear: ip(7) "Only one IP 
> socket may be bound to any given local (address, port) pair". In 
> technical terms, PCEP requires duplicate bindings at the socket level. 
> Some systems may implement SO_REUSEPORT (1) socket option which could 
> allow that, although it main use is to bind multicast addresses, or 
> enable some sort of load sharing amongst TCP listeners in several 
> threads. Unfortunately this is not implemented for example in 
> GNU/Linux, although some non-official patches have been proposed (2). 
> It is not clear whether it will be integrated or not.
>
> a) what's the reasoning to force clients to bind on 4189 rather than 
> say like in http? -- sometimes it is argued that this simplifies 
> management such as firewall rules, etc. but I do not agree --
>
> b) why limit to a single PCEP connection between two peers (other than 
> a direct consequence of a) ), rather than allowing several parallel

---

From: Ramon Casellas <[email-masked]>
Date: Tue, 15 Feb 2011 11:54:58 +0100

Julien,

Thank you for your mail. Please see inline

On 15/02/2011 11:06, Julien Meuric wrote:
> Hi Ramon.
>
> Thank you for sharing your implementation experience. This reminds me 
> of an old thread on this list: 
Indeed, the thread applies completely (apologies for duplicating the 
question). It seems to me that  Vishwas end suggestion was to raise an 
Errata ?

> patch you mention, as immediate solution I suggest to do so and to 
> post any bug report/implementation feedback that may advocate 
> including it for further system releases.
>
We will indeed test the patch, although I am not sure about this being 
the optimal solution, for several reasons:

- First, of course, from my point of view, what I would not want is the 
maintenance burden of patching successive kernel releases in the event 
that Linux kernel developers do not agree to implement that for the sole 
purposes of being able to conform to an RFC which, in someone's opinion, 
may be flawed or excessively restrictive. A partially satisfactory 
solution would indeed be that SO_REUSEPORT support was added, but that 
patch may be frowned upon as "not needed".

- Second, even if the implementation of SO_REUSEPORT enabled multiple 
initiators to bind to the same local address/port, other issues could 
appear in the sense that they would have to coexist with the listener.

However, that thread is still not convincing in the sense that "we do 
this because the RFC says so", still does not show the rationale. Worse, 
it seems to me that draft versions were indeed less restrictive and that 
"a posteriori" it was added in order to enforce (maybe without 
implementers feedback) the one-to-one restriction (which is, in its own, 
also debatable ;-) To date, all people I have humbly contacted in this 
matter see this as a non necessary requirement and some implementations 
end up allowing random ports for clients, in the spirit of most 
client/server TCP namely BGP, HTTP, etc.

Is an errata to RFC5440 potentially relaxing this out of the question?

> associated to the OS you use, would be very appreciated. Experience 
> over BSD and any Unix system would be much valuable. You may contact 
> the chairs privately if you like. 

Agreed, other feedback would be more appreciated, before further action :)

thanks and best regards

Ramon

---

From: [email-masked]
Date: Fri, 25 Feb 2011 10:05:31 +0800

Dear Ramon

Our team verified the standard implementation (RFC5440) in windows, linux 
and unix enviroment after we saw your email. Fortunately, the results 
indicate that RFC5440 is right. That is to say, there is no problem to 
bind two or more sockets in the same IP address and port number. 

I will send the detailed implementation to you and chairmen.

Thanks and best regards

Fei 

Ramon Casellas <[email-masked]> 
发件人:  [email-masked]
2011-02-15 18:54O

收件人
[email-masked]
抄送

主题
Re: [Pce] Question / issue PCEP port restriction - multiple PCEP 
adjacencies

Julien,

thanks and best regards

Ramon

---

From: Ramon Casellas <[email-masked]>
Date: Fri, 25 Feb 2011 07:02:15 +0100

----------------original message----------------- 

From: [email-masked] 

> 
>Dear Ramon 
>
>Our team verified the standard implementation (RFC5440) in windows, linux and unix 
>enviroment after we saw your email. Fortunately, the results indicate that RFC5440 is right. That 
>is to say, there is no problem to bind two or more sockets in the same IP address and port 
>number.
>
>Dear Fei,
>
>Thank you very much for your feedback and insight, that is indeed reassuring!.
>
>I guess we are still puzzled by Linux man page[1] ( "Only one IP socket may be bound to any 
>given local (address, port) pair"). We will check the details you provided, and get back to 
>you (privately) if any other issues. We will re-check the use of SO_REUSEADDR, which 
>seemed to be, according to the same url, to reuse sockets in a wait/close state. ("A TCP local 
>socket address that has been bound is unavailable for some time after closing, unless the 
>SO_REUSEADDR flag has been set. Care should be taken when using this flag as it makes TCP less 
>reliable")
>
>In any case, that is one less problem, thanks again
>
>Ramon
>
>
>[1] http://linux.die.net/man/7/ip
>
>
--

---

From: Ramon Casellas <[email-masked]>
Date: Fri, 25 Feb 2011 10:00:48 +0100

On 25/02/2011 3:05, [email-masked] wrote:
>
> Dear Ramon
>
> Our team verified the standard implementation (RFC5440) in windows, 
> linux and unix enviroment after we saw your email. Fortunately, the 
> results indicate that RFC5440 is right. That is to say, there is no 
> problem to bind two or more sockets in the same IP address and port 
> number.
>

Dear Fei,

Unfortunately, we are still having some issues in Linux, as discussed in 
the mail sent separately.

I would be more than happy to realize my (more than probable) mistake 
and stand corrected, but the use of SO_REUSEADDR does not seem to solve 
the problem (even relying in a non portable interpretation of 
SO_REUSEADDR / SO_REUSEPORT).

I was reluctant to post a sample code that reproduces the issue in the 
pce@ietf to illustrate the problem, so I apologize in advance. The net 
result (as detailed in previous mails) is that binding to the same local 
address after a server has started listening fails.

Things work if I remove either the client or the server part. Am I 
missing something?

Thanks in advance

Ramon

-------------------------------------

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <sys/ioctl.h>
#include <net/if.h>

int one = 1;
int portno = 4189;

void error(const char *msg)
{
         perror(msg);
         exit(EXIT_FAILURE);
}

void get_iface_ipv4(char *ifname, struct sockaddr *addr)
{
         struct ifreq ifr;
         int skfd = socket(AF_INET, SOCK_DGRAM, 0);
         strcpy(ifr.ifr_name, ifname);
         ioctl(skfd, SIOCGIFADDR, &ifr);
         *addr = ifr.ifr_addr;
         close(skfd);
}

int client (struct sockaddr_in* local_addr)
{
         int sockfd = socket(AF_INET, SOCK_STREAM, 0);
         if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &one, 
sizeof(one)) < 0)
                 error("SO_REUSEADDR client");
         if (bind(sockfd, (struct sockaddr *)local_addr, sizeof(struct 
sockaddr_in)) < 0)
                 error("bind client");
         return sockfd;
}

int main(int argc, char *argv[])
{
         struct sockaddr_in local_addr;
         memset(&local_addr, 0, sizeof(struct sockaddr_in));
         get_iface_ipv4("eth0", (struct sockaddr*) &local_addr);
         local_addr.sin_family = AF_INET;
         local_addr.sin_port = htons(portno);

         int sockfd = socket(AF_INET, SOCK_STREAM, 0);
         if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &one, 
sizeof(one)) < 0)
                 error("SO_REUSEADDR server");
         if (bind(sockfd, (struct sockaddr *) &local_addr, 
sizeof(local_addr)) < 0)
                 error("ERROR on binding");
         if (listen(sockfd, 10) <0)
                 error("ERROR on listen");

         int c1 = client (&local_addr);
         return 0;
}

---

From: Oscar González de Dios <[email-masked]>
Date: Fri, 25 Feb 2011 12:44:25 +0100

Hi Ramon, Fei,

        We have tested the PCE implementation in both Windows and Linux and we get different results. The implementation is made in java, and the SO_REUSADOR option is set in the sockets.

        Behavior in Linux:

                -> Test 1: Create and connect a client socket with both local and remote ports 4189, OK, Create and connect more client sockets with both local and remote ports 4189, OK,
                -> Test 2: create and bind server socket in 4189. Then create and connect a client socket with both local and remote ports 4189, fails!
                -> Test 3: create and connect a client socket with both local and remote ports 4189, ok. Then create and bind a server socket in 4189, OK!!

                Thus... seems in Linux the order matters....

        Behavior in Windows:

                -> Test 1: Create and connect a client socket with both local and remote ports 4189, OK, Create and connect more client sockets with both local and remote ports 4189, OK,
                -> Test 2: create and bind server socket in 4189. Then create and connect a client socket with both local and remote ports 4189, OK!!
                -> Test 3: create and connect a client socket with both local and remote ports 4189, ok. Then create and bind a server socket in 4189, OK!!

                Thus... seems Windows does not have any issues.

        In light of the results, RFC 5440 has no issues when connecting a client to several servers (test 1). However, as would be the case in the hierarchical PCE, a PCE that needs to act as both client and server (e.g. the child PCE), has issues in Linux, as once it is listening in 4189, no client sockets can be established (to connect, e.g. to the parent PCE). Then, for the hierarchical PCE case, RFC5440 enforcement of local port binding should be removed.

        Best Regards,

                Óscar

>-----Mensaje original-----
>De: [email-masked] [mailto:[email-masked]] En nombre de
>Ramon Casellas
>Enviado el: viernes, 25 de febrero de 2011 10:01
>Para: [email-masked]
>CC: [email-masked]
>Asunto: Re: [Pce] Question / issue PCEP port restriction - multiple PCEP
>adjacencies
>

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at.
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

---

From: Ramon Casellas <[email-masked]>
Date: Fri, 25 Feb 2011 13:01:32 +0100

Oscar, Fei all,

Please see inline

>                  ->  Test 2: create and bind server socket in 4189. Then create and connect a client socket with both local and remote ports 4189, fails!

Indeed this confirms our main case (as illustrated with the previous 
test code). Additionally, it does not seem to matter whether the server 
binds to a specific address or binds to a INADDR_ANY. As soon as the 
server binds to port 4189, the client cannot.

>          In light of the results, RFC 5440 has no issues when connecting a client to several servers (test 1). However, as would be the case in the hierarchical PCE, a PCE that needs to act as both…

As you may guess, it is not only a matter of H-PCE, but I'm afraid that 
(unless mistake on my part) we cannot be compliant with RFC 5440 the 
moment that we want to establish any adjacency with a peer,  BRPC, etc. 
Moreover, the natural order seems to be to bind on the listening port 
and if/when necessary, to connect to a peer.

Thanks for your feedback

Ramon

---

From: <[email-masked]>
Date: Fri, 25 Feb 2011 14:52:38 +0200

To answer the original question (why must the source port take a specific
value), the use of a specific source port was added in draft-ietf-pce-pcep-14,
the draft which later became RFC 5440.  The reason for the specific 
source port was discussed on the PCE mail list, see for example 
the mail sent Mon, 2 Jun 2008, 16:03:02 -04:00.  The argument for the
specific source port was, that the PCE working group could imagine
a scenario of only one PCC-to-PCE session.

Christian 
________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Ramon Casellas [[email-masked]]
Sent: Friday, February 25, 2011 2:01 PM
To: Oscar González de Dios
Cc: [email-masked]
Subject: Re: [Pce] Question / issue PCEP port restriction - multiple PCEP adjacencies

Oscar, Fei all,

Please see inline

Thanks for your feedback

Ramon

---

From: [email-masked]
Date: Mon, 28 Feb 2011 14:48:07 +0800

Ramon，Oscar，all

We verify the implementation in Linux again , just as Oscar said, 
different orders give different results.

Platform：Ubuntu 4.4.3-4ubuntu5 

The red region uses the old codes，It fails to bind client

However it will succeed if we justify the orders (connect to a peer 
firstly, then listen) .The result is shown in the first figure around by 
green square.

There is no problem with：FREEBSD 8.1(Successed!) and windows xp(sp3).

As you said, although we can keep alignment with RFC5440 when establishing 
the TCP connection by adjusting the sequence, the natural order seems to 
be to bind on the listening port and if/when necessary, to connect to a 
peer.

But in the scenarios of the BRPC or H-PCE, some PCEs play the role of 
clients and servers, and it is not effective to adjust the implementing 
orders. See the details below:
Windows platform：（linux、freebsd platforms have the same issues）
Step1: Create a TCP server (named server1) by starting to listen on 
192.168.0.1:4189 and waiting for connection. As shown in the left above 
figure, it’s successful.
Step2: Create a TCP server (named server2) by starting to listen on 
192.168.0.2:4189 and waiting for connection. As shown in the left down 
figure, it’s successful.
Step3: Create a TCP client (named client2) by binding on 192.168.0.2:4189 
and trying to connect the peer server1.As shown in the right down figure, 
it’s successful.
Step4: Create a TCP client (named client1) by binding on 192.168.0.1:4189 
and trying to connect the peer server2.As shown in the right down figure, 
it’s failed whose error is 10048 meaning “address already in use”.

I cannot be certain whether it’s my code fault or it’s the fact. So I 
deeply wish you would try to process the steps as above and certify my 
results. 

Just my two cents, although it MUST be careful to change the description 
of standards, it seems that we SHOULD do so to cover all kinds of 
scenarios and for easy implementation if it is necessay.

Thanks and best regards

Fei and Xuping

2011-02-25 20:01

收件人
Oscar González de Dios <[email-masked]>
抄送
"[email-masked]" <[email-masked]>, "[email-masked]" 
<[email-masked]>
主题
Re: [Pce] Question / issue PCEP port restriction - multiple PCEP 
adjacencies

Oscar, Fei all,

Please see inline

On 25/02/2011 12:44, Oscar González de Dios wrote:

>          We have tested the PCE implementation in both Windows and Linux 
and we get different results. The implementation is made in java, and the 
SO_REUSADOR option is set in the sockets.
>          Behavior in Linux:
>
>                  ->  Test 2: create and bind server socket in 4189. Then 
create and connect a client socket with both local and remote ports 4189, 
fails!

>          In light of the results, RFC 5440 has no issues when connecting 
a client to several servers (test 1). However, as would be the case in the 
hierarchical PCE, a PCE that needs to act as both client and server (e.g. 
the child PCE), has issues in Linux, as once it is listening in 4189, no 
client sockets can be established (to connect, e.g. to the parent PCE). 
Then, for the hierarchical PCE case, RFC5440 enforcement of local port 
binding should be removed.

Thanks for your feedback

Ramon

---

From: Ramon Casellas <[email-masked]>
Date: Mon, 28 Feb 2011 08:10:55 +0100

Fei, All,

See my comments inline

> different orders give different results.
As a summary: even with the use of socket options, one cannot bind a 
client socket after binding a listening socket, portably.

In practice this seems to affect GNU/Linux in particular and windows / 
*BSD seem to work ok, but the fact that e.g. java programs behave 
differently on different platforms is not good.

> Just my two cents, although it MUST be careful to change the 
> description of standards, it seems that we SHOULD do so to cover all 
> kinds of scenarios and for easy implementation if it is necessay.

I tend to agree, this is not something trivial, but we can raise the 
issue in Prague. Although it is an important change, I would say that 
old implementations that bind source ports should work without change.

It does affect implementations that enforce that the (remote) source 
port is 4189.

According to Christian's mail and references within, the rationale for 
this does not seem to apply anymore and the use cases have 
changed/evolved (i.e. there are use cases with 2 ore more PCC 
'processes' within a node)

There are two potential changes: a) do not restrict source port and b) 
allow for parallel connections between nodes, the latter no longer being 
restricted by the former.

IMHO a) should be discussed, b) is a more complex change and the 
implications may be different. A given implementation may 
(nonconforming-ly) allow different connections from the same IP address 
(e.g. NAT?)

All comments welcome.

Thank you

R.