Subject: [BEHAVE] REQ 1 and REQ 7 of RFC5382 were supposed to be fixed years ago

From: ivan c <[email-masked]>
Date: Mon, 17 Jun 2013 15:15:40 +0200

Speaking of this, reading draft-penno-behave-rfc4787-5382-5508-bis
reminded me of a conversation I had many years ago with the Editor of
RFC5382. 

Back then, it was acknowledged that the most desirable behavior
for a NAT to support TCP simultaneous open between 2 peers behind NATs was
to have TCP port preservation. REQ 1 is a weaker condition that can be used
by NAT that do not implement port preservation, but can use a third party
server to perform port prediction. The Editor said at the time that this
would be updated in the next version of RFC5382, but I actually haven't
heard of him since then as he seems to have vanished off the group a long
time ago. 

REQ 7 was supposed to be fixed too, as the condition it
requires is way too strong as everyone can see. Port overloading for TCP is
perfectly acceptable when the remote endpoints are distinct. With large
scale deployments of CGNs, it has now become a desirable behavior.

Another point, as far as I'm aware, no peer to peer application
implements the SO_REUSEADDR hack that RFC5382 advocates to perform TCP
simultaneous open. Again, TCP port preservation is generally used as there
is more support for this scheme in NAT deployments in the wild. Use of the
SO_REUSEADDR hack violates RFC793 and should be used with extra care. 

If
NAT implementers are really reading this RFC as it is, this is going to be
the source of much confusion. 

I'm not sure whether the correct thing to
do would be to write an Errata for these points or simply to write a new
RFC based on the new work of draft-penno-behave-rfc4787-5382-5508-bis .

Please don't hesitate to ask me for technical clarifications, as these
discussions happened many years ago and some of the context might have been
lost by the current participants. 

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Mon, 17 Jun 2013 16:54:43 +0200

Le 2013-06-17 15:15, ivan c a écrit :
> Back then, it was acknowledged that the most desirable behavior for a
> NAT to support TCP simultaneous open between 2 peers behind NATs was to
> have TCP port preservation.

I would argue that port preservation is useless for traversal.

> REQ 1 is a weaker condition that can be used
> by NAT that do not implement port preservation, but can use a third
> party server to perform port prediction.

I don't see how EIM can be used for port prediction.

> REQ 7 was supposed to be fixed too, as the condition it requires is way
> too strong as everyone can see. Port overloading for TCP is perfectly
> acceptable when the remote endpoints are distinct.

In your opinion, what is it about TCP that makes EDM OK?

> Use of the SO_REUSEADDR hack violates RFC793 and should be used with
> extra care.

Please explain.

Simon

---

From: ivan c <[email-masked]>
Date: Tue, 18 Jun 2013 04:21:55 +0200

Hi Simon,

You will find my explanations to your 4 questions interleaved below.

First of all, a word about the context considered in this discussion: 
We have peers, all behind NATs, attempting to connect to each other via
TCP simultaneous open (TCP simultaneous open being described in TCP RFC793
Section 3.4 Figure 8).
The first step is for these peers to discover or guess what each other's
remote external endpoint will be for the TCP connection. This step is
called
"remote external endpoint prediction" or informally "port prediction" (as
the hard part is to predict the external port, as created by the NAT).
Once
external mapping have been negotiated/discovered, this information is
shared amongst the peers over some arbitrary out-of-band communication
channel. 
The second step consists in performing the actual TCP simultaneous open
to the relevant endpoint.

These steps constitute the TCP Hole Punching process, it can be used for
NAT Traversal.

> I would argue that port preservation is useless for traversal.

TCP port preservation in NAT mappings allows peers to perform port
prediction trivially: the external port value of their own mapping is
the same as the local/internal port. The information of the external
endpoint can then be shared
amongst all peers over an arbitrary out-of-band 
channel. This completes the port prediction step of TCP Hole Punching.

The alternative to port preservation for the purpose of port prediction
is to use an EIM-NAT, and use
SO_REUSEADDR set on all local sockets, and use a third-party public
server. I explained this process as the
answer to your second question below. However, such technique uses
POSIX-unspecified behavior (SO_REUSEADDR is left unspecified) and is TCP
non-compliant (as it violates the TIME_WAIT state).

As a result, port preservation is desirable (and implies EIM anyway) for
TCP NAT traversal and adopted in the vast majority of NAT
implementations (in Saikat Gupta's paper, he mentions 89% of EIM NATs).
Port preservation becomes absolutely necessary in the context of fully
decentralized p2p networks. In such networks, third-party STUNT public
servers
are generally not available, and thus a TCP Hole Punching
technique using a STUNT server is not possible.
Port Preservation alleviates the need
of a third-party STUNT server for the purpose of
performing port prediction.

See Saikat Guha (Editor of existing RFCs) paper
http://saikat.guha.cc/pub/imc05-tcpnat.pdf, section 5.

> I don't see how EIM can be used for port prediction.

The single purpose of a EIM NAT for TCP is to allow an easy port
prediction with a STUNT server.

First of all, we use the term "port prediction" as a synonym for any
remote endpoint port discovery or guessing mechanism. When 2 peers behind
NATs
attempt to communicate with each other through TCP, they need to predict
each
other's external endpoint to perform a TCP simultaneous open.

An EIM NAT guarantees that two successive outgoing TCP connections from
the same internal endpoint are mapped to the same external endpoint on
the NAT.
This is port prediction: the first connection assists in predicting the
external endpoint that will be used by the NAT for the second connection.

Such behavior can be used for TCP Hole Punching as follows:
Let S be a server (no NAT) with a listening port Ps, let A and B be peers
behind NATs.
Port prediction works as follows:
Peer A binds a local socket on port Pa, then makes a TCP connection with
this socket to S on port Ps. Peer B does the same.
S gives the information of their respective port mapping to both peers.
Port Prediction has now been achieved: because the NAT uses
Endpoint-Independent Mapping, any subsequent connection from the same
internal endpoint will reuse the mapping.

The rest of the TCP Hole Punching process will then go as follows:
Peer A closes its local socket. Peer B does the same.
Peer A creates a new socket and binds it on the *same* port Pa. Most OS
stacks allow this as long as the SO_REUSEADDR option is set on *both*
sockets. Peer B does the same.
Then both make a connect() on this socket, to each respective endpoint.
TCP
simultaneous open is now in progress.

This is also explained in Saikat Guha's paper
http://saikat.guha.cc/pub/imc05-tcpnat.pdf, in section 5.

> In your opinion, what is it about TCP that makes EDM OK?

My point is about the "TCP Port overloading" behavior and showing that
REQ 7 is too stringent.
This is orthogonal to Endpoint-Dependent Mapping (EDM) behavior.
Please explain your question.

>> Use of the SO_REUSEADDR hack violates RFC793 and should be used with

The SO_REUSEADDR socket option was added exclusively to bypass the TCP
TIME_WAIT state for listening servers. (which is an annoyance in
development)
The TCP TIME_WAIT state is there to avoid the old duplicate problem. The
TIME_WAIT state lasts for 2*MSL, which is about 4 minutes.
When SO_REUSEADDR is used on sockets, an application can bind a socket
on a port that has a TIME_WAIT state pending. Bypassing the TIME_WAIT
state violates the data corruption provisions of TCP, and should only be
used by servers, in non-production
environments.
Furthermore, the behavior with
SO_REUSEADDR is intentionally left unspecified in POSIX: "SO_REUSEADDR:
Specifies that the rules
used in validating addresses supplied to bind() should allow reuse of
local addresses".

> 
> Simon
> _______________________________________________
> Behave mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/behave

Please do not hesitate to ask me for further clarification about the
points
above.

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Tue, 18 Jun 2013 10:05:17 +0200

Ivan,

Thanks for the explanation. I believe I have a good understanding of 
what you are proposing now.

There are several smaller technical issues with your reasoning which I 
won't address now because they are tangential. I want to focus on your 
suggestion that we should recommend port preservation. I see the 
following major issues:

- Adding new constraints on NAT implementations is not going to meet 
much success. I don't believe anyone is going to change their existing 
NAT code now in such a fundamental way as to implement port preservation 
just because the IETF asks for it in a new RFC.

- Port preservation is not applicable to CGN, where a subscriber often 
only has access to a limited range of external ports.

- There are ways to improve the scalability of EIM without killing it. 
I've been advocating for some time that NATs should be allowed to use 
EDM for protocols that they know will not break, with EIM as a default. 
For example, using EDM for TCP port 80 and UDP port 53 is easy, 
harmless, and has a big impact.

Simon

---

From: "Senthil Sivakumar (ssenthil)" <[email-masked]>
Date: Tue, 18 Jun 2013 16:16:33 +0000

>just because the IETF asks for it in a new RFC.

I just want to add that I did a port preservation implementation a while
ago, but realized that the number of times that the ports couldn¹t be
preserved were getting more and more. Even though this wasn't causing any
application behavior issues, the customers were complaining because they
construed that as NAT not working properly, (whether their assumption that
is right or wrong is a different discussion), so we ended up not using the
port preservation as a default behavior in later implementations. It also
improves the performance.

>only has access to a limited range of external ports.

Exactly, with the allocation of port sets in CGN, it becomes very
difficult to do any kind of port preservation.

>harmless, and has a big impact.

For allowing EDM, one has to know their applications they are using, but
as you say port 80 can work well with EDM.

---

From: ivan c <[email-masked]>
Date: Tue, 18 Jun 2013 18:57:07 +0200

Hi Simon,

See my comments interleaved.

> just because the IETF asks for it in a new RFC.

(First, to avoid any confusion, it is important to use the term "TCP port
preservation" instead of just "port preservation", as this only applies to
TCP)
In reality, TCP port preservation has already implemented by most home
NATs at least.
As already noted in Saikat Guha's paper back in 2005, almost all EIM NATs
in his sample set were already implementing TCP port preservation.
Nowadays, this is even more pervasive: Linksys, Netgear, and many others;
in a country like France, this is implemented by most ISP-made internet
triple-play gateways (called "Boxes" there) including the ones of Orange,
Free, SFR, Numericable, offering a near 100% coverage of the country.
The goal is to standardize what has already been widely deployed
worldwide, following the IETF mantra, "running code wins".
The alternatives for applications when a NAT does not have TCP port
preservation are not appealing: STUNT when the NAT is EIM, otherwise UPnP,
port forwarding, and, when all options have been exhausted, fallback on
the
worst of the worst: TCP over UDP (extensively used by bittorrent), which
is
dangerous for the internet as a whole.

> only has access to a limited range of external ports.

It would be nice to elaborate on this. 
A CGN should leave the whole ephemeral ports range to all its subscribers,
which is defined by the IANA to be [2^15+2^14;  2^16-1]. Combined with TCP
port overloading, this allows usage of TCP port preservation by CGNs.
Rationale of why port overloading is acceptable for TCP:
- as long as endpoints are distincts, port overloading preserves the
uniqueness of the 5-tuple.
- when 2 endpoints are the same, the NAT refuses, silently or actively,
the second TCP connection. The application can then retry with a new
ephemeral port. If p is the probability of conflict, and n the number of
tries, the probability of failure decreases exponentially in p^n, which
gives an essentially deterministic behavior. As a collision is expected to
be a rare event, the probability p would be small anyway.

> harmless, and has a big impact.

Agreed, although some extra care is needed, as explained below, the
benefits clearly outweigh the complexity of the implementation.

As a preliminary remark, the use case that you mention of DNS request is
out of scope because each DNS request uses a new socket (non-bound, so it
uses a new local endpoint). As a result this doesn't reuse an existing NAT
mapping. Anyway, EDM should not be used as a fallback for UDP, for reasons
i'll skip for now, as I'm assuming the contentious point for CGNs is TCP,
not UDP.

First of all, the context for TCP: fallback on EDM is somewhat desirable
when there is a conflicting mapping. The alternative would be to actively
or silently refuse the outgoing TCP SYN, which is not desirable as it
somewhat breaks end-to-end connectivity guarantees. (although browsers for
instance are known to retry connections)
A NAT can fallback on EDM when it is clearly assumed that the application
issuing the TCP SYN is not P2P. This is often the case for HTTP, and can
be
used as a heuristic by the NAT/CGN.

However, these heuristics might be a lot more complicated that what we may
have anticipated, for the following non-exhaustive list of reasons:
- although HTTP is usually used with active/passive TCP socket pairs
(client/server), nothing prevents it from being used in a P2P context with
TCP simultaneous open. This is already being seen in the wild.
- port numbers do not give any information about the upper-layer protocol,
at most they can provide some heuristics. We know that P2P applications
*love* to impersonate well-known ports to bypass firewall policies. For
example, Skype *always* binds on 80 or 443 if it sees you have UPnP
activated in your gateway
- DPI doesn't prove to be a much help either for our heuristics: for
instance, Skype simulate an HTTP or SSL handshake on ports 80 or 443. The
reason, again, is to bypass intermediate firewall policies.

On the other hand, in the following use case, EDM is perfectly acceptable:
a TCP SYN is sent by a client behind a NAT, and the destination appears to
be a well-known HTTP server belonging to Google, Facebook, etc.
As this use case represents a sizeable amount of the current internet
traffic, I would definitely mention this in an RFC. A CGN would simply
need
to carry a static, updatable list of ip addresses ranges of well-known
popular HTTP servers, to allow fallback on EDM when desired.

-- 
_Ivan Chollet_

---

From: ivan c <[email-masked]>
Date: Tue, 18 Jun 2013 19:10:56 +0200

Hi Senthil,

See my comments below.

> improves the performance.

Are you talking about UDP port preservation? I think we all agree UDP port
preservation is not necessary, as long as the NAT is EIM for UDP.
About TCP port preservation, most NAT implementations have it. In some
countries (like France), all ISP-provided gateways have it.
As you note, TCP port preservation improves latency and thus performance,
when compared to the alternative of using a STUNT server.
It does not need the intermediate step of contacting the STUNT server to
perform TCP port prediction.

> difficult to do any kind of port preservation.

Nope, you would need to elaborate on this.
The probability that of an internal port collision is high because of the
birthday paradox, but port overloading is perfectly acceptable, and when a
full collision occurs (when internal ports and remote endpoints are the
same for two outgoing TCP connections), which is supposed to be a rare
event, the CGN can fallback on EDM or simply drop the connection.

> as you say port 80 can work well with EDM.

Exactly, see my message to Simon for a more in-depth discussion about this
topic.

-- 
_Ivan Chollet_

---

From: "Senthil Sivakumar (ssenthil)" <[email-masked]>
Date: Tue, 18 Jun 2013 19:24:50 +0000

Hi Ivan,

>perform TCP port prediction.

Both TCP & UDP. The latest implementation in some router families is not
to have port preservation
(for both TCP & UDP).

>event, the CGN can fallback on EDM or simply drop the connection.

Most of the NATs that I know don’t do port overloading any more.
RFC 5382 also says,
REQ-7:  A NAT MUST NOT have a "Port assignment" behavior of "Port
      overloading" for TCP.

Thanks
Senthil

---

From: ivan c <[email-masked]>
Date: Tue, 18 Jun 2013 22:33:22 +0200

Hi Senthil,

Thanks for your message. You might want to read
http://tools.ietf.org/html/draft-ietf-behave-requirements-update-00 
It features the list of updates to the existing RFC, adapted to new usage
contexts such as CGNs, mobile networks, etc.

On Tue, 18 Jun 2013 19:24:50 +0000, "Senthil Sivakumar (ssenthil)"

>>
>>Are you talking about UDP port preservation? 
> 

The discussion here is not about UDP. UDP port preservation should
generally not be implemented by NATs, as it could generate conflicts when 2
internal hosts using the same local port, each with a session to the same
endpoint. This would break end-to-end connectivity in rare cases, as there
is no fallback mechanism (as opposed to TCP).

To be clear, the requirements are as follows:
* NAT SHOULD NOT use UDP port preservation
* NAT SHOULD use TCP port preservation

Here are some NATs that have been tested and implement TCP port
preservation:
in the UK:
BT (British Telecom) (house-made gateway) - number 1 provider
Virgin (house-made gateway) - number 2 provider
Sky (Sagemcom) - number 3 provider

in France:
Orange (Livebox) (house-made gateway) - number 1 provider
Free (Freebox) (house-made gateway) - number 2 provider
SFR/Neuf (Neufbox) (house-made gateway) - number 3 provider

This covers the overwhelming majority of subscribers.

We have the same results in Spain in Italy.

Misc:
Linksys E1200, latest generation.
Netgear, various models

It is difficult to find any NAT that does *not* implement TCP port
preservation in these countries.
If you think some gateway implementations have dropped TCP port
preservation, you need to substantially support your claim by a reference.
We are not aware of anything that would support it. 

> REQ-7:  A NAT MUST NOT have a "Port assignment" behavior of "Port

This is the purpose of the new draft
http://tools.ietf.org/html/draft-ietf-behave-requirements-update-00 , to
make the use case for port overloading, as it is needed in CGNs. It wasn't
needed nearly as much in 2005 when RFC 5382 was first written.

REQ-7: A NAT MUST NOT have a "Port assignment" behavior of "Port
overloading" for TCP
is too stringent, as proved in my earlier message and in the
draft-ietf-behave-requirements-update-00, and not needed.

Port overloading is perfectly acceptable when the remote endpoints are
distinct, as it preserves the uniqueness of the 5-tuple.

As a result, REQ-7 must be corrected.

-- 
_Ivan Chollet_

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 18 Jun 2013 20:57:20 +0000

It seems worthwhile to differentiate home router NAT implementations from enterprise/SP router NAT implementations. And the question to ask ourselves is whether we need a common set of requirements for both scenarios !!

Cheers,
Rajiv

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of ivan c
> Sent: Tuesday, June 18, 2013 4:33 PM
> To: Senthil Sivakumar (ssenthil)
> Cc: Behave
> Subject: Re: [BEHAVE] (no subject)
> 
> Hi Senthil,
>

---

From: ivan c <[email-masked]>
Date: Wed, 19 Jun 2013 00:05:04 +0200

On Tue, 18 Jun 2013 20:57:20 +0000, "Rajiv Asati (rajiva)"
<[email-masked]> wrote:
> It seems worthwhile to differentiate home router NAT implementations
from
> enterprise/SP router NAT implementations. And the question to ask
ourselves
> is whether we need a common set of requirements for both scenarios !!
> 
> Cheers,
> Rajiv
> 
> 

Indeed, hopefully our new work in draft-ietf-behave-requirements-update-00
will help defining in what extent the requirements for both use cases (CGN
versus home gateways) might diverge.
We need to make each requirement's justification very clear and detailed,
so implementers can make informed decisions based on the intended usage.

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Wed, 19 Jun 2013 10:13:14 +0200

> is no fallback mechanism (as opposed to TCP).

Please explain how TCP is not subject to the same problem.

Simon

---

From: "Senthil Sivakumar (ssenthil)" <[email-masked]>
Date: Wed, 19 Jun 2013 14:38:12 +0000

Hi Ivan,

>contexts such as CGNs, mobile networks, etc.

I have reviewed this draft and we had a bit of discussion around these
topics in the last IETF.
>This covers the overwhelming majority of subscribers.

Ok thanks for the pointers. The NATs that I was talking about were
enterprise/SP NAT boxes,
not the home routers. Many of the CGN implementations don¹t do port
preservation. 

>> Most of the NATs that I know don¹t do port overloading any more.
>> RFC 5382 also says,

There are two distinct things here:
1. Port preservation is a must
2. Port preservation requires port overloading to work deterministically.

I don¹t have a lot of problem with an implementation choosing to do port
preservation if it can.
I do have issues with doing port overloading. Two reasons again.
1. Port overloading requires the destination information to be tracked by
NAT devices. That kills scalability.
2. Port overloading causes packets to be dropped if two connections going
to the same destination, causing indeterministic behavior.
   
Maybe as Rajiv suggested in another thread, maybe we need a CPE NAT
requirements document where the above requirement might be placed.
But I don¹t know what good it is, if there is another CGN on the headend
that would not honor the port preservation. As I said, in my earlier
Email, the CGNs can now have a pre-allocated set of ports (like 1k
ports/sub), where there is no way to do port preservation.

Senthil

---

From: Dan Wing <[email-masked]>
Date: Wed, 19 Jun 2013 20:21:50 -0700

> preservation, you need to substantially support your claim by a reference.

It is impossible to preserve TCP ports on a large or busy NAT (or a NAT that is both large and busy) because multiple clients will simultaneously use the same source port, especially if the clients are using the same operating system because most operating systems unfortunately do not bother to randomize their source ports.

It is also impossible to preserve TCP ports with A+P-related IPv4 address sharing mechanisms.  A+P was first described in RFC6346, and in the MAP-E and MAP-T specifications in Softwires (draft-ietf-softwire-map).  There are 2-3 similar Internet Drafts which have a static assignment of ports to subscribers.

-d

---

From: Dan Wing <[email-masked]>
Date: Wed, 19 Jun 2013 20:25:14 -0700

> ports/sub), where there is no way to do port preservation.

The impact to applications is the same, no matter if the NAT function is within the user's home or in the service provider's network.  With some of the A+P techniques, the ISP controls the in-home CPE behavior, assigning it a limited number of ports.  Thus the difference between a "carrier" NAT and the "consumer" NAT is far too blurred to make a useful distinction between port overloading behavior.

This discussion should continue, even if to repeat the arguments that were made years ago when the port overloading text was the WG's consensus.  If WG consensus has changed, let's change the updated document.  But we do need a careful analysis and discussion of the impact of such a change to applications and to NATs.

-d

---

From: ivan c <[email-masked]>
Date: Thu, 20 Jun 2013 20:27:03 +0200

> Please explain how TCP is not subject to the same problem.

This question is really about Port Overloading and is the crux of the
discussion, so i'm giving a short answer and a longer, more detailed
answer.
I voluntarily leave the long answer on this mailing-list, so it can
provide raw material for the draft later on.

THE SHORT ANSWER:
Port overloading (UDP or TCP) introduces the small possibility of a
non-deterministic behavior of the NAT: the fallback on EDM in the event of
a 5-tuple conflict.
It's a rare event, and only affects p2p applications, who are already
prepared to deal with it. After all, a large part of NATs in the wild
aren't cooperative, so p2p applications always have fallback mechanisms in
place. 
Beacause of the potential benefits of port overloading for CGNs, this
should be a no brainer.
In short, port overloading is fine for both UDP and TCP and won't break
any p2p application.
That been said, the use case for NAT port overloading was always only TCP,
as applications do not need to create that many local endpoints for UDP. So
let's not even bother with the corner cases for UDP for simplicity sake and
tell the NAT not do port overloading for UDP.
Conclusion: 
- UDP port overloading is not particularly useful, Req 3 of RFC 4787 is
fine.
- TCP port overloading can be very useful, RFC 5382 should mention support
for it.

THE LONGER ANSWER:
There is one major difference between UDP and TCP from the application
point of view:

(1) one UDP socket (local endpoint) can have multiple communication
sessions. (with multiple remote endpoints)
(2) one TCP socket can have only one session.

Condition (2) is not implied by RFC 793, but is enforced by POSIX. This
originates from the fact that Unix had a special recv() function for
connected socket, as opposed to recvfrom() for general sockets. POSIX later
standardized on this Unix behavior. This Unix idiosyncrasy prevents
sessions multiplexing for connected sockets.

Consequence: 
Applications don't need to use more than one UDP local endpoint, but need
to use many TCP local endpoints.

The impact on applications behavior:
For UDP, a p2p application will tend to multiplex all its p2p sessions
over one UDP socket (actually, some use 2 or 3 sockets for convenience).
This is what you observe with Skype, uTorrent, etc. but is applicable to
virtually all applications: it saves scarce resources (ports) and is
considered good design.
Unfortunately, since POSIX doesn't support the same thing for TCP, and a
large number of TCP sockets will be created by the application, on many
local endpoints, as you have witnessed for most "torrent-like"
applications.

Consequence (1):
The use case for port overloading should be TCP, not UDP.

Back to NAT port overloading. A collision occurs when 2 sessions share the
same 5-tuple. It is usually a rare event. Let's look at the corner case of
when a collision occurs, with p being the probability of a collision:
- NAT fallbacks on EDM for this particular session (note:
non-deterministic behavior from the application pov, with probability p)
- two possible outcomes for the application:
  1) session initiation succeeds, OK.
  2) session initiation fails, application fallbacks on the following:
     * create a new socket (with a new local endpoint) and restart.
Probability of success increases exponentially in (1-p^n) at each re-try.
     * use another fallback mechanism (UPnP, relaying, etc.), but this is
out of scope and not part of the TCP Hole Punching protocol.

Note 1: The session usually often succeeds when NAT fallbacks to EDM. It
only fails in the case where the remote endpoint is also behind a NAT.

Note 2: Here, the slight probability of failure for a connection attempt
is corrected by the TCP Hole Punching protocol as a whole.
(Analogy with TCP: the non-deterministic behavior introduced by packet
loss is abstracted away by the TCP layer, by resending the lost packets)

The failure case when the session fails is dealt with by the more
complicated code path in 2). Here the application creates a new socket.
What it implies, in both UDP and TCP cases:
UDP: creating a new socket for UDP is ok, but an annoyance. Goes against
good design practices.
TCP: creating a new socket is ok, this is what the application is doing
anyway for each new session.

Conclusion:
UDP: there is not much of a case for port overloading, and the more
complicated behavior path could possibly go against application good design
practices.
TCP: port overloading fits well with the TCP Hole Punching protocol.

In my opinion, although we can leave the requirements for UDP as they are
for the sake of simplicity, we need to write about the possibility of port
overloading for TCP in the documents.

-- 
_Ivan Chollet_

---

From: ivan c <[email-masked]>
Date: Thu, 20 Jun 2013 21:20:55 +0200

> not the home routers. Many of the CGN implementations don¹t do port

Yes, CGNs are not expected to be p2p-friendly at this stage, and I don't
think having any MUST requirements to help with TCP Hole Punching would do
any good. EIM for TCP can probably be safely ignored too as there is no
real use case, however if there is one thing they should try to implement
that's RFC 4787 for UDP. 
Having said that, I would support mentioning TCP Hole Punching and its
subvariants in the document and what a NAT should do to support them as a
recommendation.

> 2. Port preservation requires port overloading to work
deterministically.

Not exactly, port preservation can *never* work deterministically.
Consider the rare case where the two remote endpoints are the same too.
This breaks port preservation.
Not that it matters anyway, what we want is the interface provided by the
TCP Hole Punching protocol to be deterministic. The NAT falling back on EDM
in case of a collision is a case that is taken care deterministically by
the TCP Hole Punching protocol. (See my email in reply to Simon a few
minutes ago)
This is what home NATs do: TCP port preservation (which implies EIM for
TCP, so makes it easy for them), and in case of a source collision,
fallback on EDM.

> to the same destination, causing indeterministic behavior.

(my email to Simon is more detailed regarding TCP port overloading and the
relationship with TCP Hole Punching)
I restrict the discussion to port overloading for TCP.
1. It kills scalability when logging is required (but in that case a CGN
would probably use static port ranges to segregate subscribers).
As far as session tracking is concerned, port overloading only makes the
key of the NAT's mapping hashtable twice as big. Not a concern for
scalability.
2. That's a bit extreme, the NAT doesn't need to drop packets, it just
switches to EDM. This only adversely affects TCP Hole Punching, which is
prepared to deal with the case in its protocol.

> ports/sub), where there is no way to do port preservation.

At this stage, CGNs are not very keen on EIM for TCP, and TCP port
preservation is a stronger condition, so it looks like a lost battle to me.
And in my opinion, if CGNs could at least do EIM for UDP, it would be a
good start indeed. That would have a higher priority.

There is one big difference between home NATs and CGNs: behind a home
NATs, users know each other, and can have a friendly policy to share
resources like ports forwardings. As opposed to a CGN which is more
concerned about isolating users from one another for privacy and security
reasons.
This makes a big difference in what you can require from a CGN.

-- 
_Ivan Chollet_

---

From: ivan c <[email-masked]>
Date: Thu, 20 Jun 2013 22:13:27 +0200

Hi Dan,

On Wed, 19 Jun 2013 20:21:50 -0700, Dan Wing <[email-masked]> wrote:
> 
> It is impossible to preserve TCP ports on a large or busy NAT (or a NAT
> that is both large and busy)
> 

Yes, it's impossible (regardless of the NAT's size). Home NATs usually
fall back on EDM in collision cases. This doesn't bother the TCP Hole
Punching protocol.

> It is also impossible to preserve TCP ports with A+P-related IPv4
address
> sharing mechanisms. 

Yes, such NATs map the ports topology of the host to a coarser topology
(less available ports), and although easy to implement it isn't too
satisfying. A NAT should be as transparent as possible by trying to mimic
the host's behavior and its use of ports. It's nicer to leave the host
topology intact and superpose usages of each subscriber instead. Squeezing
the number of available ports is expected to break quite a few things,
although not in major ways. This is not really an issue in the context of
p2p applications, as they are used to deal with a variety of corner cases
and weirder NATs behaviors.

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Fri, 21 Jun 2013 09:40:36 +0200

> The use case for port overloading should be TCP, not UDP.

I don't understand how you jump from sockets on the host to external 
ports on the NAT. I don't see how you reach this conclusion.

> same 5-tuple.

Stop right here. Some NATs don't track sessions. Those NATs don't care 
about 5-tuples. All they do is map internal 3-tuple to external 3-tuple. 
So for those NATs there is no conflict. They will just translate the 
packets according to the existing mapping.

All the following doesn't make sense to me given this.

Simon

> It is usually a rare event. Let's look at the corner case of

-- 
DTN made easy, lean, and smart --> http://postellation.viagenie.ca
NAT64/DNS64 open-source        --> http://ecdysis.viagenie.ca
STUN/TURN server               --> http://numb.viagenie.ca

---

From: Simon Perreault <[email-masked]>
Date: Fri, 21 Jun 2013 09:48:13 +0200

Le 2013-06-20 21:20, ivan c a écrit :
> Yes, CGNs are not expected to be p2p-friendly at this stage

I, for one, would expect them to be! See RFC 6888.

> and I don't
> real use case,

 From the start of this discussion, it has appeared to me like you do 
have a use case for EIM TCP. Don't you want to do TCP NAT traversal?

> however if there is one thing they should try to implement
> that's RFC 4787 for UDP.

Many NATs don't care about remote endpoints.

> TCP Hole Punching protocol to be deterministic. The NAT falling back on EDM

Assuming that NATs will "fall back on EDM" seems very risky.

> TCP, so makes it easy for them), and in case of a source collision,

Please understand that there are many different ways to implement NAT, 
and this is just one of them.

> key of the NAT's mapping hashtable twice as big. Not a concern for

Some NATs don't track sessions.

> prepared to deal with the case in its protocol.

Some NATs don't implement EDM and thus cannot switch to it.

Simon

---

From: ivan c <[email-masked]>
Date: Fri, 21 Jun 2013 19:15:41 +0200

On Wed, 19 Jun 2013 20:25:14 -0700, Dan Wing <[email-masked]> wrote:
> 
> This discussion should continue, even if to repeat the arguments that
were
> made years ago when the port overloading text was the WG's consensus. 
If
> WG consensus has changed, let's change the updated document.  But we do
> need a careful analysis and discussion of the impact of such a change to
> applications and to NATs.
> 

Applications are not impacted. Port overloading is a transparent NAT-only
behavior that preserves that 5-tuple uniqueness.

As for NATs, they should be free to do port overloading if they wish. It's
not reasonable to force them into one behavior or the other, since none
affect interoperability.

-- 
_Ivan Chollet_

---

From: ivan c <[email-masked]>
Date: Tue, 25 Jun 2013 18:46:34 +0200

On Fri, 21 Jun 2013 09:40:36 +0200, Simon Perreault
<[email-masked]> wrote:
>> (...)
>> (...)

We have, for all UDP communications and all outbound TCP sessions: 
- one socket is bound to one local endpoint on the host. Additionally, two
sockets have to bind to two different local endpoints.
- one internal endpoint (=local endpoint) is mapped to one external
endpoint on the NAT

As a result, if no overloading, the NAT has to allocate a new external
endpoint for every outbound TCP session.
This doesn't apply to UDP.

> All the following doesn't make sense to me given this.

Sure, *stateless* NATs also can't do much endpoint dependent filtering and
lots of other
things. They certainly can't do port overloading and are out of scope of
this discussion.
Nevertheless, NATs that do look at the full 5-tuple are free to implement
port overloading if they wish. The purpose of my text was to explain how
p2p applications interact with this, and in particular the relationship
with TCP Hole Punching.

Hope this helps.

-- 
_Ivan Chollet_

---

From: ivan c <[email-masked]>
Date: Tue, 25 Jun 2013 19:20:03 +0200

Hi,

> have a use case for EIM TCP. Don't you want to do TCP NAT traversal?

TCP NAT traversal has many variants but as far as I'm aware no application
has ever used EIM for TCP. It's nice for NAT to support different variants
for applications to fall back on.

> Many NATs don't care about remote endpoints.

Senthil and I were making the point that port preservation can never be
satisfied for all TCP connections, regardless of whether we do port
overloading or whether the NAT looks at the whole 5-tuple or not.
Your comment is also a true statement, although not directly related to
this point.

> Some NATs don't track sessions.

Sure, NATs can behave in many ways. The main point of the discussion is to
provide suggestions so NAT can have some support for p2p applications,
both
for UDP and TCP. It is expected that stateless NATs won't have too much
support for this.
On the other hand, home NATs do offer such support and have done so for
many years now.

> Some NATs don't implement EDM and thus cannot switch to it.

By definition, all NATs are "EDM".
"Endpoint-dependent Mapping" really means "Address and port-dependent
Mapping". This is the most general case of NAT, where no requirement is
made on the type of mapping. For instance, "EIM" is a particular case of
this general case.
Maybe we should use the term APDM instead of EDM for the sake of clarity,
but the best is probably to avoid using the "EDM" acronym altogether if it
leads to confusion. Instead, we can use the term "no particular
requirement
on the mapping scheme" or something similar. This way, people new to the
discussion can understand the point immediately and we also avoid the use
of new obscure acronyms.

Again, the goal here should be to suggest options for NAT to behave in p2p
friendly ways. Some useful optional features for CGNs, like port
overloading, do have minor caveats that should be detailed in the
document,
and they also don't apply to "3-tuple" NATs as you mentioned. 
This is the best way in my opinion to write a Best Current Practices
document.

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Wed, 26 Jun 2013 10:10:55 +0200

> sockets have to bind to two different local endpoints.

Not true. If you call connect() on a UDP socket, it behaves just like a 
TCP socket.

> This doesn't apply to UDP.

Not true because of the above.

> things. They certainly can't do port overloading and are out of scope of

I'm not referring to stateless NATs. I'm referring to stateful NATs that 
only map internal 3-tuple to external 3-tuple and do not do anything 
with 5-tuples.

> port overloading if they wish.

I still haven't seen any explanation why the following excerpts do not 
apply.

RFC 4787:

    REQ-3:  A NAT MUST NOT have a "Port assignment" behavior of "Port
       overloading".

    Justification:  This requirement must be met in order to enable two
       applications on the internal side of the NAT both to use the same
       port to try to communicate with the same destination.  NATs that
       implement port preservation have to deal with conflicts on ports,
       and the multiple code paths this introduces often result in
       nondeterministic behavior.  However, it should be understood that
       when a port is randomly assigned, it may just randomly happen to
       be assigned the same port.  Applications must, therefore, be able
       to deal with both port preservation and no port preservation.

RFC 5382:

    Justification:  This requirement allows two applications on the
       internal side of the NAT to consistently communicate with the same
       destination.

Simon

---

From: Simon Perreault <[email-masked]>
Date: Wed, 26 Jun 2013 10:15:33 +0200

Please explain.

> made on the type of mapping. For instance, "EIM" is a particular case of

I don't understand how this could be true.

> leads to confusion.

EDM is defined in RFC 6887. Let's keep using already-defined terminology 
if it fits.

> Instead, we can use the term "no particular
> requirement

EIM and EDM are not requirements. They are qualifiers of NAT behaviour.

> This is the best way in my opinion to write a Best Current Practices

I would simply like to first understand technically what you're proposing.

Simon

---

From: ivan c <[email-masked]>
Date: Wed, 26 Jun 2013 15:51:57 +0200

> Not true. If you call connect() on a UDP socket, it behaves just like a 

"Not true". interesting...
Do you mind providing a counterexample, ie some pseudo-code that binds two
sockets of the same protocol to the same local endpoint?

-- 
_Ivan Chollet_

---

From: "cb.list6" <[email-masked]>
Date: Wed, 26 Jun 2013 06:56:33 -0700

TCP socket.
> I'm not referring to stateless NATs. I'm referring to stateful NATs that
only map internal 3-tuple to external 3-tuple and do not do anything with
5-tuples.
apply.
>       internal side of the NAT to consistently communicate with the same

i have the port overloading feature and plan to use it.

Furthermore, i run a substantial endpoint dependent cgn. P2p on ipv4 is
dead, has been dead.

CB _______________________________________________

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 07:45:44 +0200

Le 2013-06-26 15:51, ivan c a écrit :
>>> - one socket is bound to one local endpoint on the host. Additionally,
>>> two sockets have to bind to two different local endpoints.
>>

s1 = socket(..., SOCK_DGRAM, ...);
s2 = socket(..., SOCK_DGRAM, ...);
connect(s1, ...);
connect(s2, ...);

The kernel is free to use the same local endpoint for those two sockets 
as long as the remote endpoint is different.

If you really want to force reusing the same local endpoint, you do the 
SO_REUSEADDR+bind() dance before calling connect().

Simon

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 11:20:52 +0200

> Furthermore, i run a substantial endpoint dependent cgn. P2p on ipv4 is

One of the motivations behind our draft is to describe how port 
overloading can be safely enabled without adversely affecting P2P, based 
on CGN deployment experience. Are you saying that that effort is useless?

Simon

---

From: "Reinaldo Penno (repenno)" <[email-masked]>
Date: Thu, 27 Jun 2013 10:07:58 +0000

Can you cite any references of P2P being dead in Wireline both US and outside?

From: "cb.list6" <[email-masked]<mailto:[email-masked]>>
Date: Wed, 26 Jun 2013 06:56:33 -0700
To: Simon Perreault <[email-masked]<mailto:[email-masked]>>
Cc: <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>
Subject: Re: [BEHAVE] (no subject)

Furthermore, i run a substantial endpoint dependent cgn. P2p on ipv4 is dead, has been dead.

---

From: ivan c <[email-masked]>
Date: Thu, 27 Jun 2013 13:29:09 +0200

> SO_REUSEADDR+bind() dance before calling connect().

Your pseudo-code will never bind the sockets on the same local endpoint.
It is forbidden POSIX behavior.
The option SO_REUSEADDR won't help either, this option is used to bind
over an *already closed* socket in TIME_WAIT state, that's all.

This is basic BSD-sockets: you can't bind two sockets to the same local
endpoint. Never, ever.
Not knowing this reveals deep ignorance of the basics of the topic. And
since this point is key to the discussion about NATs... 

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 14:23:48 +0200

> It is forbidden POSIX behavior.

I'm ready to accept that, with a proper citation.

> over an *already closed* socket in TIME_WAIT state, that's all.

I humbly believe that you are mistaken. It works, try it. This is a full 
example:

#include <netinet/in.h>
#include <string.h>

int
main()
{
         struct sockaddr_in       sin;
         int                      s1, s2, on = 1;

         memset(&sin, 0, sizeof(sin));
         sin.sin_family = AF_INET;
         sin.sin_addr.s_addr = htonl(0xc1319f4e);
         sin.sin_port = htons(12345);

         s1 = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
         s2 = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

         setsockopt(s1, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
         setsockopt(s2, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));

         bind(s1, (struct sockaddr *)&sin, sizeof(sin));
         bind(s2, (struct sockaddr *)&sin, sizeof(sin));

         sin.sin_addr.s_addr = htonl(0xce7b1f02);
         sin.sin_port = htons(53);
         connect(s1, (struct sockaddr *)&sin, sizeof(sin));

         sin.sin_addr.s_addr = htonl(0x42e42d6e);
         sin.sin_port = htons(53);
         connect(s2, (struct sockaddr *)&sin, sizeof(sin));

         return 0;
}

Here's the output of strace'ing that program on Linux:

> socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 3
> socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 4
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> bind(3, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("193.49.159.78")}, 16) = 0
> bind(4, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("193.49.159.78")}, 16) = 0
> connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("206.123.31.2")}, 16) = 0
> connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("66.228.45.110")}, 16) = 0

connect()ing UDP sockets has two big advantages:

1) It allows using the same programming model for UDP and TCP. More code 
can be factored out, which is good software engineering.

2) It propagates ICMP errors up through the BSD API, allowing 
straightforward error handling that is again similar to TCP and can be 
factored out. For non-connected UDP sockets, ICMP errors are simply ignored.

Since file descriptors are basically free, come at no additional 
performance cost when you use something like epoll, kqueue, or windows 
IOCP, and since this method does not consume additional ports, I 
recommend it over sendto()/recvfrom() when I teach network programming.

This comes from W. Richard Stevens "UNIX Network Programming". The 
technique is used a lot in the wild. I'm not inventing anything.

Simon

---

From: ivan c <[email-masked]>
Date: Thu, 27 Jun 2013 15:27:04 +0200

> I'm ready to accept that, with a proper citation.

Without the option SO_REUSEADDR, this is forbidden POSIX behavior

> I humbly believe that you are mistaken. It works, try it. 

It surely *works* on platforms that support this use of SO_REUSEADDR, we
discussed it numerous times already, including on the phone.
I also made the point in numerous posts already that SO_REUSEADDR is used
to bypass the TIME_WAIT state on closed sockets, and even this is not
specified by POSIX. See
http://www.ietf.org/mail-archive/web/behave/current/msg10876.html
In another post, I explained that if you use SO_REUSEADDR on all your TCP
sockets you expose them to the old duplicate problem, and the worst is that
this crosses applications boundaries. See
http://www.ietf.org/mail-archive/web/behave/current/msg10906.html

However, applications should be free to use it if they wish.
I also made the point numerous times that applications should have as many
ways as possible to cross NATs.

-- 
_Ivan Chollet_

---

From: ivan c <[email-masked]>
Date: Thu, 27 Jun 2013 15:44:04 +0200

>     REQ-7:  A NAT MUST NOT have a "Port assignment" behavior of "Port

Because some NATs would like to do port overloading, which is in
contradiction with these requirements. See section 4. of
http://tools.ietf.org/html/draft-ietf-behave-requirements-update-00 .
You're mentioned as an author on this draft by the way.
In this post I explain why port overloading can be somewhat desirable:
http://www.ietf.org/mail-archive/web/behave/current/msg10896.html
I make the same point across a large number of my posts.

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 15:45:40 +0200

> Without the option SO_REUSEADDR, this is forbidden POSIX behavior

Please provide a citation.

> discussed it numerous times already, including on the phone.

So what is the problem exactly?

My point is that UDP and TCP are not different from the point of view of 
the NAT w.r.t. port preservation, since applications routinely call 
connect() on UDP sockets and it is entirely possible that two UDP 
sockets share the same local endpoint. POSIX standards have nothing to 
do with it. The functionality is available on many (all?) platforms, and 
applications use it.

> http://www.ietf.org/mail-archive/web/behave/current/msg10906.html

We're talking about UDP here. There is no TIME_WAIT.

Simon

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 15:55:24 +0200

Le 2013-06-27 15:44, ivan c a écrit :
> contradiction with these requirements.

"Would like to" is not a valid reason. We need technical arguments.

> See section 4. of

I'm not disagreeing with myself. I am only observing that this 
discussion still has not yielded any good technical argument that we 
could add to our draft.

I have suggested that one condition where port overloading could be used 
is when the NAT knows that it will not disrupt the application protocol. 
For example, the protocols running on TCP port 80 and UDP port 53 (HTTP 
and DNS) are purely client-server and therefore would not be affected by 
port overloading. Allowing NATs to do port overloading for those ports 
only would probably solve the scalability problem since they account for 
a large portion of the traffic.

Do we really need anything more complex?

> I make the same point across a large number of my posts.

We know that port overloading is desirable for a number of reasons.

What we need to argue is why the undesirable effects of port overloading 
do not apply or can be ignored.

Simon

---

From: Mark Andrews <[email-masked]>
Date: Fri, 28 Jun 2013 00:14:34 +1000

In message <[email-masked]>, Simon Perreault writes:
> Le 2013-06-27 15:44, ivan c a crit :

And overloading DNS could potentially defeat the port randomisation
done by the server even though nameservers do port overloading
themselves to send traffic out a large set of ports choosen at
random and reselected from at random.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: ivan c <[email-masked]>
Date: Thu, 27 Jun 2013 16:17:21 +0200

> So what is the problem exactly?

I have no problem, however it seems that you do have an attitude problem
:)
(joking, but please do try keep it in friendly way, as this is a public
mailing list after all)

> do with it. 

It has everything to do with POSIX standards. Options that are outside the
POSIX standards are usually quite flaky. See the shortcomings of the
SO_REUSEADDR option that I mention in my previous post.
I also totally agree that: UDP and TCP are not different from the point of
view of the NAT w.r.t. port overloading (i assume you meant port
overloading, you wrote "port preservation")
It's just that applications usually use many more TCP local endpoints than
UDP ones. See bittorrent and all others, empirical observations. So if
there is a shortage of internal endpoints, it's more likely to happen for
TCP. But it can also happen for UDP, i totally agree.

> The functionality is available on many (all?) platforms, and 
> applications use it.

Which applications use it for TCP? Reference needed.
(Although I fully agree that p2p applications should consider using it)
This was the point of this post:
http://www.ietf.org/mail-archive/web/behave/current/msg10884.html
With this follow-up:
http://www.ietf.org/mail-archive/web/behave/current/msg10906.html

> We're talking about UDP here. There is no TIME_WAIT.

No, we are talking about TCP and UDP, this is the point of the whole
discussion.
SO_REUSEADDR is not needed on UDP anyway since you can already multiplex
multiple sessions over the same UDP socket. Applications can still use it
for pure convenience though, i agree, your previous post being a nice
illustration of this use case.

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 16:21:13 +0200

> random and reselected from at random.

Good point.

Could that be solved with operational advice? In the case of CGN, we 
could advise the ISP could to make sure that its recursive nameserver 
sits on the border between the internal and external realm such that no 
DNS traffic is handled by the CGN.

Would that fully address your concern?

Simon

---

From: "ietfdbh" <[email-masked]>
Date: Thu, 27 Jun 2013 10:23:29 -0400

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of ivan c

Without the option SO_REUSEADDR, this is forbidden POSIX behavior
[dbh>] 
Ivan, can you provide a reference to the POSIX document and section that says this is forbidden?
If we design for this restriction, the document should provide a reference so readers know why.

David Harrington
[email-masked]

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 16:29:11 +0200

> overloading, you wrote "port preservation")

No, I meant port preservation. I'm having a hard time following, and I 
thought that the UDP-vs-TCP distinction was w.r.t. port preservation.

> TCP. But it can also happen for UDP, i totally agree.

Fully agree.

> Which applications use it for TCP? Reference needed.

By "it", I was referring to "the ability to open multiple sockets bound 
to the same client endpoint". With TCP, applications do it by default.

> illustration of this use case.

Exactly. My point is that UDP applications do sometimes create 5-tuples 
sharing the same local 3-tuple. Empirically we see that it is a minority 
of the traffic, but the functionality is there, is being used, and needs 
to be considered in the recommendations for NAT that we end up writing.

Simon

---

From: Mark Andrews <[email-masked]>
Date: Fri, 28 Jun 2013 00:36:12 +1000

> Would that fully address your concern?

No.  Many ISP's have a history of mucking with DNS results when you
use their servers.  Most ISP's don't muck DNS queries not directed
to their servers.  For those that do you can usually see that they
are mucking with results.

You can't just redirect queries at a normal recursive server and
think that will provide a "transparent DNS caching server".

Mark

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 16:48:14 +0200

> think that will provide a "transparent DNS caching server".

Right, but, port overloading is not what kills the randomization done by 
the DNS client. Non-port preserving NAT is what kills it.

Non-port preserving NATs are already required to implement port 
randomization according to:

https://tools.ietf.org/html/rfc6056#section-4

Port overloading is not incompatible with port randomization. Maybe we 
should explicitly write this and add a reference to RFC 6056?

Simon

---

From: ivan c <[email-masked]>
Date: Thu, 27 Jun 2013 16:51:07 +0200

On Thu, 27 Jun 2013 10:23:29 -0400, "ietfdbh" <[email-masked]> wrote:
> If we design for this restriction, the document should provide a
reference
> so readers know why.

This is taken from "The Open Group Base Specifications" (or informally The
Single Unix Specification), by the IEEE.
Volume "System Interfaces", function "bind()":

"The bind() function shall fail if:

[EADDRINUSE]
    The specified address is already in use."

As we mentioned in our posts, this can be bypassed by the use of the
SO_REUSEADDR option, which behavior is unspecified by POSIX but has
consistent semantics across major OSes.
It is now defined with the following: 
"SO_REUSEADDR
    Specifies that the rules used in validating addresses supplied to
bind() should allow reuse of local addresses, if this is supported by the
protocol. This option takes an int value. This is a Boolean option."

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 16:56:44 +0200

>      The specified address is already in use."

Unless I misunderstand, there is no call to bind() in the sample that 
you assert is prohibited by POSIX:

About that code, I said: "The kernel is free to use the same local 
endpoint for those two sockets as long as the remote endpoint is different."

To which you replied: "Your pseudo-code will never bind the sockets on 
the same local endpoint. It is forbidden POSIX behavior."

Simon

---

From: ivan c <[email-masked]>
Date: Thu, 27 Jun 2013 17:29:53 +0200

> "Would like to" is not a valid reason. We need technical arguments.

That's right, but I think it's the above requirements that needed a valid
technical argument in the first place.
The only argument that was given is that the set of requirements
guarantees a deterministic (consistent) behavior of the NAT from the
application point of view.
But NATs never behave in a deterministic way, they might drop packets, or
random NATs can sometimes appear to be deterministic. And it is not the
point.
What is required is that the *interfaces* of each *protocol* behaves in a
deterministic way (ie, a function must behave according to its
specification 100% of the time). Generally, a deterministic NAT allows you
to write simpler protocols.
In this post, I explain how the TCP Hole Punching protocol determinism is
not affected:
http://www.ietf.org/mail-archive/web/behave/current/msg10896.html

> discussion still has not yielded any good technical argument that we 

see above, I feel it is a valid point.

> Do we really need anything more complex?

Allowing port overloading on 80 is the same as allowing it on all ports
(applications are free to use any port for any protocol, Skype for instance
uses port 80 for p2p communications).
Your suggestion is good though, because I don't think any protocol is
affected by port overloading. If they were, they would be badly designed,
but still I have never heard of any such protocol.

> do not apply or can be ignored.

Definitely, this is the main point of our argument.
I think we have progressed in this argument and provided useful material.
In this post:
http://www.ietf.org/mail-archive/web/behave/current/msg10896.html , I
explain (among other things), why the TCP Hole Punching Protocol is not
affected by the rare cases of collision due to port overloading. The
application simply retries with a new TCP socket (on a new local endpoint).

Also, it is mentioned in many RFCs that p2p applications need to expect
NATs to have a variety of non-deterministic behaviors wrt port allocation,
and have fallback mechanisms in place to deal with all cases.

-- 
_Ivan Chollet_

---

From: Tom Taylor <[email-masked]>
Date: Thu, 27 Jun 2013 11:34:53 -0400

The POSIX.1 specification for Connect() says the following:

"If the socket has not already been bound to a local address, connect() 
shall bind it to an address which, unless the socket's address family is 
AF_UNIX, is an unused local address."

---

From: Simon Perreault <[email-masked]>
Date: Thu, 27 Jun 2013 17:59:00 +0200

> application simply retries with a new TCP socket (on a new local endpoint).

...as long as the NAT does port preservation, right?

If so, the problem I see is that CGNs do not / can not / will not do 
port preservation, as a few people already pointed out.

Simon

---

From: ivan c <[email-masked]>
Date: Thu, 27 Jun 2013 18:03:15 +0200

> the same local endpoint. It is forbidden POSIX behavior."

Yes, the kernel will actually call bind() in the connect() function from
within the kernel (implicit bind()). strace wouldn't show it as it is a
kernel-mode call.

-- 
_Ivan Chollet_

---

From: ivan c <[email-masked]>
Date: Thu, 27 Jun 2013 18:46:59 +0200

> ...as long as the NAT does port preservation, right?

No, I meant it in the general case with port overloading, with or without
TCP port preservation.
As you know, port overloading breaks the EIM requirement in the general
case. This is why port overloading is a MUST NOT in the existing RFCs.
But since collisions are supposed to be rare, p2p application can simply
retry with a new socket (on a new local endpoint) if the NAT happened to
refuse or drop a packet.

> port preservation, as a few people already pointed out.

Sure, regarding port preservation, on very busy CGNs it will surely not be
granted all the time. But they are free to use the port allocation scheme
they like. As you say, I assume a lot of CGNs won't support it.
Port preservation should probably be an optional feature (especially for
home NATs, that generally already have it). It is not necessarily related
to port overloading.
Port overloading, i feel, should probably be an optional feature too,
since it is used in the wild and has limited impact on TCP Hole Punching.
I'm not a NAT implementer though and thus my voice is not the most
important here, as I see it as a NAT only feature that doesn't seriously
affect p2p applications.

Hope this helps clarifying the point.

-- 
_Ivan Chollet_

---

From: Mark Andrews <[email-masked]>
Date: Fri, 28 Jun 2013 08:11:33 +1000

> the DNS client. Non-port preserving NAT is what kills it.

Deterministic (e.g. sequential) port assignment kills it.  Port
overloading kills it if not done sensibly.

---

From: ivan c <[email-masked]>
Date: Fri, 28 Jun 2013 00:13:44 +0200

> can be factored out, which is good software engineering.

Usually applications use UDP and TCP differently and for different things,
but that's an interesting point.

> factored out. For non-connected UDP sockets, ICMP errors are simply

All errors (including ICMP) are received for all UDP sockets.
If you call select() on your sockets, the sockets with the errors will be
select()-ed, and the subsequent recvfrom() call will return the error code.
An ICMP message will trigger a ECONNRESET error I think.

-- 
_Ivan Chollet_

---

From: Simon Perreault <[email-masked]>
Date: Fri, 28 Jun 2013 14:01:31 +0200

> retry with a new socket (on a new local endpoint) if the NAT happened to

I think I'm starting to understand your proposal. Correct me if I'm 
wrong: you're proposing that NATs should be allowed to do port 
overloading as long as their default behaviour is EIM. They would only 
do port overloading if they're running out of ports. Is that correct?

That would solve the port scalability problem for busy NATs.

Your argument is that this kind of NAT can still be easily traversed by 
P2P applications, correct? Now, the thing I don't understand is how. You 
say "p2p application can simply retry with a new socket (on a new local 
endpoint) if the NAT happened to refuse or drop a packet".
- What packet gets dropped?
- How does the application know?
- How will using a new local endpoint result in no port overloading if 
the NAT is already busy enough that it has started overloading ports?

> home NATs, that generally already have it). It is not necessarily related

Ok. Then it seems nothing needs to be said about port preservation, correct?

Simon

---

From: Simon Perreault <[email-masked]>
Date: Fri, 28 Jun 2013 14:03:06 +0200

> overloading kills it if not done sensibly.

Sure, but isn't all that already covered by RFC 6056?

That is, does anything still need to be said about this?

Simon

---

From: Simon Perreault <[email-masked]>
Date: Fri, 28 Jun 2013 14:16:03 +0200

> An ICMP message will trigger a ECONNRESET error I think.

...on Linux. Not on BSD. Don't know about Windows.

W. Richard Stevens says: "Asynchronous errors are returned to the 
process for connected UDP sockets. The corollary, as we previously 
described, is that unconnected UDP sockets do not receive asynchronous 
errors."

Also: "Linux returns most ICMP "destination unreachable" errors even for 
unconnected sockets, as long as the SO_BSDCOMPAT socket option is not 
enabled. All the ICMP "destination unreachable" errors from Figure A.15 
are returned, except codes 0, 1, 4, 5, 11, and 12."

Simon