Subject: [BEHAVE] Inbound and outbound connections

From: Brian E Carpenter <[email-masked]>
Date: Tue, 05 Aug 2008 13:41:20 +1200

In case people aren't already aware of it, I think the work at the
second URL below is very interesting in discussing the possible
implications of CGN, and in particular the number of outbound and
inbound ports needed per active host.

(Sorry about cross-posting, please feel free to trim any replies.)

    Brian

-------- Original Message --------
Subject: Re: [nznog] Importance of end-to-end IPv4 - research - please help
Date: Tue, 05 Aug 2008 12:20:13 +1200
From: Richard Nelson <[email-masked]>
To: Brian E Carpenter <[email-masked]>
CC: Shane Alcock <[email-masked]>
References: <[email-masked]> <[email-masked]>

Yes that is fine.  Glad to have the work used.  Point out that it was
sponsored by Alcatel and presented by David Miles at Apricot this year.

If you are interested in the source of the data a description of the
trace is at http://www.wand.net.nz/wits/localisp/b/3/

Richard.

Brian E Carpenter wrote:
> Shane,
>
>   
>> More detailed results (plus pretty graphs!) can be found at 
>> http://www.wand.net.nz/~spa1/someisp/flow_counting/result_page.html#inbound
>>     
>
> This is wonderful information. Are you watching the current discussion
> in the IETF about carrier grade NAT? Do you have any objection to
> me forwarding the URL there?
>
>     Brian
>   

_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 5 Aug 2008 11:40:17 +0200

> inbound ports needed per active host.

Hm, it looks like the high water marks for TCP and UDP sessions per  
subscriber (including non-active subscribers) is around 20 and 40.

For our purposes we can probably assume that the UDP is lower than TCP  
because UDP in this is very likely mostly DNS which shouldn't have to  
go through a NAT64.

So that means a single IPv4 address can support about 3000 subscribers.

However, in the future it's likely that each subscriber will have a  
larger number of devices so we shouldn't be too surprised if this goes  
down.

I'd say based on this, our order of magnitude number would have to be  
1000 subscribers per IPv4 address = a /22 per million users.
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 5 Aug 2008 07:59:32 -0500

Agreeing with Iljitch's math here, but ...

> because UDP in this is very likely mostly DNS which shouldn't have to  go 
> through a NAT64.
>

... the "planning for 500 connections per user" reference in the plenary 
presentation last week was way higher than I've heard previously, but the 
working number from the P2PI workshop in May was about 80 for BitTorrent, 
wasn't it? So we're already looking at a factor-of-two reduction in what we 
can support on one IPv4 address, before someone comes up with an even cuter 
connection-use strategy...

:-(

Spencer

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 5 Aug 2008 15:08:35 +0200

> ... the "planning for 500 connections per user" reference in the  
> plenary presentation last week was way higher than I've heard  
> previously, but the working number from the P2PI workshop in May was  
> about 80 for BitTorrent, wasn't it? So we're already looking at a  
> factor-of-two reduction in what we can support on one IPv4 address,  
> before someone comes up with an even cuter connection-use strategy...

The problem is that today, BitTorrent is a very bad citizen when it  
comes to port number use: it uses many connections, and it keeps them  
open for a long time.

I would assume that a "carrier grade NAT" would implement per-user  
port number limits. Hopefully these will be such that BitTorrent etc  
can still run at night when other users don't need the ports.

But the real solution would be update the BitTorrent implementations  
so that they use a single source port, or at least a very small number  
of source ports, which has two benefits: it uses up fewer ports and  
it's now possible to use ICE to compensate for the loss of UPnP and  
NAT-PMP.

So the future doesn't look too bleak (and that's not even including  
the ability to use native IPv6!) but the road getting there could be a  
bit bumpy.
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Tue, 5 Aug 2008 16:26:12 +0300

> it's now possible to use ICE to compensate for the loss of UPnP and

Except that ICE does not really support anything except UDP at this stage. 
BitTorrent, as a bulk transfer protocol, requires TCP. Also BitTorrent 
requires inbound connections to work on a significant proportion of nodes, 
which requires UPnP, NAT-PMP, a public IPv4 address, or, if the user is 
knowledgeable, static port forwarding.

For real-time traffic (such as VoIP), ICE is great, and we really need ISP 
NATs to support it. However, when it comes to peer-to-peer bulk transfer, I 
suspect a "carrier-grade NAT-PMP" will be more practical than "TCP-grade 
ICE".

-- 
Rémi Denis-Courmont
Maemo Software, Nokia Devices R&D
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Tue, 5 Aug 2008 16:28:29 +0300

> because UDP in this is very likely mostly DNS which shouldn't have to

Real-time traffic, including online games, VoIP, videoconferencing, relies 
heavily on UDP, if only to avoid TCP head-of-line blocking.

If NAT vendors had not assumed that DNS was the only usage of UDP, we would 
probably have overall much better NAT in the real-world today. Please do not 
make the same mistake.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 5 Aug 2008 15:47:15 +0200

> Except that ICE does not really support anything except UDP at this  

What's the difficulty with TCP? If I understand things correctly, once  
you use a certain port as a source port number on the public side, a  
behave-compliant NAT will forward incoming sessions towards that port  
number to you. So the only thing the application has to do is figure  
out what the address/port is that others see and not release the port  
number and it's in business.

So BitTorrent doesn't really need ICE, just STUN functionality for TCP.
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 5 Aug 2008 15:49:47 +0200

> heavily on UDP, if only to avoid TCP head-of-line blocking.

Huh? Head of line blocking?

> probably have overall much better NAT in the real-world today.  

Obviously UDP is used for many things, but everything that you mention  
keeps a UDP "session" going for some time unless I'm very much  
mistaken. DNS queries on the other hand use a different port for  
pretty much every new packet.
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Tue, 5 Aug 2008 17:09:25 +0300

> number and it's in business.

The passive side of the TCP connections can hardly be behind a NAT. So there 
is no way the NAT can know about it. That's why we have UPnP IGD and NAT-PMP.

So... One proposed work-around is for both sides to be simultaneously active. 
The TCP spec allows this, but:
1/ Most firewalls and many NATs do not support this.
   They reset the connection.
2/ Some OS's do not support it either.
3/ When the OS supports it, it needs a cumbersome socket API hack involving a 
listening socket and an connecting socket on the same port.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 5 Aug 2008 16:42:14 +0200

> The passive side of the TCP connections can hardly be behind a NAT.

Isn't that the whole point of endpoint independent NATing?

I first open a session with source port 10 and destination address X.  
The NAT rewrites the source port into 93 and the source address into  
Y. The server at X tells me values 93 and Y. I then start listening on  
port 10 and tell others to contact me on Y port 93.
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 05 Aug 2008 16:58:37 +0200

> Isn't that the whole point of endpoint independent NATing?

To allow for it yes, but there is also the filtering behavior. NATs 
doesn't allow traffic in unless you have sent to that address or even 
address and port.
> The NAT rewrites the source port into 93 and the source address into Y. 
> The server at X tells me values 93 and Y. I then start listening on port 
> 10 and tell others to contact me on Y port 93.

Yes, then when someone is going to connect you need to be told by the 
rendevous server that a client at address Z and port 4042 wants to 
connect. Then you can send a TCP SYN to Z:4042 from your port 10. That 
way the NAT opens for traffic. At the same time the client send to Y:93. 
Now the NAT needs to understand that it actually should consider this a 
simultaneous open situation.

Unfortunately I have to agree with Remi, for TCP and carrier NAT you 
really like to have a whole punching protocol for listing sockets.

There are some NATs that makes semi-static bindings without filtering in 
the NAT that makes what you suggest working for certain ports. So one 
can also consider what the full implications are of NATs without 
filtering rules.

Cheers

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Färögatan 6                | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 5 Aug 2008 17:22:25 +0200

> doesn't allow traffic in unless you have sent to that address or  
> even address and port.

Right, because of the unfortunate notion that NAT provides security.

With NAT64 this isn't an issue because even if the NAT64 were to  
filter, incoming sessions over IPv6 would still be possible so a  
filtering by a NAT64 doesn't provide any security. So any filtering  
would have to be done on the CPE and/or the hosts in question.

For a dual stack light type of solution this is slightly different,  
but in this case I think it would also be more secure to have the  
encapsulating CPE provide filtering rather than the ISP NAT, if  
filtering is desired.

In the absense of filtering on the NAT incoming sessions the way I  
described in my previous message should work, right?

> really like to have a whole punching protocol for listing sockets.

So which is it then?

If the endpoint independent behavior doesn't buy us incoming sessions,  
what's the use and why would we impair the NAT box scalability for it?
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Wed, 6 Aug 2008 09:21:33 +0300

> Isn't that the whole point of endpoint independent NATing?

If you prefer it that way:
most NATs and just about all firewalls are not at all BEHAVE-TCP-compliant 
today.

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 6 Aug 2008 15:45:38 -0500

This got skipped over ... but since you asked ...

> Huh? Head of line blocking?

TCP provides a reliable, stream-oriented service - so if you drop a packet, 
none of the packets that follow the dropped packet can be delivered to the 
application until the dropped packet is retransmitted and received. So, the 
sequence number gap blocks all subsequent delivery until that happens.

Spencer

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 07 Aug 2008 10:28:06 +1200

On 2008-08-07 08:45, Spencer Dawkins wrote:
> TCP provides a reliable, stream-oriented service - so if you drop a
> packet, none of the packets that follow the dropped packet can be
> delivered to the application until the dropped packet is retransmitted
> and received. So, the sequence number gap blocks all subsequent delivery
> until that happens.

Which is the fundamental weirdness about using TCP to deliver media streams,
since codecs generally prefer to forget about a dropped packet ASAP and
just get on with the next packet.

    Brian

---

From: Marshall Eubanks <[email-masked]>
Date: Thu, 7 Aug 2008 00:09:45 -0400

Dear Brian;

> just get on with the next packet.

Streaming media (as opposed to video conferencing or VOIP)
typically use large (multi-second) buffers, so they can use even  
fairly late packets.
The tcp streaming generally cannot recover from a timeout of the TCP  
session, though.

Regards
Marshall

> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 7 Aug 2008 09:42:05 +0200

> delivered to the application until the dropped packet is  
> retransmitted and received. So, the sequence number gap blocks all  
> subsequent delivery until that happens.

Oh, right. It read like TCP packets would be stuck in router  
buffers... I wasn't aware that the term head of line blocking was also  
used in an end-to-end context.
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 13 Aug 2008 21:45:18 +0200

On 13 aug 2008, at 19:32, Marc Blanchet wrote:

>>> Hm, it looks like the high water marks for TCP and UDP sessions  
>>> per subscriber (including non-active subscribers) is around 20 and  
>>> 40.
>>

>> previously,

> I think Miyakawa-san meant "up to 500", and he was referring to AJAX  
> connections used in applications such as google maps.

I would like to see some evidence for this. Google Maps only generates  
some 20 sessions for me.

Note that 500 ports per subscriber means only 125 subscribers per IPv4  
address, which means that we can't really afford endpoint independent  
mappings without limitations. We've had a short discussion on limiting  
endpoint independence to exclude stuff like HTTP in behave, but we  
haven't arrived at a useful conclusion yet.
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 13 Aug 2008 15:30:34 -0700

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Ted Lemon
> Sent: Wednesday, August 13, 2008 12:54 PM
> To: Iljitsch van Beijnum
> Cc: [email-masked]; [email-masked]
> Subject: Re: [Int-area] [BEHAVE] Inbound and outbound connections
> 
> On Aug 13, 2008, at 12:45 PM, Iljitsch van Beijnum wrote:
> Another problem with the statistic is that it assumes that all users  
> are surfing google web simultaneously.   In practice, use 
> tends to be  
> staggered.   As long as your NAT is noticing FIN and RST 
> packets, and  
> recycling ports quickly,

Caution needs to be exercised in recylcing TCP ports quickly;
see draft-ananth-tsvwg-timewait-00.txt and 
draft-ietf-tsvwg-port-randomization-01.txt.

> you can probably do just fine with 
> many more  
> subscribers per IP address than Miyakawa-san's worst-case scenario  
> would suggest.
> 
> That's all out the window if they're all doing P2P, of 
> course, because  
> those connections stay up longer.   But if we have IPv6 in the  
> network, we should really encourage people running P2P to use it,  
> because it should work better for them than IPv4 anyway - no NAT  
> traversal problems.

Without a way to accept incoming TCP sessions (e.g., UPnP or
some other way to ask for a publicly-routable transport address that
accepts incoming TCP SYNs), BitTorrent will consider p2p clients as 
leeches and download speeds will suffer:  keeping TCP connections
up even longer.  

We can't just 'encourage' p2p users to move to v6 until there is
content on v6 -- this is the very same problem for v6-only clients
wanting to access v6 content (there isn't any -- www.google.com,
www.cnn.com, www.amazon.com, are not on v6.  "ipv6.google.com" != 
"www.google.com").

-d

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 13 Aug 2008 17:06:04 -0700

> On Aug 13, 2008, at 3:30 PM, Dan Wing wrote:
> Actually, that's my point.   p2p is *not* cnn.com.   p2p can 
> discover  
> ipv6 peers automatically.   So okay, you can't get to 
> google.com with  
> v6.   But you can make your p2p stuff work better.   So there's some  
> value that ipv6 could add sooner rather than later.

Gotcha:  So you're saying "if you're behind a CGN (doing v4v4 NAT or
v6v4 NAT), you get 100 ports.  But if you are doing native v6, you
get 64K ports.  Your massive port-consuming p2p application will
work better with 64K ports".

Am I characterising that correctly?  

-d

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 13 Aug 2008 17:48:43 -0700

> Ted Lemon a écrit :
> > On Aug 13, 2008, at 5:06 PM, Dan Wing wrote:
> > Just about.   I was also making the additional point that 
> this provides 
> > a meaningful incentive to the end-user to go dual-stack.
> 
> and that with v6, you are not hidden by default (only by 
> firewall policy if desired), therefore it makes servers (ie. 
> a listening port) more easier to deploy. think about SIP today... 
> I've been involved in voipv6 and it is 10 times simpler 
> to deploy for that reason.

The filtering recommendations in 
http://tools.ietf.org/html/draft-ietf-v6ops-cpe-simple-security-03
would change your experience.  You might bring that up in v6ops.

-d

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 14 Aug 2008 12:01:56 +0200

> www.cnn.com, www.amazon.com, are not on v6.  "ipv6.google.com" !=

Actually I'm downloading a file with BitTorrent right now, and I'm  
connected to 133 peers, 131 IPv4 and 2 IPv6 (6to4, but still). The  
current trackers and clients usually won't let you do IPv6, but the  
DHT that's in Azureus and some other clients will exchange IPv6  
addresses as well as IPv4 addresses, so users of clients that support  
DHT and IPv6 running on systems with IPv6 connectivity will  
automatically make use of IPv6.

Note that for the purpose of IPv4->IPv6 transition there are two types  
of peer-to-peer: the BitTorrent type, where an individual peer doesn't  
need to be able to talk to _all_ other peers, but just to a large  
enough subset to get the download going.

The other type is VoIP, where there is the requirement that all peers  
can (potentially) talk to all other peers.

The BitTorrent type doens't need NAT64 but just a small set of dual  
stack peers that connect the IPv4 and IPv6 clouds. The VoIP type needs  
a solution like NAT64 to make the IPv6 transition.
_______________________________________________
Behave mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/behave

---

From: Lars Eggert <[email-masked]>
Date: Thu, 14 Aug 2008 07:45:24 -0700

> subscriber (including non-active subscribers) is around 20 and 40.

http://www.wand.net.nz/~salcock/someisp/flow_counting/result_page.html

The graph you commented on talks about "peak sessions per subscriber  
(mean)", and if I understand the text correctly, that means dividing  
all active sessions by all active subscribers at the time. It's not  
actually a per-user measure.

If you scroll down on that page, the "peak tcp session per active  
subscriber (percentiles)" graph shows you how heavy-tailed the  
underlying distribution is - 5% of users have more than 100  
connections active, and 1% between 500-1000. If a CGN enforced a limit  
of 100 connections, an ISP would get calls from 5% of his customers -  
probably not what they'd like to see happen.

(These are interesting numbers, by the way - I wish other ISPs could  
share similar data!)

Lars

---

From: Marc Blanchet <[email-masked]>
Date: Wed, 13 Aug 2008 13:32:07 -0400

Spencer Dawkins a écrit :
> presentation last week was way higher than I've heard previously, but 
> the working number from the P2PI workshop in May was about 80 for 
> BitTorrent,

if the deployment scenario is based on multiple computers behind a NAT 
in the home, and 200 connections per computer and a few people in the 
home accessing Internet services, given that even now the TiVO and other 
boxes are accessing Youtube and other services, then I think 500 is a 
good cap number since it will happen over time. We shall need to take 
this number as a reference point for our work.

Marc.

> wasn't it? So we're already looking at a factor-of-two 
> reduction in what we can support on one IPv4 address, before someone 
> comes up with an even cuter connection-use strategy...
> 
> :-(
> 
> Spencer
>

---

From: Ted Lemon <[email-masked]>
Date: Wed, 13 Aug 2008 12:54:14 -0700

Another problem with the statistic is that it assumes that all users  
are surfing google web simultaneously.   In practice, use tends to be  
staggered.   As long as your NAT is noticing FIN and RST packets, and  
recycling ports quickly, you can probably do just fine with many more  
subscribers per IP address than Miyakawa-san's worst-case scenario  
would suggest.

That's all out the window if they're all doing P2P, of course, because  
those connections stay up longer.   But if we have IPv6 in the  
network, we should really encourage people running P2P to use it,  
because it should work better for them than IPv4 anyway - no NAT  
traversal problems.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 13 Aug 2008 16:02:19 -0700

> www.cnn.com, www.amazon.com, are not on v6.  "ipv6.google.com" !=

Actually, that's my point.   p2p is *not* cnn.com.   p2p can discover  
ipv6 peers automatically.   So okay, you can't get to google.com with  
v6.   But you can make your p2p stuff work better.   So there's some  
value that ipv6 could add sooner rather than later.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 13 Aug 2008 17:27:37 -0700

> Am I characterising that correctly?

Just about.   I was also making the additional point that this  
provides a meaningful incentive to the end-user to go dual-stack.

---

From: Marc Blanchet <[email-masked]>
Date: Wed, 13 Aug 2008 20:38:42 -0400

Ted Lemon a écrit :
> Just about.   I was also making the additional point that this provides 

and that with v6, you are not hidden by default (only by firewall policy 
if desired), therefore it makes servers (ie. a listening port) more 
easier to deploy. think about SIP today... I've been involved in voipv6 
and it is 10 times simpler to deploy for that reason.

Marc.