Subject: [BEHAVE] Fwd: New Version Notification for draft-nishizuka-cgn-deployment-considerations-00.txt

From: kaname nishizuka <[email-masked]>
Date: Tue, 02 Apr 2013 17:37:14 +0900

Dear all,

I'm kaname from NTT communications in Japan.
We are testing CGN under the support of Japanese Government.
Now, we've uploaded a new draft based on the result of our verification.
The useful information about the average consumption of the ports are available on the document.
Please look through it, and all kind of feedback are welcome.

By conducting realistic experiment, this draft is answering to "draft-ietf-behave-lsn-requirements-10" which will be the newest RFC very soon.

The document is *NOT* intended to be Standards Track. It's for Informational.
The wrong description is just mere mistake, so we'll soon correct it in the next revision.

The full report of our work will be available soon on the Web in English.
I'll also announce it when it's available to this mailing-list.

Best regards,

kaname

> -------- Original Message --------
> Subject: 	New Version Notification for 
> draft-nishizuka-cgn-deployment-considerations-00.txt
> Date: 	Thu, 28 Mar 2013 07:12:25 -0700
> From: 	[email-masked]
> To: 	[email-masked]
>
>
>
> A new version of I-D, draft-nishizuka-cgn-deployment-considerations-00.txt
> has been successfully submitted by Kaname Nishizuka and posted to the
> IETF repository.
>
> Filename:	 draft-nishizuka-cgn-deployment-considerations
> Revision:	 00
> Title:		 Carrier-Grade-NAT (CGN) Deployment Considerations.
> Creation date:	 2013-03-29
> Group:		 Individual Submission
> Number of pages: 16
> URL:http://www.ietf.org/internet-drafts/draft-nishizuka-cgn-deployment-considerations-00.txt
> Status:http://datatracker.ietf.org/doc/draft-nishizuka-cgn-deployment-considerations
> Htmlized:http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-00
>
>
> Abstract:
>     This document provides deployment considerations for Carrier-Grade-
>     NAT (CGN) based on the verification result include the investigation
>     of the number of sessions of applications.  The verification was
>     conducted in StarBED which is one of the largest scale network
>     experiment environment in Japan.  A million of subscribers was
>     emulated and it revealed the realistic behavior of CGN.
>
>                                                                                    
>
>
> The IETF Secretariat
>
>
>

-- 
----
Kaname Nishizuka
Innovative Architecture Center
NTT Communications Corporation
+81-50-3812-4704

---

From: Dan Wing <[email-masked]>
Date: Tue, 2 Apr 2013 09:07:43 -0700

> Please look through it, and all kind of feedback are welcome.
Thanks for publishing this document.

I like the description of DNS location at http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-00#section-6.3, as this is an important mechanism to reduce the transactional load on the CGN.  Have you analyzed the number of subscribers that over-ride the ISP-provided DNS servers to use other DNS servers (e.g., Google, OpenDNS), as that DNS query traffic will traverse the CGN.

http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-00#section-5.3.2 would benefit from some discussion of the privacy impact of an ISP storing destination information, and should also describe memory impact (in the CGN) if the subscriber uses the same source port to visit many different destinations (if CGN does not store the list of destinations, CGN will generate a log for every packet sent to a new destination).  Applications such as bittorrent can consume a lot of memory in a CGN that is configured for destination logging.

-d

>> Subject:	New Version Notification for draft-nishizuka-cgn-deployment-considerations-00.txt
>> URL:             http://www.ietf.org/internet-drafts/draft-nishizuka-cgn-deployment-considerations-00.txt
>> Status:          http://datatracker.ietf.org/doc/draft-nishizuka-cgn-deployment-considerations
>> Htmlized:        http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-00
>> 
>> 
>> Abstract:
> _______________________________________________
> Behave mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/behave

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 2 Apr 2013 16:26:18 +0000

This document would benefit from including the throughput/capacity discussion a bit along the following lines:

- challenges in spreading the traffic load (100G, say) among the available NAT capacity (distributed intra-chassis or inter-chassis). For ex, NAT entry is created by the upstream traffic, which is negligible to the downstream traffic that can exhaust the NAT capacity.
	For ex, poor usage of NAT capacity by fragmenting the IP pool (both inside and outside)

More inline,

> I like the description of DNS location at http://tools.ietf.org/html/draft-
> nishizuka-cgn-deployment-considerations-00#section-6.3, as this is an
> important mechanism to reduce the transactional load on the CGN.  Have

It is certainly desirable trait to let the DNS server not exhaust the CGN capacity. However, such a trait is not attainable when the number of users is too high to avoid overlapping (of private address space). This ends up moving DNS behind the CGN. :(

Cheers,
Rajiv

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Dan Wing (dwing)
> Sent: Tuesday, April 02, 2013 12:08 PM
> To: kaname nishizuka
> Cc: Shin Miyakawa; [email-masked]
> Subject: Re: [BEHAVE] New Version Notification for draft-nishizuka-cgn-
> deployment-considerations-00.txt
> 
> 
> I like the description of DNS location at http://tools.ietf.org/html/draft-
> nishizuka-cgn-deployment-considerations-00#section-6.3, as this is an
> important mechanism to reduce the transactional load on the CGN.  Have
> you analyzed the number of subscribers that over-ride the ISP-provided
> DNS servers to use other DNS servers (e.g., Google, OpenDNS), as that DNS
> query traffic will traverse the CGN.
> 
> http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-
> 00#section-5.3.2 would benefit from some discussion of the privacy impact
> of an ISP storing destination information, and should also describe memory
> impact (in the CGN) if the subscriber uses the same source port to visit many
> different destinations (if CGN does not store the list of destinations, CGN
> will generate a log for every packet sent to a new destination).  Applications
> such as bittorrent can consume a lot of memory in a CGN that is configured
> for destination logging.
> 
> -d
> 
> 
> 
> 	By conducting realistic experiment, this draft is answering to "draft-
> ietf-behave-lsn-requirements-10" which will be the newest RFC very soon.
> 
> 		URL:             http://www.ietf.org/internet-drafts/draft-
> nishizuka-cgn-deployment-considerations-00.txt

---

From: "Senthil Sivakumar (ssenthil)" <[email-masked]>
Date: Tue, 2 Apr 2013 22:03:40 +0000

Hi Kaname,
It is a pretty interesting study. I have a few comments on the sections on your IP address requirements calculations.In the dynamic assignment, the ports of pool address are allocated

"   # of pool address (P) = # of Subscriber (S) * a * N / (65536 - R)

   Here, (R) is reserved TCP/UDP port list referred in
   [I-D.donley-behave-deterministic-cgn<http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-00#ref-I-D.donley-behave-deterministic-cgn>].  CGN should eliminate the
   wellknown ports (0-1023 for TCP and UDP) to avoid the bad
   interpretation from destination servers.  It is natural to translate
   source port of outgoing packet to ephemeral ports.  The ports after
   32768 is considered to be used without any problems because ephemeral
   ports are 32768-61000 for Linux and 49152-65535 in the IANA
   recommendations.  As the result, 3,051 pool addresses are sufficient
   for 1,000,000 subscribers.  The feasibility of configuration was
   confirmed in the verification."

[Senthil] Most of the NATs use the entire port space from 1024-65536, did you find differently?

If you had used the entire port space, you would only require 1526 addresses instead of 3051.

I think not using the 1024-32768 is not practical and inefficient.

   On the other hand, in static assignment, the ports are allocated a
   priori for every users.  The pool addresses and ports are reserved to
   every users, so most of them could be a dead stock because there are
   light users and heavy users in aspect of port consumption.  The max
   number of port consumption in all subscribers is the key value for
   static assignment.  The true peak number of the session by a heavy
   user could be over 10,000 sessions.  However it is assumed that such
   a severe consumption of ports to be an abuse, so the number of
   statically assigned port (M) is controllable parameter by each
   providers.  In the static assignment, the required number of the pool
   address is as follows:

   # of pool address (P) = # of Subscriber (S) * M / (65536 - R)

   Taking account into the investigation of number of sessions of
   applications, the desirable value of (M) is over 1,000.  As the
   result, no less than 30,517 pool addresses are needed for 1,000,000
   subscribers.  The compression ratio is one tenth of the case of
   dynamic assignment."

[Senthil] Going by the same standards of using the entire available port range, you should require 15259 addresses.
But the point is the cost of static allocation to dynamic allocation is 10:1, which is huge and given the fact there is only so little
address space that is available, it should be used efficiently. Just by looking at that calculation, it seems amply clear that dynamic is preferred over static,
as you have noted.
That leads us to the next topic on logging.

  " In addition, the indicator of the allocation and deallocation are
   needed because it assures that the identified subscriber certainly
   had been using the translated IP address and port.  Plus, including
   the index of the CGN host, the average size of NAT log is about 120
   byte in ASCII format.  Every active subscriber generate 400 sessions
   in average for a certain amount of time.  It is assumed that the
   event happens every 5 minute in the most severe condition.  The size
   of the log (L) for time frame (T) can be estimated as follows:

   The size of log (L) = # of Subscriber (S) * a * N * 120byte * 2 * (
   Time frame(T) / 5 min. )

   It should be noted that the log is generated at the timing of NAT
   table creation and freeing.  As the result, for 1,000,000 users, the
   size of log is piled up to 6.4 terabytes per day.  The verification
   result confirm the existing estimation referred in
   [I-D.donley-behave-deterministic-cgn<http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-00#ref-I-D.donley-behave-deterministic-cgn>]."

[Senthil] I think it should be mentioned that with binary format for the above logging information, you need 26 bytes.

Transport Protocol -  1 byte

 Source IP address:port – 6 bytes
 Source IP address:port after translation – 6 bytes
 Timestamp – 8 bytes

 Add/Delete – 1 byte

 Subscriber ID/VRF ID – 4 bytes

I would help a lot to add the binary logging calculation in addition to the ascii format to help people choose the right method for them.

Going by the same formula you need 1.36 TB/day with binary logging. Assuming that the compression ratio is the same

For both ascii and binary data, you have 1/4-1/5th of advantage with binary logging.

You spread the cost of 1.36TB across a 1,000,000 subscribers assuming cost of 1TB is US$80,

we come to 0.00008 cents/subscriber/day or 0.024 cents/subscriber/month. Even with ASCII logging requirements this doesn’t sound like a mammoth cost.

Sure, there are other operating expenses involved but comparing that with the efficiency of a dynamic port allocation mechanism and requiring 10 times less

public addresses seems to be a compelling winning argument than changing every CGN implementation to save for logging costs.

Thanks

Senthil

From: kaname nishizuka <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, April 2, 2013 4:37 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: Shin Miyakawa <[email-masked]<mailto:[email-masked]>>
Subject: [BEHAVE] Fwd: New Version Notification for draft-nishizuka-cgn-deployment-considerations-00.txt

Dear all,

Best regards,

kaname

-------- Original Message --------
Subject:        New Version Notification for draft-nishizuka-cgn-deployment-considerations-00.txt
Date:   Thu, 28 Mar 2013 07:12:25 -0700
From:   [email-masked]<mailto:[email-masked]>
To:     [email-masked]<mailto:[email-masked]>

Abstract:

The IETF Secretariat

---

From: Simon Perreault <[email-masked]>
Date: Wed, 03 Apr 2013 10:45:36 +0200

Le 2013-04-02 18:07, Dan Wing a écrit :
> http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-00#section-5.3.2
> would benefit from some discussion of the privacy impact of an ISP
> storing destination information, and should also describe memory impact
> (in the CGN) if the subscriber uses the same source port to visit many
> different destinations (if CGN does not store the list of destinations,
> CGN will generate a log for every packet sent to a new destination).
>   Applications such as bittorrent can consume a lot of memory in a CGN
> that is configured for destination logging.

Another issue with this section is that it ignores bulk port allocation. 
It says:

"It is confirmed by the verification that by logging destination 
address, only 4% of amount of log is increased in ASCII format."

The percentage would be much higher with bulk port allocation.

Simon

---

From: Simon Perreault <[email-masked]>
Date: Wed, 03 Apr 2013 10:52:33 +0200

Le 2013-04-03 00:03, Senthil Sivakumar (ssenthil) a écrit :
> [Senthil] Going by the same standards of using the entire available
> port range, you should require 15259 addresses. But the point is the
> cost of static allocation to dynamic allocation is 10:1, which is
> huge and given the fact there is only so little address space that is
> available, it should be used efficiently. Just by looking at that
> calculation, it seems amply clear that dynamic is preferred over
> static, as you have noted.

My reaction was different. I was thinking "only 10:1, this is pretty
good for static."

> You spread the cost of 1.36TB across a 1,000,000 subscribers assuming
> cost of 1TB is US$80, we come to 0.00008 cents/subscriber/day or
> 0.024 cents/subscriber/month. Even with ASCII logging requirements
> this doesn’t sound like a mammoth cost. Sure, there are other
> operating expenses involved

The raw cost of storage is probably insignificant compared to those 
"other operating expenses"...

Simon

---

From: "Senthil Sivakumar (ssenthil)" <[email-masked]>
Date: Wed, 3 Apr 2013 13:19:16 +0000

>My reaction was different. I was thinking "only 10:1, this is pretty

Why so? With the same 15259 addresses that I have, I could add 10 times
more subscribers,
I can add up to 10Million subscribers as per this example, that is 10
times more 
revenue. That was also the revenue lost, why is it good for business?
>> this doesn¹t sound like a mammoth cost. Sure, there are other

Can you put a dollar value to the significant other operating expenses?
Does it offset the 10x revenue generated by doing dynamic
port allocation and logging ? I think the key would be what the business
proposition is at the end of the day.

Senthil

---

From: Simon Perreault <[email-masked]>
Date: Wed, 03 Apr 2013 15:59:25 +0200

> revenue. That was also the revenue lost, why is it good for business?

You're assuming that the limiting factor is addresses. You're assuming 
that even with dynamic allocation, the ISP is going to "fill" all its 
available public space. That's not what the text says. All the text says 
is that dynamic allows ~10 times more users behind each address than 
static. Nowhere does it say that static isn't "good enough".

> proposition is at the end of the day.

It would vary per organization. An organization that is already well 
equipped in logging infrastructure and procedures would have a much 
lower cost deploying additional logging for CGN. But an organization 
that doesn't have this infrastructure would quickly say "no way!" and 
would look hard for alternatives. But all this is over-generalization, 
of course.

Simon

---

From: "Senthil Sivakumar (ssenthil)" <[email-masked]>
Date: Wed, 3 Apr 2013 18:57:05 +0000

>static. Nowhere does it say that static isn't "good enough".

If it is not the address, what is the limiting factor? The reason ISP is
deploying CGN is the shortage of addresses and cant provide a single
address to each
of his subscribers. Maybe you meant to say the address is not the only
limiting factor.
I never said the text was saying static isnt good enough :-), I deduced
from the study that
the usage of ports is far more compellingly efficient with dynamic port
allocation and the cost
of logging infra can be justified.

Most of the studies in the past projected how bad the logging problem is
but didn’t have
any data on the other side of the equation on how inefficient the static
port allocation is.
I wouldn’t want this draft to say one is better than the other, but let
the operators choose, if 10:1
static to dynamic port allocation is justified for their deployment.

Senthil

---

From: "Howard, Lee" <[email-masked]>
Date: Wed, 3 Apr 2013 16:40:05 -0400

>>static. Nowhere does it say that static isn't "good enough".

I have not been following this thread closely, but it keeps popping up.
It doesn't say it's not good enough, but it spends two paragraphs
describing why it is inferior.
And says, "In the aspect of the efficiency, the dynamic assignment is
preferable
   than the static assignment.  However the size of the log is the main
   consideration."

I agree that there's an important consideration in whether more efficiency
is "compression ratio" is needed.  If static port allocation allows an ISP
to take the addresses they would normally consume in one month and make
them last for five years (60:1 compression), then just a month or a couple
of months' worth of addresses lasts through the transition.

By the way, improving your CGN does not increase revenue.

>of logging infra can be justified.

Maybe it should say, "IF the cost of logging can be justified."  Even 30%
of 6.2TB per day is a lot of data, and depending on your data retention
requirements, might require a significant system to retrieve records
efficiently.

>static to dynamic port allocation is justified for their deployment.

It could be clearer in the draft.

Lee

>>would look hard for alternatives. But all this is over-generalization,

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: kaname nishizuka <[email-masked]>
Date: Thu, 04 Apr 2013 10:57:39 +0900

Thanks for your comment.

> I like the description of DNS location at http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-00#section-6.3, as this is an important mechanism to reduce the transactional load…
Unfortunately, we have not yet investigated the proportion of the 
subscribers who are using provided DNS versus who are using public DNS.
Before investigating it, the test we managed to do was that all DNS 
traffic traverse the CGN as the most severe case.
The proportion could be different in providers,  but the impact of the 
DNS query traffic is relatively small if DNS timeout is adjusted.

> http://tools.ietf.org/html/draft-nishizuka-cgn-deployment-considerations-00#section-5.3.2 would benefit from some discussion of the privacy impact of an ISP storing destination information, and sho…
We also should care for port-overlapping behavior.

kaname

---

From: Dan Wing <[email-masked]>
Date: Wed, 3 Apr 2013 19:21:19 -0700

On Apr 3, 2013, at 6:57 PM, kaname nishizuka <[email-masked]> wrote:

> Unfortunately, we have not yet investigated the proportion of the subscribers who are using provided DNS versus who are using public DNS.
> Before investigating it, the test we managed to do was that all DNS traffic traverse the CGN as the most severe case.
> The proportion could be different in providers,  but the impact of the DNS query traffic is relatively small if DNS timeout is adjusted.

Yeah, and most of the DNS traffic is UDP (until we have larger records) so even if the UDP timers were pretty long I bet the user port limit wouldn't be exhausted.

> We also should care for port-overlapping behavior.

Yes, I believe it's the same thing using a different term.

-d

---

From: kaname nishizuka <[email-masked]>
Date: Thu, 04 Apr 2013 11:39:03 +0900

Thank you for your comments,

> I think not using the 1024-32768 is not practical and inefficient.
What you pointed out is reasonable enough, but our concern is that there 
are de-facto well-known port in 1024-32768 like 1433/tcp.
Especially in the static assignment, the subscriber who has a "dirty" 
range would be attacked more frequently than others.
We know that most of the NATs use the  entire port space from 
1024-65536, but we described it in the most safe side.
As you mentioned, the ratio of dynamic:static is the same.

>   Subscriber ID/VRF ID – 4 bytes
I agree with that. It's useful and it supports the logic.

Thanks,
kaname

---

From: Simon Perreault <[email-masked]>
Date: Thu, 04 Apr 2013 11:49:05 +0200

Le 2013-04-03 20:57, Senthil Sivakumar (ssenthil) a écrit :
> If it is not the address, what is the limiting factor? The reason ISP
> is deploying CGN is the shortage of addresses and cant provide a
> single address to each of his subscribers. Maybe you meant to say the
> address is not the only limiting factor. I never said the text was
> saying static isnt good enough :-), I deduced from the study that the
> usage of ports is far more compellingly efficient with dynamic port
> allocation and the cost of logging infra can be justified.

I'll try to illustrate my point with an example with numbers.

An ISP is running out of addresses. It considers two options: static CGN 
vs dynamic CGN. Static allows, let's say, 32 users per public IPv4 
address. Given the 1:10 figure from the draft, it follows that dynamic 
allows 320 users per public IPv4 address.

If 32 is "enough", why suffer the trouble of logging (among others) just 
to get to 320? If 32 and 320 are both "enough", then considerations 
other than efficient use of public IPv4 addresses must take priority.

"Enough" could mean something like "enough to support projected growth 
for X years".

> Most of the studies in the past projected how bad the logging problem
> is but didn’t have any data on the other side of the equation on how
> inefficient the static port allocation is. I wouldn’t want this draft
> to say one is better than the other, but let the operators choose, if
> 10:1 static to dynamic port allocation is justified for their
> deployment.

The 10:1 figure is useful. However, the conclusion "therefore dynamic is 
better" is premature. There are tons of other criteria to consider. Even 
worse, it is very possible that the 10:1 figure does not even matter: if 
static is "good enough", you may not care that dynamic is 10 times better.

Simon
-- 
DTN made easy, lean, and smart --> http://postellation.viagenie.ca
NAT64/DNS64 open-source        --> http://ecdysis.viagenie.ca
STUN/TURN server               --> http://numb.viagenie.ca

---

From: Dan Wing <[email-masked]>
Date: Thu, 4 Apr 2013 09:04:44 -0700

On Apr 3, 2013, at 7:39 PM, kaname nishizuka <[email-masked]> wrote:

> What you pointed out is reasonable enough, but our concern is that there are de-facto well-known port in 1024-32768 like 1433/tcp.
> Especially in the static assignment, the subscriber who has a "dirty" range would be attacked more frequently than others.

No more so than every subscriber is being attacked on those ports today, when subscribers are getting the entire port range.

-d

> We know that most of the NATs use the  entire port space from 1024-65536, but we described it in the most safe side.