Subject: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

From: Alexandre Petrescu <[email-masked]>
Date: Mon, 1 Oct 2018 09:38:16 +0200

Le 28/09/2018 à 19:47, Lee Howard a écrit :
> 
> 
> On 09/28/2018 01:01 PM, Alexandre Petrescu wrote:
>> For me it looks quite challenging to develop a meaningful vision in
>> this space.
>> 
>> On one hand, this Internet witnesses fast digital evolutions, and 
>> disruptions, every other day. On another hand IPv6 overcoming IPv4
>>  seems to advance very slowly. On a same timescale Windows leaped
>> from 95 to 10, while IP still steps from 4 to 6.
> Windows 95 was a significant rebuild over Windows 3.11.  Each of the
>  next several versions were incremental upgrades, and all were both 
> forward and backward compatible.
> 
> Meanwhile, lots of work had to go into making IPv6 deployable, and it
>  didn't really start until 2011. In that time, Windows has evolved
> from Windows 7 to Windows 10 (skipping 9, so only two desktop
> upgrades).
> 
>> 
>> In that sense, I think IPv6 and IPv4 side by side as a double stack
>> is still a viable evolution in the next decades, although not very
>> fancy. The translation from one to another seems like an artificial
>> path that keeps reinventing itself but with no clear advantage of
>> one translation mechanism over another. These IPv4-IPv6 translation
>> mechanisms are like satisfaction for our needs of getting there
>> faster.
> 
> How does a network run dual-stack when it's out of IPv4 addresses? If
> it buys addresses, why does it run dual-stack?

Because it wants to run both versions: one for the near term and one to
be future proof.

> I guess you consider IPv6-only plus translation to be dual-stack?

No.

I consider simultaneous IPv6-only and IPv4-only (as far as 
'simultaneous' and 'only' can be in a same phrase), i.e. 'double stack', 
to be a perfect migration mechanism.

In other words, I do not consider any form of translation or tunnelling 
or source routing to be good enough for long term migration from IPv4 to 
IPv6.

> Agreed, for now different people are choosing different mechanisms,
> and there isn't yet a clear winner.
> 
> 
>> 
>> (but I also speculated at a time that Twitter would disappear in
>> its infancy because finger existed - the reverse happened; I also
>> speculated that iPhone 4 would do _only_ IPv4 and iPhone 6 _only_
>> IPv6 - it did not happen).
 >
> I haven't seen finger enabled on a system since 1992.

Right, :-), IIRC it was disabled because of security risks of fingerd.

Maybe the comparison is too far-fetched.

>>> In other words, the development of the technologies implied by
>>> IPv6 took almost as long to develop as the technologies required
>>> to use IPv4 (and involved a few mis-steps). However, the impetus
>>> to use them was much lower - whenever a technology was
>>> unavailable to make IPv6 work, "well, you can always use IPv4".
>>> IPv6 didn't become "important" until "always" became untrue. That
>>> started in 2011, when the IPv4 address space available from the
>>> IANA (and eventually the RIRs) was depleted; it is becoming
>>> increasingly untrue as the cost of purchasing an IPv4 address
>>> spirals upward.
>> 
>> I would like to react to this.
>> 
>> We should reckon frankly what happened with this depletion.  It has
>>  been hugely advertised.  It is a few years now.
>> 
>> It depleted.  But it is still possible to get IPv4 as long as the 
>> unused space gets back.
> 
> At about US$20 each.

20 american dollars an IPv4 address, hm.

Informatively, apnic.net seems to be an order of magnitude lower:
2 australian dollars for an IPv4 address in a Class C, with membership.

Their formula, as expressed in the URL below:
IPv4: AUD 1,050 x 1.308 ^ (log2(2^(32 - requested_plen))) - 8)
IPv6: AUD 1,050 x 1.308 ^ (log2(2^(57 - requested_plen)) - 22)

http://submit.apnic.net/cgi-bin/feecalc.pl?ipv4=%2F24&ipv6=&action=Calculate

>> We can talk about refuelling IPv4.
>> 
>> Maybe we need to wait for several cycles of depletion-refuelling in
>>  order to sense a stronger need of IPv6.
> 
> 
> Right now, the cloud and CDN companies are essentially all of the
> IPv4 demand. In terms of number of addresses and dollars spent, they
> are more than 90% of the market. The CDNs are mostly okay in IPv6
> support, even if they ingest over IPv4. The cloud providers are in a
> pile of trouble. AWS is way ahead of anyone else for IPv6 support,
> and last I heard they're still buying IPv4 addresses to make sure
> they have enough cushion. Azure can do a VM or a Load Balancer, but
> not both, which is more than Google Cloud can do (only LB).

It's good to know.

Last time I checked Azure had IPv6 access with API ok, but login was
mandatory through IPv4.

> Oracle will allow you to open a ticket requesting IPv6, but unless
> it's a Service Cloud, they'll close the ticket with no action. 
> Alibaba promises an IPv6-only cloud by 2025, but so far has nothing
> but dreams.

It's good to know.

> Something like seven years ago I predicted 500M IPv4 addresses in the
>  market. About half have moved, but the market didn't really get
> moving until about 2015. It's important to note that large net blocks
> are valued by large companies: nobody wants to manage hundreds of
> /18s. Unless the US government/Department of Defense releases several
> /8s, the high end of the market is all but gone. Yes, there may be
> thousands of /16s still available, for over a million USD each and
> rising.
> 
> 2018 has seen an inflection in pricing. But of course, nobody knows
> what happens next. My concern is that instead of "permissionless
> innovation" the IPv4 Internet will require significant capital just
> to get connected.

Could be.  Could be too something about the point you mention about
difficulty in managing hundreds of IPv4 /18s.  In new network management 
tools Fragmentation could be programmed, like in disk 'defrag' tools, or 
like in software managing the Fragmentation headers of IPv6.

> We're also on track for 50% of the world to reach Google over IPv6 in
>  two years. Several mobile networks are already IPv6-only internally,
> and several large landline ISPs are either there or on their way.

YEs.

I am also happy to report that from my lab's perspective the IPv6 RTT to
google  is - now for the first time in months - lower than on IPv4 than
on IPv6.
(it's not a huge difference, but it is lower: 10ms vs 11ms)

That could be an incentive.

> So, what do you advocate we wait and see?

For my part, it is a little bit to wait: wait that users ask for it,
rather than enforcing it on users.

> How long should we wait, and what are we looking for? More
> importantly, once we see it, how much time will we have to do 
> something about it, and will it be enough time?

YEs, I trust there will be enough time to react when a massive need for 
IPv6 arises.  I base that on the experience on seeing people routinely 
fixing huge security problems in matters of days, the Y2K fixing was a 
matter of months, turning on IPv6 on some large datacenters done 
quickly, and more.

But yes, preparedness is very good.

Alex

> 
> Lee
> 
>> 
>> Alex
>> 
>>> 
>>> "Pick something else" - I have to wonder what "something else"
>>> might be in view. We have a number of research projects on the
>>> table, such as ICN/CCN/NBN, and they likely have their uses. But
>>> I don't see the architectural or business implications following
>>> through; if they were rationally available, they would have
>>> started a decade back, I suspect. So I see IPv6 deployment as the
>>> only rational option on the table.
>>> 
>>> 
>>> What I am increasingly hearing from ISPs and other networks is
>>> that it is less expensive to deploy an IPv6-only network with (or
>>> without) an IPv4 overlay  than to deploy a dual stack network.
>>> There are a few networks that have drawn that conclusion; as more
>>> and more do, IPv4 will become less of a requirement.
>>> 
>>> I'm not sure what to expect next, and maybe people can inform me.
>>> One option would be two networks that implement IPv6 natively and
>>> IPv4 as an overlay service connect, meaning that the address of a
>>> 464XLAT PLAT (or whatever) from one carrier is an address in a
>>> different carrier for some part of one of their networks. Or,
>>> maybe, a network simply stops providing an IPv4 PLAT or other
>>> support for IPv4 as an overlay. Maybe they are running lw4o6 and
>>> it becomes necessary to do IPv4 routing inside that overlay, so
>>> that they want to charge for it as a service. The obviously
>>> implied next question is "when" - I'm guessing early 2020s or
>>> something like that.
>>> 
>>> 
>>> So, what do I expect the next few years to look like? I see
>>> three broad classes of ISPs, IXPs, and networks at the edge:
>>> those that see IPv6 as the least expensive endpoint and drive to
>>> that end, those that have enough IPv4 for the moment and aren't
>>> thinking any further ahead than that, and those that are balanced
>>> on a knife's edge in between. We have those now. I expect to see
>>> a shifting balance, with different markets adopting different
>>> perspectives and changing over time. I see a slow but definitely
>>> discernible motion from "IPv4 works" to
>>> "https://www.youtube.com/watch?v=_y36fG2Oba0" "I suppose we ought
>>> to give it a try". One at a time, I hear operators say "I don't 
>>> want to buy any more IPv4 addresses". The networks that are fixed
>>> in the "IPv4 laggard" camp tend to be networks that aren't faced
>>> with IPv4 address purchases and ask for a business justification
>>> that doesn't involve a monetary outlay.
>>> 
>>> Ron, Lee, and I asked for a change to the v6ops charter to
>>> discuss IPv6-only networking with that in view. What I want to
>>> see in v6ops is more of that - lets move IPv4 to an overlay
>>> service, and watch it become less relevant over time.
>>> 
>>> So I personally expect transition technologies of various kinds
>>> - lw4o6, 464xlat, MAP-E, MAP-T, and so on - to remain important,
>>> but I expect that they will become the aftermarket more than the
>>> market. If cloud services are accessible by IPv6, I expect
>>> residential access to become IPv6 service to become larger and
>>> IPv4 to become smaller. If you want a picture of that, look at
>>> Trinidad and Tobago. As I understand it, there are some services
>>> (Google, Facebook, etc) that only offer IPv6 access there, so the
>>> "IPv6" picture is a picture of their market dominance. 
>>> https://stats.labs.apnic.net/ipv6/CC?x=1&s=1&p=1&w=30&c=TT
>>> 
>>> I expect service providers to do as little with IPv4 as they can
>>> get away with, and for IPv4 to increasingly become ragged around
>>> the edges. There are still a variety of places where I would get
>>> tarred and feathered and sent home wearing a barrel for saying
>>> that, but the SNA people were of that opinion too - until APPN
>>> made SNA networks irrelevant and people started deploying
>>> IP-native applications. 
>>> --------------------------------------------------------------------------------
>>> 
>>> 
>>> 
>>> 
>> Victorious warriors win first and then go to war,
>>> Defeated warriors go to war first and then seek to win. Sun Tzu
>>> 
>>> 
>>> 
>>> _______________________________________________ v6ops mailing
>>> list [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
>>> 
>> 
>> _______________________________________________ v6ops mailing list 
>> [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
> 
> _______________________________________________ v6ops mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 1 Oct 2018 10:22:18 +0200 (CEST)

On Wed, 26 Sep 2018, Fred Baker wrote:

> I expect service providers to do as little with IPv4 as they can get 
> away with, and for IPv4 to increasingly become ragged around the edges. 
> There are still a variety of places where I would get tarred and 
> feathered and sent home wearing a barrel for saying that, but the SNA 
> people were of that opinion too - until APPN made SNA networks 
> irrelevant and people started deploying IP-native applications.

My take on this is that different things will happen in different markets, 
and I think you're expectations are correct.

We'll see highly standardised sectors choose their access 
protocols/mechanisms between them and customer device (NAT64 and 464XLAT 
(or other mapping) for mobile, ds.lite for DOCSIS) and go with that.

We'll see markets (differs per country and ISP) where the ISP provides the 
residential gateway, and they choose some mechanisms (MAP/LW4o6 or 
similar) for IPv6 only as they see fit when it fits their deployment 
plans.

Other markets will stay in dual stack WAN, probably because they need to 
support electronics-store purchased generic devices, and they don't 
control what the customer does with their device. It's not uncommon 
already with NAT444 (or more 4:s), and this will continue to be common.

ISPs will need to provide IPv4 Internet reachability for decades to come, 
and we'll need multiple ways to do it because of the different sector 
choices.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Ackermann, Michael" <[email-masked]>
Date: Sat, 6 Oct 2018 15:23:57 +0000

Fred
A big +1 on your great comments below. 

I especially like your last sentence,  which is a keeper for me!      The itemized charging for separate protocols and required services is probably the #1 thing that will finally get the attention of us Enterprises Ostriches,  that currently have our heads in the ground on these issues. 
Thanks again!
Mike

-----Original Message-----
From: v6ops <[email-masked]> On Behalf Of Fred Baker
Sent: Friday, September 28, 2018 2:26 PM
To: Alexandre Petrescu <[email-masked]>
Cc: [email-masked]
Subject: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

On Sep 28, 2018, at 10:01 AM, Alexandre Petrescu <[email-masked]> wrote:
> In that sense, I think IPv6 and IPv4 side by side as a double stack is 
> still a viable evolution in the next decades, although not very fancy.

As I theorized the other day, I suspect that the IPv4 Internet becomes something strictly at the edge. It might very well be dual stack within the edge network, but it will come to a NAT of some kind before it leaves the premises. The ISPs run IPv6-only, and they serve residential and mobile customers/services using it. CLAT-PLAT translation means that the ISP doesn't have to think about it too much; lw4o6 tunnels become a network of their own that someone has to manage. As much as I dislike LISP, LISP might be a rational tunneling technology; it allows for the dynamic creation of tunnels whenever and wherever needed, which would be useful for enterprise connection to its hosting sites and business partners.

Also, I suspect that said aftermarket business becomes a PITA that people charge to support. Given the choice of a bill for one protocol or a bill for two separate protocols with some sort of PLAT or tunnel endpoint support built in, expect customers to stop asking about a business case and start asking about reducing their costs.

--------------------------------------------------------------------------------
Victorious warriors win first and then go to war, Defeated warriors go to war first and then seek to win.
     Sun Tzu

The information contained in this communication is highly confidential and is intended solely for the use of the individual(s) to whom this communication is directed. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information is prohibited. Please notify the sender, by electronic mail or telephone, of any unintended receipt and delete the original message without making any copies.
 
 Blue Cross Blue Shield of Michigan and Blue Care Network of Michigan are nonprofit corporations and independent licensees of the Blue Cross and Blue Shield Association.

---

From: Yannis Nikolopoulos <[email-masked]>
Date: Mon, 15 Oct 2018 09:57:05 +0300

> and there isn't yet a clear winner. 

I've had this in my mailbox for days/weeks now. I just wanted to write 
"are they? are they really?" . I don't see people choosing mechanisms, 
people are not choosing mechanisms.People are deploying CGN en masse, 
people are buying IPv4 addresses under the table or people are just not 
touched by the IPv4 exhaustion. That is what people are doing, at least 
most of them. And I wonder: what went wrong? 3-4 years ago, I was 
willing to bet that by 2017, fixed ISPs would've switched to MAP or 
lw4o6. How wrong was I... I feel very disappointed .

We switched to dual-stack about 10 years ago. 4 years ago, we started 
considering our IPv6-only service options and we've been running lw4o6 
for the past 2 years. Our lw4o6 deployment is pretty small (40k users 
out of 1.5M) still because of various issues: CPE is a pain, DHCP is a 
pain, even lwaftr is a pain (more info can be provided on request). Most 
importantly, we do not see the economy at scale here. CGN (NAT444) is 
strangely cheaper and troubles-free. On top of that, we're about to test 
it with IPv6 support.

I'm called to present a case to mgmt and I despise the fact that I've 
ran out of arguments for supporting an IPv6-only residential service: 
the only one I have left is the trouble-free native IPv6 path, but that 
doesn't really hold up since the percentage of complaining users is so 
small. So, it comes down to that: we're about to have our lw4o6 
deployment shut and our CGN is almost consuming half our userbase 
already. The future is bleak, indeed. I've even considered 464xlat for 
everyone (since we're also a mobile carrier, where dual-stack and CGN 
are deployed)

thanks for listening,
Yannis

---

From: Alexandre Petrescu <[email-masked]>
Date: Mon, 15 Oct 2018 09:52:10 +0200

Le 15/10/2018 à 08:57, Yannis Nikolopoulos a écrit :
> I'm called to present a case to mgmt

The case is the following: because IPv6 deployment is growing with low 
speed it is called a migration from IPv4 to IPv6; it is not a total 
switch like from Windows 97 to Windows 10.

The case for operator is just to be ready to listen more and more end 
users requesting IPv6, but slowly.

Is it difficult to just be ready?

> and I despise the fact that I've 
> the only one I have left is the trouble-free native IPv6 path,

No no, IPv6 native is not trouble free.  There are tons of in-house end 
user devices that prefer IPv4.

Just wait for them to ask you, and certainly do not feed them IPv6 by force.

Alex

  but that
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Mon, 15 Oct 2018 12:16:18 +0200

Hi Yannis,

The only think I don't agree is that CGN (NAT444) is not troubles-free!

Also, the cost of CGN is much higher than buying IPv4 addresses, and keep doing dual-stack, but Sonner or later, you will need to do something when you can't buy anymore IPv4 addresses ... or become too expensive.

People investing in CGN, don't realize that it is a very bad decision. Because more and more content/application providers (like Sony Play Station Network), black-list your IPv4 CGN ranges, sooner or later you need again to buy IPv4 addresses ... And, wait a minute ... have you invested in CGN because this way you don't need to buy IPv4 addresses? Bad decision!

Regards,
Jordi

-----Mensaje original-----
De: v6ops <[email-masked]> en nombre de Yannis Nikolopoulos <[email-masked]>
Fecha: lunes, 15 de octubre de 2018, 8:57
Para: Lee Howard <[email-masked]>, <[email-masked]>
Asunto: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

    

    thanks for listening,
    Yannis
    
    _______________________________________________
    v6ops mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/v6ops

**********************************************
IPv4 is over
Are you ready for the new Internet ?
http://www.consulintel.es
The IPv6 Company

This electronic message contains information which may be privileged or confidential. The information is intended to be for the exclusive use of the individual(s) named above and further non-explicilty authorized disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited and will be considered a criminal offense. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited, will be considered a criminal offense, so you must reply to the original sender to inform about this communication and delete it.

---

From: Yannis Nikolopoulos <[email-masked]>
Date: Mon, 15 Oct 2018 13:43:13 +0300

HI Alexandre,

On 10/15/18 10:52 AM, Alexandre Petrescu wrote:
>> I've had this in my mailbox for days/weeks now. I just wanted to 
>> write "are they? are they really?" . I don't see people choosing 
>> mechanisms, people are not choosing mechanisms.People are deploying 
>> CGN en masse, people are buying IPv4 addresses under the table or 
>> people are just not touched by the IPv4 exhaustion. That is what 
>> people are doing, at least most of them. And I wonder: what went 
>> wrong? 3-4 years ago, I was willing to bet that by 2017, fixed ISPs 
>> would've switched to MAP or lw4o6. How wrong was I... I feel very 
>> disappointed .
>>
>> considering our IPv6-only service options and we've been running 
>> lw4o6 for the past 2 years. Our lw4o6 deployment is pretty small (40k 
>> users out of 1.5M) still because of various issues: CPE is a pain, 
>> DHCP is a pain, even lwaftr is a pain (more info can be provided on 
>> request). Most importantly, we do not see the economy at scale here. 
>> CGN (NAT444) is strangely cheaper and troubles-free. On top of that, 
>> we're about to test it with IPv6 support.
>>

No, we've been ready for almost a decade. Also, the amount of end users 
(residential or business) requesting IPv6 is really negligible

>
>> and I despise the fact that I've ran out of arguments for supporting 
>> an IPv6-only residential service: the only one I have left is the 
>> trouble-free native IPv6 path,
>
> No no, IPv6 native is not trouble free.  There are tons of in-house 
> end user devices that prefer IPv4.

correct but our basic residential service is dual-stack

regards,
Yannis

-- 
Yannis Nikolopoulos		OTE S.A
e-mail: [email-masked]	IP Network Planning & Engineering
tel: +302106116293
----------------------------------------------------------------------------

---

From: Yannis Nikolopoulos <[email-masked]>
Date: Mon, 15 Oct 2018 13:57:09 +0300

Hello Jordi,

> The only think I don't agree is that CGN (NAT444) is not troubles-free!

as far as maintaining it, it is trouble-free. Some end users face all 
the known address sharing issues but as I wrote earlier, the percentage 
is very small and all those users are just rolled back to the basic 
dual-stack service.

>
> Also, the cost of CGN is much higher than buying IPv4 addresses, and keep doing dual-stack, but Sonner or later, you will need to do something when you can't buy anymore IPv4 addresses ... or become too expensive
>

In our case, we did not invest in CGN. We invested in preparing for IPv6 
early, and so we did. When IPv4 exhaustion hit us, we did not decide to 
invest in either CGN or buying IPv4 addresses. Again , we invested in 
IPv6 (lw4o6). In the mean time between deploying lw4o6, we had to 
mitigate the issue, in a quick n dirty manner... that's when CGNcame 
into the picture. Some years later, we have a lot of CGN users and not 
so many in IPv6-only (lw4o6).  It just seems that, IPv6-only 
technologies (lw4o6 in our case) were not so straighforward to deploy 
and more importantly, not so cheap (performance-wise) either. Maybe the 
"stateless card" has been overplayed or exaggerated at the very least.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 15 Oct 2018 13:09:32 +0200 (CEST)

> (residential or business) requesting IPv6 is really negligible

Well, I would imagine that the amount of end users requesting IPv4 is also 
negligible.

Users want to access cat pictures ie "working Internet access". In order 
to grow the Internet, we need IPv6 to work, and it's up to us on the 
technical part to make sure the Internet continues working. This means 
over time migrate all of our users to IPv6.

Anyone waiting for customers to demand IPv6 access is waiting for the 
wrong thing, it's like an airline waiting for the customers to demand that 
their engine fan blades would be made of a special alloy. Most customers 
do not know, don't want to know, they just want things to work.

---

From: Alexandre Petrescu <[email-masked]>
Date: Mon, 15 Oct 2018 13:10:27 +0200

Le 15/10/2018 à 12:43, Yannis Nikolopoulos a écrit :
>>>> Agreed, for now different people are choosing different
>>>> mechanisms, and there isn't yet a clear winner.
>>> 
>>>  mechanisms, people are not choosing mechanisms.People are
>>> deploying CGN en masse, people are buying IPv4 addresses under
>>> the table or people are just not touched by the IPv4 exhaustion.
>>> That is what people are doing, at least most of them. And I
>>> wonder: what went wrong? 3-4 years ago, I was willing to bet that
>>> by 2017, fixed ISPs would've switched to MAP or lw4o6. How wrong
>>> was I... I feel very disappointed .
>>> 
>>> We switched to dual-stack about 10 years ago. 4 years ago, we
>>> started considering our IPv6-only service options and we've been
>>> running lw4o6 for the past 2 years. Our lw4o6 deployment is
>>> pretty small (40k users out of 1.5M) still because of various
>>> issues: CPE is a pain, DHCP is a pain, even lwaftr is a pain
>>> (more info can be provided on request). Most importantly, we do
>>> not see the economy at scale here. CGN (NAT444) is strangely
>>> cheaper and troubles-free. On top of that, we're about to test it
>>> with IPv6 support.
>>> 
>> The case is the following: because IPv6 deployment is growing with
>> low speed it is called a migration from IPv4 to IPv6; it is not a
>> total switch like from Windows 97 to Windows 10.
>> 
>> The case for operator is just to be ready to listen more and more
>> end users requesting IPv6, but slowly.
>> 
> No, we've been ready for almost a decade.

:-)

It sounds as being disappointed that a decade passed and the next
generation protocol is still not widespread.

But it was the first such decade, and much thrilling we witnessed: 6to4, 
6over4, teredo, ... 464xlat.  The next decade would slow down, and maybe
subsequent decades will again accelerate deployment.

For comparison, in AI 6 decades passed since Turing came up with a test
and until a computer won at chess; and still today most humans win.

> Also, the amount of end users (residential or business) requesting
> IPv6 is really negligible

True, it is negligible but it is not absent.

If one wants to tempt more users into IPv6 then one has to:

- tempt hardware and OS manufacturers into upgrading to IPv6 the
   devices.
- tempt the innovators in other fields (IoT, AI, biotech, etc.) to
   prefer IPv6.  Too many new concepts and products come up routinely new
   in the market that prefer IPv4 instead.  The other day I saw these
   Epson futuristic glasses.
- tempt core ISP give more bandwidth to IPv6 links than to IPv4 links.
   When 100Tbit/s links come up they should ping IPv6 only.

>>> and I despise the fact that I've ran out of arguments for
>>> supporting an IPv6-only residential service: the only one I have
>>> left is the trouble-free native IPv6 path,
>> 

Which makes me ask you: are your links to residential networks giving
more bandwidth to IPv6?  Or the same?  (in my lab, my ISP gives more
slightly more bandwidth to IPv6: ping6 google.com has slightly lower RTT
than IPv4; this IPv6 favoring is only recent).

Alex

> regards, Yannis

---

From: Alexandre Petrescu <[email-masked]>
Date: Mon, 15 Oct 2018 13:14:27 +0200

Le 15/10/2018 à 13:09, Mikael Abrahamsson a écrit :
>> No, we've been ready for almost a decade. Also, the amount of end 
>> users (residential or business) requesting IPv6 is really negligible
> 
> Well, I would imagine that the amount of end users requesting IPv4 is 
> also negligible.
> 
> wrong thing, it's like an airline waiting for the customers to demand 
> that their engine fan blades would be made of a special alloy. Most 
> customers do not know, don't want to know, they just want things to work.

I agree with you :-)

End users ask for more bandwidth.  Make IPv6 give more bandwidth than IPv4.

That's how it works with 3G, 4G, 5G.

Alex

---

From: Lencse Gábor <[email-masked]>
Date: Mon, 15 Oct 2018 13:17:14 +0200

Dear Yannis,

10/15/2018 12:57 PM keltezéssel, Yannis Nikolopoulos írta:

[...]

> It just seems that, IPv6-only technologies (lw4o6 in our case) were 
> not so straighforward to deploy and more importantly, not so cheap 
> (performance-wise) either. Maybe the "stateless card" has been 
> overplayed or exaggerated at the very least.
>

If you have some experience, which you would like to share with us, we 
would be happy to include the lessons you have learnt into our I-D 
titled "Pros and Cons of IPv6 Transition Technologies for IPv4aaS", 
https://tools.ietf.org/html/draft-lmhp-v6ops-transition-comparison-00

Best regards,

Gábor Lencse

---

From: Yannis Nikolopoulos <[email-masked]>
Date: Mon, 15 Oct 2018 21:35:46 +0300

> If one wants to tempt more users into IPv6 then one has to: 
that's the point though, we're not looking to tempt more users into 
IPv6. We're trying to find a viable case against scrapping our IPv6-only 
efforts

---

From: Alexandre Petrescu <[email-masked]>
Date: Tue, 16 Oct 2018 10:49:41 +0200

> IPv6. We're trying to find a viable case against scrapping our IPv6-only 

That is also a good idea.

Alex

---

From: Yannis Nikolopoulos <[email-masked]>
Date: Tue, 16 Oct 2018 15:09:55 +0300

On 10/16/18 11:49 AM, Alexandre Petrescu wrote:
>> IPv6. We're trying to find a viable case against scrapping our 
>> IPv6-only efforts
>

what is? Scrapping our IPv6-only efforts? :)

---

From: Alexandre Petrescu <[email-masked]>
Date: Tue, 16 Oct 2018 22:43:07 +0200

> what is? Scrapping our IPv6-only efforts? :)

(I have to understand what 'scrapping' means).

But it is a good idea to try to make a viable case for your existing 
IPv6-only efforts.

A thread for the case could be: bind a bandwidth notion to the protocol 
version notion.  Plan for 100Tbit/s links that support IPv6.  Promiss 
more bandwidth, that accidentally involves IPv6.

Bind it to Ethernet as well because that is a guaranteed success too 
compared to ATM.  Ethernet to the home.  Not pppoe, because that reduces 
bandwidth.

(the number 100Tbit/s is made up, but there must exist an equivalent 
that is fashionable these days).

Alex

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Tue, 16 Oct 2018 21:42:27 +0000

> (I have to understand what 'scrapping' means).

scrapping = trashing (get rid of)

> But it is a good idea to try to make a viable case for your existing IPv6-only
> efforts.
> 
> version notion.  Plan for 100Tbit/s links that support IPv6.  Promiss more
> bandwidth, that accidentally involves IPv6.
> 
> (the number 100Tbit/s is made up, but there must exist an equivalent that is
> fashionable these days).

I don't think this addresses the question (as I understand it).
Given an ISP with existing, reasonably problem-free dual stack deployment, what incentive is there to move the IPv4 from dual stack to being tunneled over IPv6?
Barbara
 
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_v6ops&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=LoGzhC-
> 8sc8SY8Tq4vrfog&m=MZXCE8bKs2k0wvjCX8A00Oozdtu8isSZl64lZ7L67z8&s=J
> qZ5eoT2vWnzhSPKeDl9YR_pC_l9TyHfAy6BiIylGEw&e=

---

From: Fred Baker <[email-masked]>
Date: Tue, 16 Oct 2018 15:36:35 -0700

> On Oct 16, 2018, at 1:43 PM, Alexandre Petrescu <[email-masked]> wrote:
> 
> I have to understand what 'scrapping' means

https://dictionary.cambridge.org/dictionary/english/scrap

---

From: Yannis Nikolopoulos <[email-masked]>
Date: Wed, 17 Oct 2018 10:22:04 +0300

Hello Barbara and  thank you for your comment,

> Given an ISP with existing, reasonably problem-free dual stack deployment, what incentive is there to move the IPv4 from dual stack to being tunneled over IPv6?

pretty simple incentive: public IPv4 address exhaustion

We're no longer able to provide our users with public IPv4 addresses, we 
havrn't been for the past 3 years. That is why we're deploying lw4o6 and 
that's why we have a NAT444 CGN in place until our lw4o6 deployment is 
problem-free and ready for carrier grade. Unfortunately, our CGN is not 
that expensive compared to lw4o6 and lw4o6 does not seem to deliver 
(performance-wise). And (nearly) nobody seems to be deploying IPv4aaS 
and we're out of arguments towards management

regards,
Yannis

---

From: Lencse Gábor <[email-masked]>
Date: Wed, 17 Oct 2018 09:54:18 +0200

2018.10.17. 9:22 keltezéssel, Yannis Nikolopoulos írta:
> We're no longer able to provide our users with public IPv4 addresses, 
> we havrn't been for the past 3 years. That is why we're deploying 
> lw4o6 and that's why we have a NAT444 CGN in place until our lw4o6 
> deployment is problem-free and ready for carrier grade. Unfortunately, 
> our CGN is not that expensive compared to lw4o6 and lw4o6 does not 
> seem to deliver (performance-wise). And (nearly) nobody seems to be 
> deploying IPv4aaS and we're out of arguments towards management
>

What about the costs of maintaining a dual stack network? (E.g. IPv4 
routing table will grow due to trading with public IPv4 addresses. 
Management will have to shell out a huge sum of money for memory update 
or replacement of the routers.)

Gábor

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 17 Oct 2018 09:54:47 +0200 (CEST)

> and we're out of arguments towards management

Isn't the single NAT44 (port block allocation) an argument here? Or you 
have very few complaints about the NAT444 solution?

---

From: Yannis Nikolopoulos <[email-masked]>
Date: Wed, 17 Oct 2018 12:57:11 +0300

Hi,

On 10/17/18 10:54 AM, Mikael Abrahamsson wrote:
> Isn't the single NAT44 (port block allocation) an argument here? Or 
> you have very few complaints about the NAT444 solution?
>

we have complaints, but the situation is manageable. We just roll them 
back to the dual-stack service

---

From: Lee Howard <[email-masked]>
Date: Thu, 18 Oct 2018 09:08:58 +0200

Chairs,

I'd be very interested in a lw4over6 experience report from Yannis.

Yannis, I'm interested in hearing more about what isn't working well, 
either on-line or directly.

Lee

On 10/15/2018 12:57 PM, Yannis Nikolopoulos wrote:
>
> In our case, we did not invest in CGN. We invested in preparing for 
> IPv6 early, and so we did. When IPv4 exhaustion hit us, we did not 
> decide to invest in either CGN or buying IPv4 addresses. Again , we 
> invested in IPv6 (lw4o6). In the mean time between deploying lw4o6, we 
> had to mitigate the issue, in a quick n dirty manner... that's when 
> CGNcame into the picture. Some years later, we have a lot of CGN users 
> and not so many in IPv6-only (lw4o6).  It just seems that, IPv6-only 
> and more importantly, not so cheap (performance-wise) either. Maybe 
> the "stateless card" has been overplayed or exaggerated at the very 
> least.
>
>>
>> Regards,
>> Jordi
>>
>>      > Agreed, for now different people are choosing different 
>> mechanisms,
>>      > and there isn't yet a clear winner.
>> lw4o6
>> This electronic message contains information which may be privileged 
>> or confidential. The information is intended to be for the exclusive 
>> use of the individual(s) named above and further non-explicilty 
>> authorized disclosure, copying, distribution or use of the contents 
>> of this information, even if partially, including attached files, is 
>> strictly prohibited and will be considered a criminal offense. If you 
>> are not the intended recipient be aware that any disclosure, copying, 
>> distribution or use of the contents of this information, even if 
>> partially, including attached files, is strictly prohibited, will be 
>> considered a criminal offense, so you must reply to the original 
>> sender to inform about this communication and delete it.
>>
>>
>>
>>
>

---

From: Lee Howard <[email-masked]>
Date: Thu, 18 Oct 2018 09:26:18 +0200

On 10/15/2018 08:57 AM, Yannis Nikolopoulos wrote:
> people are not choosing mechanisms.

I think this has been posted here before. It's crowd-sourced, so it has 
all the strengths and weaknesses of that.
https://docs.google.com/spreadsheets/d/1ksOoWOaRdRyjZnjLSikHf4O5L1OUTNOO_7NK9vcVApc/edit#gid=0

So, yes, people are using transition mechanisms.
I know of two other large fixed-line ISPs who are less than a year from 
deploying.

> People are deploying CGN en masse, people are buying IPv4 addresses 
> under the table or people are just not touched by the IPv4 exhaustion. 
> That is what people are doing, at least most of them. And I wonder: 
> what went wrong? 3-4 years ago, I was willing to bet that by 2017, 
> fixed ISPs would've switched to MAP or lw4o6. How wrong was I... I 
> feel very disappointed .
>
> pain, even lwaftr is a pain (more info can be provided on request). 
> Most importantly, we do not see the economy at scale here. CGN 
> (NAT444) is strangely cheaper and troubles-free. On top of that, we're 
> about to test it with IPv6 support.
>
> the only one I have left is the trouble-free native IPv6 path, but 
> that doesn't really hold up since the percentage of complaining users 
> is so small. So, it comes down to that: we're about to have our lw4o6 
I'd be interested in how many flows and bits use CGN vs. IPv6, and how 
that's been changing over the years. For management, that will point to 
how small a box (whether NAT44, lw4over6, or NAT64) is needed for 
translation. They would surely love to know when they can stop spending 
money on translators. There's been some promising growth in Greece 
recently: https://www.vyncke.org/ipv6status/project.php?metric=w&country=gr

In my experience, trouble calls are expensive, partly because sometimes 
instead of a trouble call you get a customer cancellation. That's 
probably a cost you can present to management.

You can also note that in at least one experiment, access to 
IPv6-enabled websites is 9ms faster than to IPv6-incapable web sites. 
https://stats.labs.apnic.net/v6perf/GR
That might be worth mentioning to the large IPv6-incapable web sites, too.

So I wonder if your case to management is partly that you guessed too 
early (as we all did with IPv4 depletion, and as some of us often do 
with IPv6 deployment). The question is not how much money has already 
been spent, it's whether, looking forward 2-5 years, your money is 
better spent on CGN or IPv4aaS.

Lee

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 09:33:38 +0200 (CEST)

>From my experience, the biggest problem is the same as Yannis is 
presenting, it's all about HGW support for these things. The major vendors 
support 6to4, 6RD and ds.lite (plus regular dual stack). Then it's 
problematic.

One of the major SoC vendors had a packet accelerator problem with LW4o6 
in that the accelerator "forgot" to NAT44 the packets when it took over 
the flow. I spent 9 months getting nowhere until finally I managed the use 
threats to get through to some engineering people and explain my case. 
They fixed the problem 2 weeks later. They said they didn't first 
understand what I wanted, and basically had asked the wrong question 
internally and then probably got a "err, no, not justified by the business 
case".

But now I have a fully working, fully accelerated LW4o6-supporting gigabit 
ethernet (can actually do that) HGW for LW4o6 use-case on active-ethernet.

Someone else might want to do this over PPPoE, so then you might be in 
trouble again because the accelelator might not support the 
LW4o6-over-IPv6-over-PPPoE case. I don't know, I haven't tested it because 
in my deployment we don't do PPPoE.

So the problems aren't really technical, they're all business related. The 
vendors are doing pushback on all these different versions of doing pretty 
much the same thing, that all require development resources.

So let me list some of the common use-cases the IETF/ISPs basically is 
asking them to support.

6to4
ds.lite
LW4o6
MAP(-e and -t)
6RD

all of the above over:

PPPoE over DSL, PON and ethernet
PPPoA over DSL
Native Ethernet

.... plus probably something else I forgot. That's tens of different 
combinations for doing what looks to be almost the same thing. Some others 
also expect them to have multiple VLANs on WAN, because that's how the 
operator likes to deply their IPTV and VoIP service.

The linux kernel does all of the above, but the packet accelerators 
typically in these kinds of devices do not support all of it. There are 
lots of different platforms, and all customers have whole departments that 
are penny-pinching price on each device when doing procurement.

Things are improving as the business ecosystem is moving to more common 
operating systems and fewer platforms, but it's still an issue, and it's 
going to be an issue for a long time.

The vendors don't typically jump on whatever new stuff the IETF came up 
with last month, I've heard some who say "we typically wait 2-3 years 
until we consider implementing an RFC to see if it actually gets some 
support".

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Thu, 18 Oct 2018 09:45:19 +0200

I guess you're forgetting the CLAT, which is the simpler one to implement as part of the packet accelerator.

All what you mention is the reason I started the work on draft-palet-v6ops-transition-ipv4aas, which unfortunately take much longer than expected.

We need to point the SoC vendors, not just the CE vendors, to this document.

Regards,
Jordi

-----Mensaje original-----
De: v6ops <[email-masked]> en nombre de Mikael Abrahamsson <[email-masked]>
Organización: People's Front Against WWW
Fecha: jueves, 18 de octubre de 2018, 9:34
Para: Lee Howard <[email-masked]>
CC: <[email-masked]>
Asunto: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

    

    

    
    6to4
    ds.lite
    LW4o6
    MAP(-e and -t)
    6RD
    
    all of the above over:

    ..... plus probably something else I forgot. That's tens of different 
    combinations for doing what looks to be almost the same thing. Some others 
    also expect them to have multiple VLANs on WAN, because that's how the 
    operator likes to deply their IPTV and VoIP service.

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 10:10:16 +0200

Jordi,

> I guess you're forgetting the CLAT, which is the simpler one to implement as part of the packet accelerator.

Can you substantiate that? By what metric.
If you look at number of lines of code, amount of state, or even provisioning complexity you are wrong.

Ole

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Thu, 18 Oct 2018 10:22:05 +0200

Hi Ole,

I've posted some time ago a comparison in terms of code size in OpenWRT comparing all the IPv4aaS, not going to repeat that, just for short, CLAT code size in OpenWRT is 4 Kbytes: https://openwrt.org/packages/pkgdata/464xlat

I can see the code simplicity at:

https://github.com/openwrt-routing/packages/blob/master/nat46/files/464xlat.sh

If you remove the comment lines, is less than 100 very simple code lines.

Of course, there are other implementations, all can be googled for, such as Jool, Android, VPP and the one from Tore Anderson.

Regards,
Jordi

-----Mensaje original-----
De: Ole Troan <[email-masked]>
Fecha: jueves, 18 de octubre de 2018, 10:10
Para: JORDI PALET MARTINEZ <[email-masked]>
CC: Mikael Abrahamsson <[email-masked]>, Lee Howard <[email-masked]>, <[email-masked]>
Asunto: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

    Jordi,

    
    Ole

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 10:37:29 +0200

Jordi,

So the metric you use to justify your claim is code size?

> Of course, there are other implementations, all can be googled for, such as Jool, Android, VPP and the one from Tore Anderson.

But if your claim had been correct, then the other mechanisms would have had more code lines right?
And you do need to include the whole packet path. And provisioning.

Either you substantiate your claim, or you stop making it. I would suggest the latter would save both you, me and the list some time.

Ole

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Thu, 18 Oct 2018 10:59:12 +0200

Ole,

I stated clearly CLAT not the complete 464XLAT.

Regards,
Jordi

-----Mensaje original-----
De: Ole Troan <[email-masked]>
Fecha: jueves, 18 de octubre de 2018, 10:37
Para: JORDI PALET MARTINEZ <[email-masked]>
CC: Mikael Abrahamsson <[email-masked]>, Lee Howard <[email-masked]>, <[email-masked]>
Asunto: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

    Jordi,

    

    Ole

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 11:29:40 +0200

Jordi,

> I stated clearly CLAT not the complete 464XLAT.

Packet path for the CE. Including in the case of CLAT the two stateful NATs. SIIT etc.
Another metric is to measure the number of cycles used to process the packet. Compare CLAT processing and non-translation solutions.
CLAT comes worse off both in matter of cycles and lines of code, and likely provisioning.
Enough to care… who knows. My point was, just stop spreading incorrect information. You can not claim that CLAT is simpler/cheaper/faster to implement than the others.

Ole

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 11:55:00 +0200 (CEST)

On Thu, 18 Oct 2018, Ole Troan wrote:

> Enough to care… who knows. My point was, just stop spreading incorrect 
> information. You can not claim that CLAT is simpler/cheaper/faster to 
> implement than the others.

I agree. The HGW platform I talked about does 200 megabit/s of packets in 
pure CPU, around 800-1000 megabit/s of traffic with some functionality 
turned on (checksumming and some other things then done in SoC/NIC instead 
of purely by CPU), and then bidirectional wirespeed with small packets 
with full acceleration / flow offload (2000 megabit/s with smaller than 
1500 byte packets).

I have no idea if the packet accelerator in its current form would be able 
to handle the CLAT functionality. I would doubt it until I see it working 
in practice.

---

From: Ca By <[email-masked]>
Date: Thu, 18 Oct 2018 06:31:02 -0700

> to handle the CLAT functionality. I would doubt it until I see it working

Tl,dr; CLAT likely does not need acceleration since it should not see much
traffic.

The reason CLAT is so enjoyable in my deployment of 464xlat with DNS64 is
that CLAT only handles very few flows that use ipv4 literals or force ipv4
socket by other means.

In the home, their is a great varitety of things that behave in many ways,
but i would have to assume the typical windows 7+ / ios10+ home at this
point is mostly ipv6 traffic. We certainly see Google’s overall number is
35% in the usa, and most of that traffic is Youtube, FB, Netflix ... with
v6 ... and many things ask for AAAA and could receive it and therefore
leverage standard ipv6 routing on the CPE with no special treatment at the
edge

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 15:56:02 +0200 (CEST)

On Thu, 18 Oct 2018, Ca By wrote:

> Tl,dr; CLAT likely does not need acceleration since it should not see 
> much traffic.

That's fine. Unfortunately the failure mode of the accelerator might be 
that it treats the flow wrong and doesn't do what it's supposed to do. It 
"takes" the flow but then doesn't do the needful.

And the only way to easily fix it is to turn off the entire accelerator, 
device-wide.

---

From: Lee Howard <[email-masked]>
Date: Thu, 18 Oct 2018 16:36:42 +0200

On 10/18/2018 03:56 PM, Mikael Abrahamsson wrote:
> That's fine. Unfortunately the failure mode of the accelerator might 
> be that it treats the flow wrong and doesn't do what it's supposed to 
> do. It "takes" the flow but then doesn't do the needful.

Your argument is that somebody might implement it wrong?
Doesn't that apply to literally everything?

>
> And the only way to easily fix it is to turn off the entire 
> accelerator, device-wide.
>
If a process black-holes your packets, you should turn it off. Does that 
really mean you have to turn off acceleration?

Lee

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 18:31:47 +0200 (CEST)

> Your argument is that somebody might implement it wrong?

Nope.

> Doesn't that apply to literally everything?

Yes.

> really mean you have to turn off acceleration?

That's not how it works. This is not a "process". This is HW built into 
the SoC that conflates lots of things, including NIC, L2 switch, 
processor, and heaps of other things.

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Thu, 18 Oct 2018 18:46:10 +0200

Agree, there are better ways to do that.

I've been in an equivalent situation recently.

Cheap CE (5 Gig ports with HW configurable switch as LAN/WAN, etc.) with SoC Mediatek MT7621, using OpenWRT. This SoC supports up to 2Gbps IPv4/6 routing, NAT, NAPT+HQoS with hardware offloading.

In this scenario there was also support for IPTV and VoIP. Both require specific VLANs with their own NAT44. Also, the main Internet VLAN, using PPPoE, and or course NAT44 is also needed.

For some reason (I'm guessing it is an implementation problem of the IPTV and VoIP), if you apply a "global" hardware Routing/NAT Offloading rule, just choosing that option in the OpenWRT GUI, IPTV and VoIP, stop working. You can still make outgoing phone calls, but incoming ones don't get in (which once more seems to indicate that it is an implementation or may be even configuration problem). Internet data works fine.

The solution is easy, you can just apply a specific firewall rule to enable the hw offloading only for the Internet traffic, which is the one that will have a higher bw demand:
iptables -A forwarding_rule -i <lan_if> -o <wan_if> -m conntrack --ctstate RELATED,ESTABLISHED -j FLOWOFFLOAD --hw
ip6tables -A forwarding_rule -i <lan_if> -o <wan_if> -m conntrack --ctstate RELATED,ESTABLISHED -j FLOWOFFLOAD --hw

So, in the case of different interfaces for the transition mechanisms (as it is the case in OpenWRT), you can also do the same if there is any issue in one of the implementations of a specific IPv4aaS interface. Of course, ideally, you resolve the implementation problem, but meanwhile you're not stuck with the "main" traffic not being able to use the offloading.

Saludos,
Jordi

-----Mensaje original-----
De: v6ops <[email-masked]> en nombre de Lee Howard <[email-masked]>
Fecha: jueves, 18 de octubre de 2018, 16:37
Para: <[email-masked]>
Asunto: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

    

    Lee

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 19:23:34 +0200 (CEST)

On Thu, 18 Oct 2018, JORDI PALET MARTINEZ wrote:

> The solution is easy, you can just apply a specific firewall rule to 
> enable the hw offloading only for the Internet traffic, which is the one 
> that will have a higher bw demand:

> ip6tables -A forwarding_rule -i <lan_if> -o <wan_if> -m conntrack --ctstate RELATED,ESTABLISHED -j FLOWOFFLOAD --hw

Errr, are you sure FLOWOFFLOAD even does HW offload? On most platforms it 
does not, instead it uses a slightly more optimized packet forwarding path 
that was merged into the linux kernel in the last year or so.

Also, even though it might work this way on Mediatek, this is not how it 
works on other platforms.

There are as many ways to do this as there are vendors. They all butcher 
the Linux kernel in their own peculiar way to do what they feel they need 
to do.

There is work in PRPL foundation (https://prplfoundation.org/) to try to 
standardize this ,but it'll be years until we see the result in volume 
shipping devices probably.

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Thu, 18 Oct 2018 19:29:07 +0200

Yes, 100% sure in this case.

The SoC supports it with several SDKs/OSs:

https://www.mediatek.com/products/homeNetworking/mt7621n-a

You can enable HW or SW offloading, offers both options (didn't checked what is the difference, I just go for the HW if it works, as I guess outperforms the SW one).

The official support in OpenWRT is very recent (as you said, because it is very recent in Linux kernels), and of course, may be other chip sets are not yet supported, as they don't always offer "open" access to all the specs, require using proprietary binaries, etc.

So clearly agree that it may be chipset dependent, and also Linux dependent, etc.: in short implementation dependent.

Regards,
Jordi

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 19:44:56 +0200 (CEST)

On Thu, 18 Oct 2018, JORDI PALET MARTINEZ wrote:

> You can enable HW or SW offloading, offers both options (didn't checked 
> what is the difference, I just go for the HW if it works, as I guess 
> outperforms the SW one).

I know the MT7621 does what you're saying in other OSes with their SDK, 
I'm questioning if the -j FLOWOFFLOAD is the way to do HW acceleration for 
certain flows. Did you try to run full gig speed through it and check CPU 
usage? It's the easiest way to tell if the offload works or not.

It would be the least intrusive offload implementation I've ever seen in 
Linux, so if it actually works this way I'd be very happy, and hope more 
vendors does it the same way.

Did you buy some retail device? If you did, what was it?

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Fri, 19 Oct 2018 21:49:48 +0200

Hi Mikael,

Sorry, was coming back from the RIPE and arranging a small test to show you the results, which are consistent with what I recall from my previous testing.

I'm using an ubuntu VM as the server, running in ESXi 6.5, and as client my own laptop.

There is a Cisco switch between the server, the MT7621 device and my client.

Ubuntu VM in ESXi --- Cisco Gig switch --- MT7621 under test --- client device

Server command: iperf3 -s

1) LAN-to-LAN test (so both the server and client are connected to the LANs of the MT7621)
iperf3 -c 10.10.10.3
Connecting to host 10.10.10.3, port 5201
[  4] local 10.10.10.99 port 51300 connected to 10.10.10.3 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   113 MBytes   951 Mbits/sec                  
[  4]   1.00-2.00   sec   112 MBytes   937 Mbits/sec                  
[  4]   2.00-3.00   sec   112 MBytes   938 Mbits/sec                  
[  4]   3.00-4.00   sec   112 MBytes   937 Mbits/sec                  
[  4]   4.00-5.00   sec   112 MBytes   936 Mbits/sec                  
[  4]   5.00-6.00   sec   112 MBytes   937 Mbits/sec                  
[  4]   6.00-7.00   sec   112 MBytes   937 Mbits/sec                  
[  4]   7.00-8.00   sec   112 MBytes   937 Mbits/sec                  
[  4]   8.00-9.00   sec   112 MBytes   937 Mbits/sec                  
[  4]   9.00-10.00  sec   112 MBytes   936 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  receiver

2) LAN-to-WAN NAT test without offloading
iperf3 -c 10.10.10.3
Connecting to host 10.10.10.3, port 5201
[  4] local 192.168.1.139 port 51447 connected to 10.10.10.3 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  48.8 MBytes   409 Mbits/sec                  
[  4]   1.00-2.00   sec  45.8 MBytes   384 Mbits/sec                  
[  4]   2.00-3.00   sec  45.2 MBytes   379 Mbits/sec                  
[  4]   3.00-4.00   sec  45.2 MBytes   379 Mbits/sec                  
[  4]   4.00-5.00   sec  45.0 MBytes   378 Mbits/sec                  
[  4]   5.00-6.00   sec  44.9 MBytes   377 Mbits/sec                  
[  4]   6.00-7.00   sec  45.2 MBytes   379 Mbits/sec                  
[  4]   7.00-8.00   sec  45.1 MBytes   378 Mbits/sec                  
[  4]   8.00-9.00   sec  45.3 MBytes   380 Mbits/sec                  
[  4]   9.00-10.00  sec  45.2 MBytes   379 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   456 MBytes   382 Mbits/sec                  sender
[  4]   0.00-10.00  sec   456 MBytes   382 Mbits/sec                  receiver

3) LAN-to-WAN NAT test with SOFTWARE offloading
iperf3 -c 10.10.10.3
Connecting to host 10.10.10.3, port 5201
[  4] local 192.168.1.139 port 51484 connected to 10.10.10.3 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  98.4 MBytes   826 Mbits/sec                  
[  4]   1.00-2.00   sec  98.2 MBytes   824 Mbits/sec                  
[  4]   2.00-3.00   sec  96.9 MBytes   813 Mbits/sec                  
[  4]   3.00-4.00   sec  97.6 MBytes   819 Mbits/sec                  
[  4]   4.00-5.00   sec  98.2 MBytes   824 Mbits/sec                  
[  4]   5.00-6.00   sec  96.5 MBytes   809 Mbits/sec                  
[  4]   6.00-7.00   sec  96.0 MBytes   805 Mbits/sec                  
[  4]   7.00-8.00   sec  96.1 MBytes   806 Mbits/sec                  
[  4]   8.00-9.00   sec  97.8 MBytes   820 Mbits/sec                  
[  4]   9.00-10.00  sec  97.5 MBytes   818 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   973 MBytes   816 Mbits/sec                  sender
[  4]   0.00-10.00  sec   971 MBytes   815 Mbits/sec                  receiver

4) LAN-to-WAN NAT test with HARDWARE offloading
iperf3 -c 10.10.10.3
Connecting to host 10.10.10.3, port 5201
[  4] local 192.168.1.139 port 51523 connected to 10.10.10.3 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   110 MBytes   926 Mbits/sec                  
[  4]   1.00-2.00   sec   112 MBytes   938 Mbits/sec                  
[  4]   2.00-3.00   sec   111 MBytes   933 Mbits/sec                  
[  4]   3.00-4.00   sec   111 MBytes   935 Mbits/sec                  
[  4]   4.00-5.00   sec   111 MBytes   933 Mbits/sec                  
[  4]   5.00-6.00   sec   111 MBytes   933 Mbits/sec                  
[  4]   6.00-7.00   sec   111 MBytes   933 Mbits/sec                  
[  4]   7.00-8.00   sec   110 MBytes   923 Mbits/sec                  
[  4]   8.00-9.00   sec   111 MBytes   934 Mbits/sec                  
[  4]   9.00-10.00  sec   111 MBytes   933 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.09 GBytes   932 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.08 GBytes   931 Mbits/sec                  receiver

Regarding CPU utilization, using htop, with both NO offloading and SW offloading, it is evident that the CPU is being used reaching close to 100% of one of the cores, a bit less in the others, while HW offloading CPU remains with the 1-2% utilization.

By the way the devices are cheap ones, different vendors, no difference if I do the test with one or the other (tested with 3 different ones and got same results). The most expensive one is probably around 30 USD (ZBT-WG3526-16M).

I've used the stable builds and then upgraded everything to the latest versions.

https://downloads.openwrt.org/releases/18.06.1/targets/ramips/mt7621/

If you're interested and coming to BKK, I can bring one of them and we can play a bit.

Regards,
Jordi

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 20 Oct 2018 21:12:11 +0200 (CEST)

> If you're interested and coming to BKK, I can bring one of them and we can play a bit.

Thanks, I will try to acquire one of these devices in the mean time. I 
will be at the hackathon on both saturday and sunday, so if you can bring 
one that'd be great in case I am not able to get one before then.

---

From: Yannis Nikolopoulos <[email-masked]>
Date: Mon, 22 Oct 2018 20:57:23 +0300

hello Lee, all

> either on-line or directly.

In no particular order:

1. CPE (almost OK after ~2 years and god knows how many iterations)
2. lwaftr performance (worth noting the the lwaftr is implemented as a VNF
3. vendor's reluctance to continue (much needed) dev/ment because of 
lack of interest from other ISPs

>> invested in IPv6 (lw4o6). In the mean time between deploying lw4o6, 
>> we had to mitigate the issue, in a quick n dirty manner... that's 
>> when CGNcame into the picture. Some years later, we have a lot of CGN 
>> users and not so many in IPv6-only (lw4o6).  It just seems that, 
>> IPv6-only technologies (lw4o6 in our case) were not so straighforward 
>> to deploy and more importantly, not so cheap (performance-wise) 
>> either. Maybe the "stateless card" has been overplayed or exaggerated 
>> at the very least.
>>
>>>
>>> Regards,
>>> Jordi
>>>
>>> strictly prohibited and will be considered a criminal offense. If 
>>> you are not the intended recipient be aware that any disclosure, 
>>> copying, distribution or use of the contents of this information, 
>>> even if partially, including attached files, is strictly prohibited, 
>>> will be considered a criminal offense, so you must reply to the 
>>> original sender to inform about this communication and delete it.
>>>
>>>
>>>
>>>
>>
>

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 23 Oct 2018 17:18:37 +0000

Jordi,

This is quite useful. Do you also happen to have similar test numbers with CLAT functionality vs. MAP(-T/E) CE functionality? That would really help, and OpenWRT supports them.

It would be pertinent to have consistent traffic amount and no of connections/flows in each case.

Say, 60% IPv6 and 40% IPv4 traffic (by bps) as well as connections.

Has anybody got any recent measurement about no of connections per HGW? Mine is dated in 2012-2013.

Cheers,
Rajiv

From: v6ops <[email-masked]> on behalf of JORDI PALET MARTINEZ <jordi.palet=[email-masked]>
Date: Friday, October 19, 2018 at 4:04 PM
To: Mikael Abrahamsson <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

Hi Mikael,

Regards,
Jordi

-----Mensaje original-----
De: Mikael Abrahamsson <[email-masked]<mailto:[email-masked]>>
Organización: People's Front Against WWW
Fecha: jueves, 18 de octubre de 2018, 19:45
Para: JORDI PALET MARTINEZ <[email-masked]<mailto:[email-masked]>>
CC: <[email-masked]<mailto:[email-masked]>>
Asunto: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

    --
    Mikael Abrahamsson    email: [email-masked]<mailto:[email-masked]>

_______________________________________________
v6ops mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/v6ops

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Tue, 23 Oct 2018 19:31:50 +0200

Hi Rajiv,

In the scope of draft-lmhp-v6ops-transition-comparison, we are planning some measurements, but I’m not sure when and how we will be able to do them.

Also, if we really want to have those being realistic, we should plan different testing combinations, and also differentiate the performance of the CE part vs the “network” part and the combination of both.

Last but not least, we can easily do with opensource, for example OpenWRT or VPP in for the CPE and Jool or VPP for the NAT64/BR on the other side, or combinations of them, but trying to make it with hardware from vendors, unless they want to provide it, will be difficult.

I’m happy to invest time in that, but we need to correctly plan all the details to be fair with the measurements.

It may be a topic also for a hackathon, but I’m not sure there is sufficient time in a weekend for all the testing …

Regards,

Jordi

De: v6ops <[email-masked]> en nombre de "Rajiv Asati (rajiva)" <[email-masked]>
Fecha: martes, 23 de octubre de 2018, 19:20
Para: JORDI PALET MARTINEZ <jordi.palet=[email-masked]>, Mikael Abrahamsson <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Asunto: Re: [v6ops] Trajectory for "transition" mechanisms in our working group

Jordi,

Cheers,

Rajiv

Hi Mikael,

iperf3 -c 10.10.10.3

iperf3 -c 10.10.10.3

iperf3 -c 10.10.10.3

iperf3 -c 10.10.10.3

Regards,

Jordi

-----Mensaje original-----

CC: <[email-masked]>

    -- 

 

IPv4 is over

The IPv6 Company

 

v6ops mailing list

[email-masked]

_______________________________________________ v6ops mailing list [email-masked] https://www.ietf.org/mailman/listinfo/v6ops

---

From: Lencse Gábor <[email-masked]>
Date: Tue, 23 Oct 2018 20:54:29 +0200

Hi,

10/23/2018 7:31 PM keltezéssel, JORDI PALET MARTINEZ írta:
>
> Hi Rajiv,
>
> In the scope of draft-lmhp-v6ops-transition-comparison, we are 
> planning some measurements, but I’m not sure when and how we will be 
> able to do them.
>
> Also, if we really want to have those being realistic, we should plan 
> different testing combinations, and also differentiate the performance 
> of the CE part vs the “network” part and the combination of both.
>
> Last but not least, we can easily do with opensource, for example 
> OpenWRT or VPP in for the CPE and Jool or VPP for the NAT64/BR on the 
> other side, or combinations of them, but trying to make it with 
> hardware from vendors, unless they want to provide it, will be difficult.
>
> I’m happy to invest time in that, but we need to correctly plan all 
> the details to be fair with the measurements.
>
> It may be a topic also for a hackathon, but I’m not sure there is 
> sufficient time in a weekend for all the testing …
>

As for benchmarking, iperf or RFC 2544 (RFC 5180) compliant testers may 
only be used when the devices are used in pairs. RFC 8219 call this as 
"Dual DUT" setup.
Testing according to the "Single DUT" setup would require special 
Testers. (E.g. for testing a NAT64 gateway: the Tester needs to be able 
to send IPv6 packets to the DUT and receive IPv4 traffic from the DUT in 
one direction, while it needs to send IPv4 packets to the DUT and 
receive IPv6 packets from the DUT in the other direction at the same 
time, as bidirectional traffic is required.) As far as I know, no 
special Testers available yet. (One of my PhD students working on a DPDK 
based stateless NAT64 tester, but it is far from usable.)

I think implementing RFC 8219 compliant special Testers for the five 
IPv4aaS technologies mentioned in 
draft-lmhp-v6ops-transition-comparison, would be an interesting task for 
hackathon!

I definitely recommend using DPDK to achieve the required performance. 
(Perhaps TRex https://trex-tgn.cisco.com/ may help in implementing the 
Testers, but later it may be more complicated to use them. )

Best regards,

Gábor Lencse

>
> Regards,
>
> Jordi
>
> *De: *v6ops <[email-masked]> en nombre de "Rajiv Asati 
> (rajiva)" <[email-masked]>
> *Fecha: *martes, 23 de octubre de 2018, 19:20
> *Para: *JORDI PALET MARTINEZ 
> <jordi.palet=[email-masked]>, Mikael Abrahamsson 
> <[email-masked]>
> *CC: *"[email-masked]" <[email-masked]>
> *Asunto: *Re: [v6ops] Trajectory for "transition" mechanisms in our 
> working group
>
> Jordi,
>
> This is quite useful. Do you also happen to have similar test numbers 
> with CLAT functionality vs. MAP(-T/E) CE functionality? That would 
> really help, and OpenWRT supports them.
>
> *From: *v6ops <[email-masked]> on behalf of JORDI PALET 
> MARTINEZ <jordi.palet=[email-masked]>
> *Date: *Friday, October 19, 2018 at 4:04 PM
> *To: *Mikael Abrahamsson <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>
> *Subject: *Re: [v6ops] Trajectory for "transition" mechanisms in our 
> working group
>
> Hi Mikael,
>
> Sorry, was coming back from the RIPE and arranging a small test to 
> show you the results, which are consistent with what I recall from my 
> previous testing.
>
> [  4] 1..00-2.00   sec  45.8 MBytes   384 Mbits/sec
>
> Regarding CPU utilization, using htop, with both NO offloading and SW 
> offloading, it is evident that the CPU is being used reaching close to 
> 100% of one of the cores, a bit less in the others, while HW 
> offloading CPU remains with the 1-2% utilization.
>
> By the way the devices are cheap ones, different vendors, no 
> difference if I do the test with one or the other (tested with 3 
> different ones and got same results). The most expensive one is 
> probably around 30 USD (ZBT-WG3526-16M).
>
> De: Mikael Abrahamsson <[email-masked] <mailto:[email-masked]>>
>
> CC: <[email-masked] <mailto:[email-masked]>>
>
>     > Yes, 100% sure in this case.
>
>     >
>
>     > The SoC supports it with several SDKs/OSs:
>
>     >
>
>     > https://www.mediatek.com/products/homeNetworking/mt7621n-a
>
>     >
>
>     > You can enable HW or SW offloading, offers both options (didn't 
> checked
>
>     > what is the difference, I just go for the HW if it works, as I 
> guess
>
>     > outperforms the SW one).
>
> <mailto:[email-masked]>
>
> authorized disclosure, copying, distribution or use of the contents of 
> this information, even if partially, including attached files, is 
> [email-masked] <mailto:[email-masked]>
>
> authorized disclosure, copying, distribution or use of the contents of 
> this information, even if partially, including attached files, is

---

From: Fred Baker <[email-masked]>
Date: Wed, 24 Oct 2018 14:21:53 +0200

I don't have measurements, but I have a comment on the design of the CLAT protocol operation (RFC 7915). It maintains no tables; it simply appends the IPv4 source and destination addresses to relevant /96 prefixes and writes an IPv6 header, and does the reverse on packet receipt. The PLAT, which is not very different from NAPT44 except using IPv6 as the source, is far more complex by nature. Since it keeps no session state and does nothing session-related, I would be surprised to find session-count an issue in a CLAT.

> On Oct 23, 2018, at 7:18 PM, Rajiv Asati (rajiva) <[email-masked]> wrote:
> 
> Jordi,
> 

--------------------------------------------------------------------------------
Victorious warriors win first and then go to war,
Defeated warriors go to war first and then seek to win.
     Sun Tzu

---

From: JORDI PALET MARTINEZ <[email-masked]>
Date: Wed, 24 Oct 2018 15:15:04 +0200

Hi Fred,

Agree and depending on the specific IPv4aaS technology, it may be very similar the CE part to what happens in the CLAT.

However, and that's part of the issue, the CLAT (as well as some of the other IPv4aaS) also runs a NAT44, and the combination of the NAT46 and NAT44 can be implemented in different ways, including a stateful one.

Moreover, in some CEs, even if it is stateful, is not an issue, because is done with hardware offloading. Some chipsets support it, some not. Some support it but opensource implementations are not able to use it because the chipset vendor is not providing the code ...

The other advantage, as reported by Cameron, is that if using DNS64, less than 1% of the traffic will use the CLAT, and only 25% will use the PLAT. Of course, as more content/application providers deploy IPv6, those numbers will keep going lower and lower.

Of course, when doing performance testing, we need to isolate all that, or it will not be a fair comparison.

Regards,
Jordi