Subject: Re: [Pals] [Int-area] L2TP sequencing: Commonly disabled for IP data? Or always?

From: Stewart Bryant <[email-masked]>
Date: Fri, 4 Jun 2021 16:07:10 +0100

Adding PALS and someone there may know.

Stewart

> On 4 Jun 2021, at 15:13, Bob Briscoe <[email-masked]> wrote:
> 
> Int-area list,
> 
> I'm looking for experience on common L2TP practice, most likely from operators. I tried sending this query to the [email-masked] <mailto:[email-masked]> list, as advised by Carlos Pignataro, but apparently it no longer exists. So I think int-area is the "list of last resort" for this.
> 
> The L2TP RFC says sequencing /can/ be disabled for IP data, but it doesn't say SHOULD or MUST. Is it possible that some operators enable L2TP sequencing for IP data? And if so, do you know why they would? Also, are you aware of any other types of tunnel that might try to keep IP data packets in sequence?
> 
> My reason for asking: 
> We (in tsvwg) are working on active queue management technology. Certain AQM schemes (e.g. FQ-CoDel, L4S) give lower delay to a subset of traffic. If the bottleneck queue supports such an AQM and it is within an L2TP tunnel with sequencing enabled, the egress would hold back all the nice low delay packets until it can put them back into order with the higher delay traffic.
> 
> We intend to advise that operators MUST disable L2TP sequencing if they wish to deploy these AQMs within an L2TP tunnel. So we need to know:
> Whether this will create a dilemma for any operators who need L2TP sequencing of IP data for some reason;
> Or whether we even need to bother giving the advice, because no operator would ever enable L2TP sequencing of IP data anyway.
> 
> Obviously, some operators already use existing technologies like Diffserv to reduce delay for a subset of IP data traffic, so I assume they always disable L2TP sequencing anyway. 
> 
> Regards
> 
> 
> Bob Briscoe
> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/ <http://bobbriscoe.net/>_______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 9 Jun 2021 11:23:00 +0100

Sequence number checking in the forwarder is always a problem because it is stateful so I doubt that many high-scale or high-speed forwarders ever did this.

I think there is an undisclosed assumption that go up enough levels and its IP so sequence number checking in the transport network (as opposed to the transport layer) is not really needed.

I doubt that there is much L2TP still out there. It was in its prime with dialup modems. L2TPv3 which was intended to replace it became niche with, as Andy says, operators who did not want MPLS. Much of what L2TPv3 was intended for was actually done with PW over MPLS with some replacement with by Mac in Mac for cost reasons.

If Carlos does not know the answer, Mark T would be my next port of call.

Stewart

> On 8 Jun 2021, at 22:41, Andrew G. Malis <[email-masked]> wrote:
> 
> Bob,
> 
> In addition to the cases listed by Derek, L2TPv3 can also carry non-IP pseudowire data, such as Ethernet frames (see RFC 4719 for example). Even though 4719 says that sequencing is optional, I would certainly recommend it :-).
> 
> But I guess that's really not what you were asking about, since you specifically mentioned IP data. But it is a case where you would probably see sequencing in use.
> 
> Back in the day, Sprint made good use of Ethernet over L2TPv3, as they were in the anti-MPLS camp at the time. But that's water over the bridge, and I really don't know if this solution continues to be in active use. Mark Townsley might know.
> 
> Cheers,
> Andy
> 
> 
> On Sat, Jun 5, 2021 at 10:07 AM Derek Fawcus <[email-masked] <mailto:[email-masked]>> wrote:
> On Fri, Jun 04, 2021 at 03:13:15PM +0100, Bob Briscoe wrote:
> > The L2TP RFC says sequencing /can/ be disabled for IP data, but it 
> > doesn't say SHOULD or MUST. Is it possible that some operators enable 
> > L2TP sequencing for IP data? And if so, do you know why they would? 
> > Also, are you aware of any other types of tunnel that might try to keep 
> > IP data packets in sequence?
> 
> How many intermediate headers are you considering between L2TP and where
> a carried IP header may exist?
> 
> Maybe I'm getting the wrong end of the stick, but surely this engages
> the text from section 5.4 of RFC 2661:
> 
>   "For example, if the PPP session being tunneled is not
>    utilizing any stateful compression or encryption protocols and is
>    only carrying IP (as determined by the PPP NCPs that are
>    established), then the LNS might decide to disable sequencing as IP
>    is tolerant to datagram loss and reordering."
> 
> This would then suggest if L2TP is carrying PPP, the PPP session is not
> multi-link, and is making use of compression (including one of the
> versions of IP header compression) in some form for IP packets, then
> reordering will impact the ability to decompress.
> 
> So such an L2TP data session may well make use of sequence numbers to
> prevent reordering.
> 
> I guess similarly in L2TPv3 when the PW is for PPP, and possibly also
> the fragmentation scheme in RFC 4623 which requires sequence numbers;
> and such PWE3 links could ultimately be carrying IP packets.
> 
> 
> DF
> 
> (not an operator)
> 
> _______________________________________________
> Int-area mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/int-area <https://www.ietf.org/mailman/listinfo/int-area>
> _______________________________________________
> Int-area mailing list
> [email-masked]

---

From: Alexander Vainshtein <[email-masked]>
Date: Wed, 9 Jun 2021 11:04:04 +0000

Hi all,
I mainly concur with Stewart's comments about sequence number checking.
However, there are two domains (not sure if they could bd called niches ir not) where it is always pesent and critically imporrant:

  1.  One such domain is TDM PWs where srquence numbers are used to detect lost PW packets and generate correct smount if "replacement" bits at egress.AFAIK they are still widely deployed
  2.  Another (and relatively new one) domain is Detnet where sequence numbers are used by the Packet/Frame Replication Elimination Function (PREF/FREF).

AFAIK in both cases dedicated HW is required.

My 2c

Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________
From: Pals <[email-masked]> on behalf of Stewart Bryant <[email-masked]>
Sent: Wednesday, June 9, 2021, 13:23
To: Andrew G. Malis; [email-masked]
Cc: Carlos Pignataro (cpignata); Ignacio Goyret; intarea IETF list; Derek Fawcus; [email-masked]; Stewart Bryant
Subject: [EXTERNAL] Re: [Pals] [Int-area] L2TP sequencing: Commonly disabled for IP data? Or always?

Stewart

On 8 Jun 2021, at 22:41, Andrew G. Malis <[email-masked]<mailto:[email-masked]>> wrote:

Bob,

Cheers,
Andy

On Sat, Jun 5, 2021 at 10:07 AM Derek Fawcus <[email-masked]<mailto:[email-masked]>> wrote:

DF

(not an operator)

_______________________________________________
Int-area mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/int-area<https://clicktime.symantec.com/3TrwYm64nkihRWfK7Da9QhH6H2?u=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fint-area>
_______________________________________________
Int-area mailing list
[email-masked]<mailto:[email-masked]>
https://clicktime.symantec.com/3TrwYm64nkihRWfK7Da9QhH6H2?u=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fint-area

Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. and its Affiliates that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.

Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. and its Affiliates that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.

---

From: Derek Fawcus <[email-masked]>
Date: Wed, 9 Jun 2021 12:19:52 +0100

On Wed, Jun 09, 2021 at 11:23:00AM +0100, Stewart Bryant wrote:
> 
> I doubt that there is much L2TP still out there. It was in its prime with dialup modems.

Stewart,

Isn't the BT wholesale ADSL product still using it?

Someone recently pointed to their SIN 472, which reference RFC 2661 not 3931.
I don't know what other countries are doing for ADSL service,
and in how many ADSL is a live vs legacy product.

Certainly my ISP describes how one can connect to them using L2TP mainly because
they have the backend available for supporting the various 'broadband' access
facilities.  They also offer ADSL access via another provider, so maybe they're
also using L2TP?

So maybe the UK deployment doesn't count a 'much'.

DF

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 9 Jun 2021 07:42:48 -0400

There is plenty of L2TP still in use.
Yours,
Joel

On 6/9/2021 6:23 AM, Stewart Bryant wrote:
> Sequence number checking in the forwarder is always a problem because it 
> is stateful so I doubt that many high-scale or high-speed forwarders 
> ever did this.
> 
> I think there is an undisclosed assumption that go up enough levels and 
> its IP so sequence number checking in the transport network (as opposed 
> to the transport layer) is not really needed.
> 
> I doubt that there is much L2TP still out there. It was in its prime 
> with dialup modems. L2TPv3 which was intended to replace it became niche 
> with, as Andy says, operators who did not want MPLS. Much of what L2TPv3 
> was intended for was actually done with PW over MPLS with some 
> replacement with by Mac in Mac for cost reasons.
> 
>> In addition to the cases listed by Derek, L2TPv3 can also carry non-IP 
>> pseudowire data, such as Ethernet frames (see RFC 4719 for example). 
>> Even though 4719 says that sequencing is optional, I would certainly 
>> recommend it :-).
>>
>> But I guess that's really not what you were asking about, since you 
>> specifically mentioned IP data. But it is a case where you would 
>> probably see sequencing in use.
>>
>> Back in the day, Sprint made good use of Ethernet over L2TPv3, as they 
>> were in the anti-MPLS camp at the time. But that's water over the 
>> bridge, and I really don't know if this solution continues to be in 
>> active use. Mark Townsley might know.
>>
>> Cheers,
>> Andy
>>
>>
>> On Sat, Jun 5, 2021 at 10:07 AM Derek Fawcus 
>> <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>>     <https://www.ietf.org/mailman/listinfo/int-area>
>>

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 9 Jun 2021 12:48:22 +0100

> On 9 Jun 2021, at 12:19, Derek Fawcus <[email-masked]> wrote:
> 
> So maybe the UK deployment doesn't count a 'much’.
> 
> DF

I suppose the ADSL product may be using it. I thought that it was all MPLS-PW.

We need someone from Openreach to tell us what they are doing.

In any case hopefully the last of the xDSL will soon be confined to the scrap bin it should have been in years ago as part of the plan to replace copper with FTTP/5G/LEO.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 9 Jun 2021 12:50:03 +0100

Which applications still use it Joel?

Stewart

> On 9 Jun 2021, at 12:42, Joel M. Halpern <[email-masked]> wrote:
> 
>>> On 8 Jun 2021, at 22:41, Andrew G. Malis <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> Bob,
>>>

---

From: Derek Fawcus <[email-masked]>
Date: Wed, 9 Jun 2021 12:53:29 +0100

> I mainly concur with Stewart's comments about sequence number checking.

There is also another potentially extensive 'deployment' of L2TP,
especially over the last year, and that is as one of the options
in the Microsoft Windows RAS VPN feature.

If so, then that would be by relatively unsophisticated users,
so they'd not really be able to tune it for issues.

This is PPP/L2TP/ESP(transport-mode) occasionally using NAT-T,
and negotiated by IKEv1.  I don't know if the L2TP layer makes
use of sequencing.

I'd not be surprised if the ESP layer is using an anti-replay window,
in which case the points already rehearsed wrt L4S would apply,
but this time with people who won't know how to 'fix' it.

We supported use of this in our protocol stack in our product
for remote access until quiterecently, and I see there are other
vendors who still offer the same mechanism.

Specifically I'd imagine to easily interoperate with the RAS feature.

DF

---

From: Giles Heron <[email-masked]>
Date: Wed, 9 Jun 2021 13:05:03 +0100

Hi Stewart,

IIRC (and it *was* a long time ago), ADSL is BT Wholesale rather than OpenReach.

But at any rate yes, L2TP (v2 rather than v3) does tend to get used for wholesale ADSL provision (where that still exists).   There has been some move generally to PTA (PPP Termination and Aggregation) into VRFs but that depends on the wholesale provider’s offering.

Re BT specifically (and again my info is outdated and may be mis-remembered) the 20C network used L2TP, and I think it’s an option on 21C for handoff to smaller ISPs.  Should all be in the SINs somewhere?

Giles

> On 9 Jun 2021, at 12:48, Stewart Bryant <[email-masked]> wrote:
> 
> 
> 
> Pals mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pals

---

From: James Bensley <[email-masked]>
Date: Wed, 9 Jun 2021 15:07:41 +0200

> Re BT specifically (and again my info is outdated and may be mis-remembered) the 20C network used L2TP, and I think it’s an option on 21C for handoff to smaller ISPs.  Should all be in the SINs som…

I work in the UK ISP sector. I can confirm that L2TP is in wide spread
use for wholesale services by all major ISPs (except Liberty). I think
the real question here (unless I've misunderstood) is not "is L2TP
being used" but "is L2TP being used AND sequencing is being used
and/or enforced?" - is that understanding correct?

Chers,
James.

---

From: Joel Halpern Direct <[email-masked]>
Date: Wed, 9 Jun 2021 09:10:08 -0400

BNGs are still a big busienss.  And BNG resale /emote control uses L2TP 
in many cases.  The BBF has been working on (and published a first 
version of) protocol for control of split BNG.  L2TP is commonly used 
for these use cases.

Yours,
Joel

---

From: Derek Fawcus <[email-masked]>
Date: Wed, 9 Jun 2021 14:57:02 +0100

> and/or enforced?" - is that understanding correct?

Yes.

DF

---

From: David Allan I <[email-masked]>
Date: Wed, 9 Jun 2021 17:40:52 +0000

These is a significant number of operators that use PPPoE, and L2TP. Either simply for backhaul (where old ATM based DSL is still deployed and elsewhere) and/ or wholesale.  I checked BBF TR-92 which would be the like source of any recommendations on using sequencing but it is silent on the subject.

Dave

-----Original Message-----
From: Int-area <[email-masked]> On Behalf Of Joel Halpern Direct
Sent: Wednesday, June 9, 2021 6:10 AM
To: Stewart Bryant <[email-masked]>; Joel Halpern <[email-masked]>
Cc: Carlos Pignataro (cpignata) <[email-masked]>; Ignacio Goyret <[email-masked]>; intarea IETF list <[email-masked]>; Derek Fawcus <[email-masked]>; [email-masked]
Subject: Re: [Int-area] L2TP sequencing: Commonly disabled for IP data? Or always?

BNGs are still a big busienss.  And BNG resale /emote control uses L2TP in many cases.  The BBF has been working on (and published a first version of) protocol for control of split BNG.  L2TP is commonly used for these use cases.

Yours,
Joel

---

From: Mark Townsley <[email-masked]>
Date: Wed, 9 Jun 2021 15:32:24 -0700

Hi folks,

In addition to the DSL arena, L2TP is still in use with host-based VPN clients as it is embedded in Apple, Android, and Windows based operating systems (new and old). Despite most VPN solutions preferring their own client software that must be installed on hosts to operate, there are still admins that appreciate not having to ask their employees to download an app for the VPN to work - in which case PPTP and L2TP with transport-mode IPsec are your most widely available options. 

Regarding L2TPv3 replacing L2TP: L2TPv2 (RFC 2661) was PPP only. L2TPv3 generalized L2TP to support other L2 (including MPLS, but I don’t want to argue what layer MPLS operates within here). There was never a strong push to replace L2TPv2 used in DSL, Dialup and host VPN software with L2TPv3 (there was one use case for an important L2TP operator that wanted to carry PPPoE over L2TPv3 in DSL, but that was overcome by RFC3817 which achieved the same goal without altering the dataplane). Ironically, I would expect to see very little PPP over L2TPv3 in the wild, though obviously it is possible.

In the cable broadband world, the DOCSIS DEPI “Remote PHY” specification (similar I suppose to the split BNG spec Joel is referring to) standardized on L2TPv3 and is in active use.

I also know of at least one vendor that uses Ethernet over L2TPv3 for some wifi backhaul use cases. 

There could be more, this is just what I am personally aware of off the top of my head. Even I am surprised to see how much L2TP is still out there once you start really looking around ;-)

Best Regards,

- Mark

> On Jun 9, 2021, at 6:10 AM, Joel Halpern Direct <[email-masked]> wrote:
>

---

From: "Andrew G. Malis" <[email-masked]>
Date: Thu, 10 Jun 2021 07:41:40 -0400

Mark,

The original question was, how many (if any) of these L2TPv2 and v3 use
cases use sequencing, especially when carrying IP?

Cheers,
Andy

On Wed, Jun 9, 2021 at 6:32 PM Mark Townsley <[email-masked]> wrote:

> Hi folks,
>
> In addition to the DSL arena, L2TP is still in use with host-based VPN
> clients as it is embedded in Apple, Android, and Windows based operating
> systems (new and old). Despite most VPN solutions preferring their own
> client software that must be installed on hosts to operate, there are still
> admins that appreciate not having to ask their employees to download an app
> for the VPN to work - in which case PPTP and L2TP with transport-mode IPsec
> are your most widely available options.
>
> Regarding L2TPv3 replacing L2TP: L2TPv2 (RFC 2661) was PPP only. L2TPv3
> generalized L2TP to support other L2 (including MPLS, but I don’t want to
> argue what layer MPLS operates within here). There was never a strong push
> to replace L2TPv2 used in DSL, Dialup and host VPN software with L2TPv3
> (there was one use case for an important L2TP operator that wanted to carry
> PPPoE over L2TPv3 in DSL, but that was overcome by RFC3817 which achieved
> the same goal without altering the dataplane). Ironically, I would expect
> to see very little PPP over L2TPv3 in the wild, though obviously it is
> possible.
>
> In the cable broadband world, the DOCSIS DEPI “Remote PHY” specification
> (similar I suppose to the split BNG spec Joel is referring to) standardized
> on L2TPv3 and is in active use.
>
> There could be more, this is just what I am personally aware of off the
> top of my head. Even I am surprised to see how much L2TP is still out there
> once you start really looking around ;-)
>
> Best Regards,
>
> - Mark
>
>
>
>
> in many cases.  The BBF has been working on (and published a first version
> of) protocol for control of split BNG.  L2TP is commonly used for these use
> cases.
> >
> > Yours,
> > Joel
> >
> >>>> Sequence number checking in the forwarder is always a problem because
> it is stateful so I doubt that many high-scale or high-speed forwarders
> ever did this.
> >>>> I think there is an undisclosed assumption that go up enough levels
> and its IP so sequence number checking in the transport network (as opposed
> was intended for was actually done with PW over MPLS with some replacement
> with by Mac in Mac for cost reasons.
> >>>>> In addition to the cases listed by Derek, L2TPv3 can also carry
> non-IP pseudowire data, such as Ethernet frames (see RFC 4719 for example).
> specifically mentioned IP data. But it is a case where you would probably
> see sequencing in use.
> >>>>>
> >>>>> Back in the day, Sprint made good use of Ethernet over L2TPv3, as
> they were in the anti-MPLS camp at the time. But that's water over the
> bridge, and I really don't know if this solution continues to be in active
> use. Mark Townsley might know.
> >>>>>
> >>>>> Cheers,
> >>>>> Andy
> >>>>>
> >>>>>
> >>>>> On Sat, Jun 5, 2021 at 10:07 AM Derek Fawcus <
> [email-masked] <mailto:
> [email-masked]>> wrote:
> >>>>>

---

From: "Andrew G. Malis" <[email-masked]>
Date: Thu, 10 Jun 2021 07:49:21 -0400

(resending with cc: list trimmed to pass the too many recipients filter)
Mark,

Cheers,
Andy

---

From: James Bensley <[email-masked]>
Date: Thu, 10 Jun 2021 14:09:10 +0200

> > and/or enforced?" - is that understanding correct?

In that case  - sequencing isn't used for BT Wholesale services.

Cheers,
James.

---

From: Giles Heron <[email-masked]>
Date: Thu, 10 Jun 2021 13:22:54 +0100

So AFAIK SP networks don’t generally reorder packets in the steady state, but of course reordering can occur under rerouting.

As noted by Derek I’m guessing reordering is an issue for L2TPv2 if stateful PPP compression schemes are in play (which I suspect is unlikely to be the case given we usually have abundant bandwidth in the aggregation network, and given that compression can occur at other layers).  Though given that BNGs inherently keep state per subscriber maybe the NPU scaling issues that Stewart mentioned are less of an issue in that use case than for MPLS PEs doing PWE?

From a quick look at the DOCSIS rPHY specs it seems they do require support for L2TPv3 sequence numbers.  Of course in that case the payload is the DOCSIS MAC rather than IP (even though, of course, most DOCSIS frames ultimately carry an IP payload).

Giles

> On 10 Jun 2021, at 12:49, Andrew G. Malis <[email-masked]> wrote:
> 
> The original question was, how many (if any) of these L2TPv2 and v3 use cases use sequencing, especially when carrying IP?
> 
> Cheers,
> Andy
> 
> On Wed, Jun 9, 2021 at 6:32 PM Mark Townsley <[email-masked] <mailto:[email-masked]>> wrote:
> Hi folks,
> 
> > On Jun 9, 2021, at 6:10 AM, Joel Halpern Direct <[email-masked] <mailto:[email-masked]>> wrote:
> > 
> >>> On 9 Jun 2021, at 12:42, Joel M. Halpern <[email-masked] <mailto:[email-masked]>> wrote:
> >>> 
> >>>>> On 8 Jun 2021, at 22:41, Andrew G. Malis <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
> >>>>> 
> >>>>> Bob,
> >>>>> 
> >>>>> On Sat, Jun 5, 2021 at 10:07 AM Derek Fawcus <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
> >>>>> 
> >>>>>    [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
> >>>>>    <https://www.ietf.org/mailman/listinfo/int-area <https://www.ietf.org/mailman/listinfo/int-area>>
> >>>>> 
> >>>>> [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 27 Jun 2021 22:22:10 +0100

James,

[First apologies to everyone for asking the question then not following 
up on the answers until now - I went off line for a while.]
See inline tagged [BB]...

> In that case  - sequencing isn't used for BT Wholesale services.

[BB] Thank you. It sounds like you have some reason for your certainty 
here. Can you elaborate on what makes you so certain sequencing isn't used?

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Jun 2021 00:23:34 +0100

Giles, Mark,

On 10/06/2021 13:22, Giles Heron wrote:
> So AFAIK SP networks don’t generally reorder packets in the steady 
> state, but of course reordering can occur under rerouting.

[BB] The cases I'm concerned about are where the operator
* deliberately reorders packets using a multi-queue scheduler at a node 
contrived to act as the bottleneck (the BNG)
* AND the node is within an L2TPv2/3 tunnel
* AND the tunnel needs sequencing at the egress for some other reason.

In many cases, such a scheduler would be located prior to the tunnel 
ingress, so not a concern. I believe the DOCSIS rPHY case below falls 
into that category (both downstream and up).

In contrast, where a BNG sits /within/ the span of an L2TP tunnel, I 
think it will often (or at least sometimes?) have been constructed as 
the bottleneck. Any operator having designed such a QoS arrangement 
would not want to support sequencing...

>
> As noted by Derek I’m guessing reordering is an issue for L2TPv2 if 
> stateful PPP compression schemes are in play (which I suspect is 
> unlikely to be the case given we usually have abundant bandwidth in 
> the aggregation network, and given that compression can occur at other 
> layers).  Though given that BNGs inherently keep state per subscriber 
> maybe the NPU scaling issues that Stewart mentioned are less of an 
> issue in that use case than for MPLS PEs doing PWE?

My concern was that 'keep it simple' operators that are using L2TPv2/3 
and had not previously bothered with the complexities of QoS might want 
to support L4S, because it has the potential to cut out queue delay for 
/all/ traffic. Altho' L4S is eventually for all traffic, it still 
requires two queues at the bottleneck for transition - one for L4S and 
one for not-yet-L4S ('Classic') traffic flows, and therefore introduces 
reordering at the aggregate level...

 From the replies so far, even if such 'keep it simple' operators were 
using compression, I can't see any reason why having to turn off 
compression and sequencing (in order to support L4S) would be a 
significant problem nowadays.

So, in conclusion, I don't think we even need to raise any concerns 
about L2TP sequencing in the L4S specs.

If anyone here thinks otherwise, pls speak now.

Thank you everyone who has contributed to this discussion.

Cheers

Bob

>
> From a quick look at the DOCSIS rPHY specs it seems they do require 
> support for L2TPv3 sequence numbers.  Of course in that case the 
> payload is the DOCSIS MAC rather than IP (even though, of course, most 
> DOCSIS frames ultimately carry an IP payload).
>
> Giles
>
>> On 10 Jun 2021, at 12:49, Andrew G. Malis <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> The original question was, how many (if any) of these L2TPv2 and v3 
>> use cases use sequencing, especially when carrying IP?
>>
>> Cheers,
>> Andy
>>
>>     In addition to the DSL arena, L2TP is still in use with
>>     host-based VPN clients as it is embedded in Apple, Android, and
>>     Windows based operating systems (new and old). Despite most VPN
>>     solutions preferring their own client software that must be
>>     installed on hosts to operate, there are still admins that
>>     appreciate not having to ask their employees to download an app
>>     for the VPN to work - in which case PPTP and L2TP with
>>     transport-mode IPsec are your most widely available options.
>>
>>     Regarding L2TPv3 replacing L2TP: L2TPv2 (RFC 2661) was PPP only.
>>     L2TPv3 generalized L2TP to support other L2 (including MPLS, but
>>     I don’t want to argue what layer MPLS operates within here).
>>     There was never a strong push to replace L2TPv2 used in DSL,
>>     Dialup and host VPN software with L2TPv3 (there was one use case
>>     for an important L2TP operator that wanted to carry PPPoE over
>>     L2TPv3 in DSL, but that was overcome by RFC3817 which achieved
>>     the same goal without altering the dataplane). Ironically, I
>>     would expect to see very little PPP over L2TPv3 in the wild,
>>     though obviously it is possible.
>>
>>     In the cable broadband world, the DOCSIS DEPI “Remote PHY”
>>     specification (similar I suppose to the split BNG spec Joel is
>>     referring to) standardized on L2TPv3 and is in active use.
>>
>>     There could be more, this is just what I am personally aware of
>>     off the top of my head. Even I am surprised to see how much L2TP
>>     is still out there once you start really looking around ;-)
>>
>>     Best Regards,
>>
>>     - Mark
>>
>>
>>
>>
>>     > On Jun 9, 2021, at 6:10 AM, Joel Halpern Direct
>>     <[email-masked] <mailto:[email-masked]>>
>>     wrote:
>>     >
>>     > BNGs are still a big busienss.  And BNG resale /emote control
>>     uses L2TP in many cases.  The BBF has been working on (and
>>     published a first version of) protocol for control of split BNG. 
>>     L2TP is commonly used for these use cases.
>>     >
>>     > Yours,
>>     > Joel
>>     >
>>     >>>> Sequence number checking in the forwarder is always a
>>     problem because it is stateful so I doubt that many high-scale or
>>     high-speed forwarders ever did this.
>>     >>>> I think there is an undisclosed assumption that go up enough
>>     levels and its IP so sequence number checking in the transport
>>     network (as opposed to the transport layer) is not really needed.
>>     >>>> I doubt that there is much L2TP still out there. It was in
>>     its prime with dialup modems. L2TPv3 which was intended to
>>     replace it became niche with, as Andy says, operators who did not
>>     want MPLS. Much of what L2TPv3 was intended for was actually done
>>     with PW over MPLS with some replacement with by Mac in Mac for
>>     cost reasons.
>>     >>>>> On 8 Jun 2021, at 22:41, Andrew G. Malis <[email-masked]
>>     <mailto:[email-masked]> <mailto:[email-masked]
>>     <mailto:[email-masked]>>> wrote:
>>     >>>>>
>>     >>>>> Bob,
>>     >>>>>
>>     >>>>> In addition to the cases listed by Derek, L2TPv3 can also
>>     carry non-IP pseudowire data, such as Ethernet frames (see RFC
>>     4719 for example). Even though 4719 says that sequencing is
>>     optional, I would certainly recommend it :-).
>>     >>>>>
>>     >>>>> But I guess that's really not what you were asking about,
>>     since you specifically mentioned IP data. But it is a case where
>>     you would probably see sequencing in use.
>>     >>>>>
>>     >>>>> Back in the day, Sprint made good use of Ethernet over
>>     L2TPv3, as they were in the anti-MPLS camp at the time. But
>>     that's water over the bridge, and I really don't know if this
>>     solution continues to be in active use. Mark Townsley might know.
>>     >>>>>
>>     >>>>> Cheers,
>>     >>>>> Andy
>>     >>>>>
>>     >>>>>
>>     <mailto:[email-masked]>
>>     <mailto:[email-masked]
>>     <mailto:[email-masked]>>> wrote:
>>     >>>>>
>>     >>>>>    > The L2TP RFC says sequencing /can/ be disabled for IP
>>     data, but it
>>     >>>>>    > doesn't say SHOULD or MUST. Is it possible that some
>>     operators
>>     >>>>>    enable
>>     >>>>>    > L2TP sequencing for IP data? And if so, do you know
>>     why they would?
>>     >>>>>    > Also, are you aware of any other types of tunnel that
>>     might try
>>     >>>>>    to keep
>>     >>>>>    > IP data packets in sequence?
>>     >>>>>
>>     >>>>>    How many intermediate headers are you considering
>>     between L2TP and
>>     >>>>>    where
>>     >>>>>    This would then suggest if L2TP is carrying PPP, the PPP
>>     session
>>     >>>>>    is not
>>     <mailto:[email-masked] <mailto:[email-masked]>>
>>     >>>>>    <https://www.ietf.org/mailman/listinfo/int-area
>>     <https://www.ietf.org/mailman/listinfo/int-area>>
>>     >>>>>
>>     <mailto:[email-masked] <mailto:[email-masked]>>
>> [email-masked] <mailto:[email-masked]>

---

From: James Bensley <[email-masked]>
Date: Mon, 28 Jun 2021 10:17:12 +0200

>> [JB] In that case  - sequencing isn't used for BT Wholesale services.
>

Hi Bob,

Sure, my current employer is a BTW customer and I have checked our
live/current set-up and L2TP sequencing is disabled. I also worked for
 different UK operator previously that was also a BTW customer. I've
checked the old configs and outputs from when I was there, and again
L2TP sequencing was disabled (I'm in close contact with those former
colleagues they confirm it is still disabled). I've asked a few
friends at other UK operators, and none of them have L2TP sequencing
enabled on their BTW tunnels. I've also scanned the BTW docs and can't
find anything that mentions it is supported.

Cheers,
James.

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Jun 2021 09:39:34 +0100

James,

> find anything that mentions it is supported.

[BB] Thx for looking into this in such depth.

My attempt to generalize this UK-specific knowledge: No-one is going to 
add sequencing without a reason, and the only possible reason mentioned 
so far would be stateful compression, which these days would just slow 
things down more than the bandwidth saved.

Cheers

Bob

---

From: tom petch <[email-masked]>
Date: Mon, 28 Jun 2021 08:41:20 +0000

<at the bottom tp>

From: Int-area <[email-masked]> on behalf of Bob Briscoe <[email-masked]>
Sent: 27 June 2021 22:22

James,

<tp>

Perhaps  a coincidence or perhaps not, but an I-D has just been posted 
Author          : Yogesh Nautiyal
Filename        : draft-nautiyal-l2tpext-tunnel-ka-control-channel-00.txt
which uses L2TP sequencing,  I note the author has an affiliation of Juniper.

The draft would need a fair bit of work.

Tom Petch

Bob

---

From: Derek Fawcus <[email-masked]>
Date: Mon, 28 Jun 2021 19:46:13 +0100

> things down more than the bandwidth saved.

I also mentioned PWE fragmentation - rfc4623 - it depends upon the sequence
numbers as a fragment identifier.  So depending upon how it is implemented,
it may implicitly turn on sequencing, without the user knowing.

So the related question then becomes, where is that fragmentation support
being used - if anywhere?

DF

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 29 Jun 2021 11:02:08 +0100

Tom, see [BB2]

> which uses L2TP sequencing,  I note the author has an affiliation of Juniper.

[BB] Thanks for pointing this out. It's quite normal and expected for 
L2TP to use its sequencing facility for a control channel. I'm not 
concerned about that. My original question was scoped to IP as data.

But thanks anyway. Cheers

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 29 Jun 2021 11:23:59 +0100

Derek,

> being used - if anywhere?

[BB] Sorry, I omitted to explain that, in the PWE case, I didn't think 
differential delay would be a problem, because all traffic in the PWE 
would be expected to want the same low delay service. My concern was 
about sequencing of tunnelled packets undoing deliberately 
differentiated delay between different types of IP traffic within an 
aggregate.

But thanks for picking me up on not explaining that.

Cheers

Bob

---

From: Giles Heron <[email-masked]>
Date: Tue, 29 Jun 2021 12:41:08 +0100

Hi Bob,

> On 28 Jun 2021, at 00:23, Bob Briscoe <[email-masked]> wrote:
> 
> Giles, Mark,
> 
> * deliberately reorders packets using a multi-queue scheduler at a node contrived to act as the bottleneck (the BNG)

If that’s the case then the node almost certainly won’t re-order packets within the tunnel, even if it is deliberately reordering packets between clients.  As far as the node is concerned the L2TPv3 tunnel is one flow within one subscriber’s traffic.   BNG reordering is generally between subscribers (e.g. where subscribers get serviced round-robin to ensure fairness), or potentially within classes of service per subscriber (e.g. strict priority for on-net voice and IPTV over Internet traffic).

> 
> In many cases, such a scheduler would be located prior to the tunnel ingress, so not a concern. I believe the DOCSIS rPHY case below falls into that category (both downstream and up).

Agreed

> 
> In contrast, where a BNG sits /within/ the span of an L2TP tunnel, I think it will often (or at least sometimes?) have been constructed as the bottleneck. Any operator having designed such a QoS arrangement would not want to support sequencing...

Yes, BNGs are often planned to be the bottleneck.  But as I say that’s about fairness between subscribers and potentially prioritising different traffic for each subscriber.

> My concern was that 'keep it simple' operators that are using L2TPv2/3 and had not previously bothered with the complexities of QoS might want to support L4S, because it has the potential to cut out queue delay for /all/ traffic. Altho' L4S is eventually for all traffic, it still requires two queues at the bottleneck for transition - one for L4S and one for not-yet-L4S ('Classic') traffic flows, and therefore introduces reordering at the aggregate level...

Again I presume any single tunnel being passed *through* a BNG would either be L4S or not-yet-L4S, and hence not subject to reordering?

Giles

> 
> From the replies so far, even if such 'keep it simple' operators were using compression, I can't see any reason why having to turn off compression and sequencing (in order to support L4S) would be a significant problem nowadays.
> 
> So, in conclusion, I don't think we even need to raise any concerns about L2TP sequencing in the L4S specs.
> 
>>> On 10 Jun 2021, at 12:49, Andrew G. Malis <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> https://www.ietf.org/mailman/listinfo/pals <https://www.ietf.org/mailman/listinfo/pals>
>> 
> 
> -- 
> Bob Briscoe                               http://bobbriscoe.net/ <http://bobbriscoe.net/>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 29 Jun 2021 16:55:47 +0100

Giles,

On 29/06/2021 12:41, Giles Heron wrote:
> Hi Bob,
>
>> On 28 Jun 2021, at 00:23, Bob Briscoe <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Giles, Mark,
>>
>> * deliberately reorders packets using a multi-queue scheduler at a 
>> node contrived to act as the bottleneck (the BNG)
> If that’s the case then the node almost certainly won’t re-order 
> packets within the tunnel, even if it is deliberately reordering 
> packets between clients.  As far as the node is concerned the L2TPv3 
> tunnel is one flow within one subscriber’s traffic.   BNG reordering 
> is generally between subscribers (e.g. where subscribers get serviced 
> round-robin to ensure fairness), or potentially within classes of 
> service per subscriber (e.g. strict priority for on-net voice and IPTV 
> over Internet traffic).

[BB] Not so. There are certainly cases where packets /are/ reordered 
within a per-customer tunnel. For instance, in the BT wholesale case, 
the BNG uses the BBF framework to schedule a set of QoS queues for each 
customer CVLAN. I know this intimately, because it was what I had been 
asked to simplify when we came up with L4S in the first place. Indeed, I 
was given permission to include a picture and description of it in this 
public report we did for the EU-funded project: 
https://riteproject.files.wordpress.com/2015/12/rite-deliverable-3-3-public1.pdf#12

Also, there are probably many cases of accidental reordering too, e.g. 
due to channel bonding or multipath links that are left out of order 
rather than resequenced, or reroutes, including those at L7, when the 
traffic source moves to a CDN after a flow starts, etc. etc.

Whatever, the question is the converse: where packets are /not/ 
deliberately re-ordered by the network operator today, might these 
operators be sequencing IP data emerging from an L2TP tunnel? If so, 
they would have to disable sequencing if they wanted to introduce 
deliberate reordering (to transition to L4S). But (based partly on the 
insights you've given already) I doubt there is much if any sequencing 
going on of IP data over tunnels.

> Yes, BNGs are often planned to be the bottleneck.  But as I say that’s 
> about fairness between subscribers and potentially prioritising 
> different traffic for each subscriber.
>
>>> the aggregation network, and given that compression can occur at 
>>> other layers).  Though given that BNGs inherently keep state per 
>>> subscriber maybe the NPU scaling issues that Stewart mentioned are 
>>> less of an issue in that use case than for MPLS PEs doing PWE?
>>
>> My concern was that 'keep it simple' operators that are using 
>> L2TPv2/3 and had not previously bothered with the complexities of QoS 
>> might want to support L4S, because it has the potential to cut out 
>> queue delay for /all/ traffic. Altho' L4S is eventually for all 
>> traffic, it still requires two queues at the bottleneck for 
>> transition - one for L4S and one for not-yet-L4S ('Classic') traffic 
>> flows, and therefore introduces reordering at the aggregate level...
>
> Again I presume any single tunnel being passed *through* a BNG would 
> either be L4S or not-yet-L4S, and hence not subject to reordering?

[BB] Not so, I'm afraid. L4S is enabled on a per-host basis, and during 
the transition, some traffic in the IP aggregate to (or from) a customer 
site will be L4S, and some Classic (not-yet-L4S).

It's this reordering scenario that begged my original question - whether 
sequencing might be used anywhere for IP data over a tunnel.

Cheers

Bob

>>> payload is the DOCSIS MAC rather than IP (even though, of course, 
>>> most DOCSIS frames ultimately carry an IP payload).
>>>
>>> Giles
>>>
>>>>     Regarding L2TPv3 replacing L2TP: L2TPv2 (RFC 2661) was PPP
>>>>     only. L2TPv3 generalized L2TP to support other L2 (including
>>>>     MPLS, but I don’t want to argue what layer MPLS operates within
>>>>     here). There was never a strong push to replace L2TPv2 used in
>>>>     DSL, Dialup and host VPN software with L2TPv3 (there was one
>>>>     use case for an important L2TP operator that wanted to carry
>>>>     PPPoE over L2TPv3 in DSL, but that was overcome by RFC3817
>>>>     which achieved the same goal without altering the dataplane).
>>>>     Ironically, I would expect to see very little PPP over L2TPv3
>>>>     in the wild, though obviously it is possible.
>>>>
>>>>     off the top of my head. Even I am surprised to see how much
>>>>     L2TP is still out there once you start really looking around ;-)
>>>>
>>>>     Best Regards,
>>>>
>>>>     - Mark
>>>>
>>>>
>>>>
>>>>
>>>>     <[email-masked]
>>>>     <mailto:[email-masked]>> wrote:
>>>>     >
>>>>     published a first version of) protocol for control of split
>>>>     BNG.  L2TP is commonly used for these use cases.
>>>>     >
>>>>     > Yours,
>>>>     > Joel
>>>>     >
>>>>     >>> On 9 Jun 2021, at 12:42, Joel M. Halpern
>>>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>>>     >>>
>>>>     problem because it is stateful so I doubt that many high-scale
>>>>     or high-speed forwarders ever did this.
>>>>     >>>> I think there is an undisclosed assumption that go up
>>>>     enough levels and its IP so sequence number checking in the
>>>>     transport network (as opposed to the transport layer) is not
>>>>     really needed.
>>>>     replace it became niche with, as Andy says, operators who did
>>>>     not want MPLS. Much of what L2TPv3 was intended for was
>>>>     actually done with PW over MPLS with some replacement with by
>>>>     Mac in Mac for cost reasons.
>>>>     >>>>> On 8 Jun 2021, at 22:41, Andrew G. Malis
>>>>     <[email-masked] <mailto:[email-masked]>
>>>>     <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>>>>     >>>>>
>>>>     >>>>> Bob,
>>>>     >>>>>
>>>>     since you specifically mentioned IP data. But it is a case
>>>>     where you would probably see sequencing in use.
>>>>     >>>>>
>>>>     >>>>>    > The L2TP RFC says sequencing /can/ be disabled for
>>>>     IP data, but it
>>>>     >>>>>    > Also, are you aware of any other types of tunnel
>>>>     that might try
>>>>     >>>>>    to keep
>>>>     >>>>>    This would then suggest if L2TP is carrying PPP, the
>>>>     PPP session
>>>>     >>>>>    is not
>> Bob Briscoehttp://bobbriscoe.net/
>

---

From: Giles Heron <[email-masked]>
Date: Tue, 29 Jun 2021 17:16:53 +0100

Hi Bob,

> On 29 Jun 2021, at 16:55, Bob Briscoe <[email-masked]> wrote:
> 
> Giles,
> 
>>> On 28 Jun 2021, at 00:23, Bob Briscoe <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> Giles, Mark,
>>> 
> [BB] Not so. There are certainly cases where packets /are/ reordered within a per-customer tunnel. For instance, in the BT wholesale case, the BNG uses the BBF framework to schedule a set of QoS queues for each customer CVLAN. I know this intimately, because it was what I had been asked to simplify when we came up with L4S in the first place. Indeed, I was given permission to include a picture and description of it in this public report we did for the EU-funded project: https://riteproject.files.wordpress.com/2015/12/rite-deliverable-3-3-public1.pdf#12 <https://riteproject.files.wordpress.com/2015/12/rite-deliverable-3-3-public1.pdf#12>

Yup - what I said.   BNGs may reorder classes of service per sub.  But one L2TP tunnel where the endpoints are either side of the BNG ought to map to a single CoS, right?  And if it does then it won’t be reordered.

> Also, there are probably many cases of accidental reordering too, e.g. due to channel bonding or multipath links that are left out of order rather than resequenced, or reroutes, including those at L7, when the traffic source moves to a CDN after a flow starts, etc. etc.

Channel bonding and multipath links are generally set up very carefully NOT to reorder packets within one flow.

I covered rerouting in my original reply to you.

And I think we can safely ignore CDNs in discussion of L2TP ;)

> Whatever, the question is the converse: where packets are /not/ deliberately re-ordered by the network operator today, might these operators be sequencing IP data emerging from an L2TP tunnel? If so, they would have to disable sequencing if they wanted to introduce deliberate reordering (to transition to L4S). But (based partly on the insights you've given already) I doubt there is much if any sequencing going on of IP data over tunnels.

Agreed.

Giles

> [BB] Not so, I'm afraid. L4S is enabled on a per-host basis, and during the transition, some traffic in the IP aggregate to (or from) a customer site will be L4S, and some Classic (not-yet-L4S).
> 
> It's this reordering scenario that begged my original question - whether sequencing might be used anywhere for IP data over a tunnel. 
> 
> Cheers
> 
> 
> 
> Bob
> 
>> 
>> Giles
>> 
>>>

---

From: Giles Heron <[email-masked]>
Date: Tue, 29 Jun 2021 17:22:55 +0100

Re-sending trimmed to 40k

> On 29 Jun 2021, at 17:16, Giles Heron <[email-masked]> wrote:
> 
> Hi Bob,
> 
>> On 29 Jun 2021, at 16:55, Bob Briscoe <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Giles,
>>