Subject: [DMM] Comments to draft-hmm-dmm-5g-uplane-analysis-01

From: Dino Farinacci <[email-masked]>
Date: Tue, 4 Sep 2018 09:42:10 -0700

Folks, I sent comment to the authors and they asked me to forward them to the WG list. 

I also attended 3GPP/CT4 a couple of weeks ago and can report on it in Bangkok. Maybe Satoru can as well. There were two IETF-related presentations given.

Thanks,
Dino

> Begin forwarded message:
> 
> From: Dino Farinacci <[email-masked]>
> Subject: My comments to draft-hmm-dmm-5g-uplane-analysis-01 
> Date: August 30, 2018 at 4:17:09 PM PDT
> To: Dan Voyer <[email-masked]>, Satoru Matsushima <[email-masked]>, [email-masked]
> Cc: Shunsuke Homma <[email-masked]>, [email-masked], Dino Farinacci <[email-masked]>
> 
> Your draft text comes first and is indented and my comments follow.
> 
> Cheers,
> Dino
> 
> 
> In this section you should make it very clear where the tunnel terminates. It is not clear from the diagram on the N3 interface if it terminates on the eNodeB or the UE.
> 
> 
> Dan made this point verbally to me about unidirectional versus bidirectional tunnels. I don’t know if it matters because it all comes down to the amount of state that a router uses to terminate the tunnel. In cisco routers, we also used one “virtual interface” (called an idb) to support a tunnel. And we used it and all the accounting associated with it for both input and output counters, addressing, encap/decap functions, etc.
> 
> Dan made the comment that if tunnels were bidirectional, you reduce the tunnel state in half. That is only if an implementation used two data structues for different TEID values. But that would be a poor implementation where you could store a rec_teid and send_teid in one data structure not duplicating other parts of the data structure.
> 
>> 
> 
> All tunneling protocols should “copy inner to outer” fields like TOS, flow label, and TTL to outer header. That needs to be done by default and then later if there is marking policy, it happens at boundaries only in the outer header.
> 
> This won’t be an issue with IPv6/SRv6 but if one wants to preserve the original host values, and there is a rewrite at the boundaries, you will lose the information. People need to decide if this is a hard requirement.
> 
> 
> It just doesn’t matter. Link layer CRCs are very good these days and does a good job protecting bit errors and bad actor corruption of packet data.
> 
> 
> SRv6 will have this problem too. With just 1 SID can still put an user’s MTU sized packet over the effective path MTU.
> 
> 
> If this is the case, then the ECMP algorithm in GTP is broken. No packets should be reordered. So why are they?
> 
> 
> It was never clear to me and no one could ever explain exactly why a TEID is needed. I presumed for accounting reasons. But if there was a one-to-one mapping between tunnel and user, why couldn’t the inner addresses be used for accounting?
> 
> 
> How can packets be sent if the session is not setup. If the session is not setup, the encapsulator should have no state. And packets should be dropped locally and not go far to get an error back. This sounds architecturally broken.
> 
> 
> You should explain in summary form the model the control-plane uses. Does it use TCP for reliability, does it use multicast, is it like a routing protocol, is it like a management protocol. What are the failure modes, the state/bandwidth tradeoffs.
> 
> 
> 
> You should state if a UP/CP pair that is self contained with a slice. Or is there a CP that operates each UP in each slice. And how much fault isolation and resource reservation is obtained by slicing. I hope its more like containers than VRFs.  ;-)
> 
> 
> What I got from the CT4 meeting is that the LISP mapping system can be put in a box above and you could combine SMF, AMF, and PCF functions just in the mapping database data structure. I think more of those compoents could be combined, but I don’t understand what they all do yet. LOL.  ;-)
> 
> Note a holistic architecture would have the N6 interface terminate on a top-of-rack router in the DN data-center. 
> 
> 
> So I think you need a CP equivalent to this document. Because the complexity, even though present here in the UP, is order of magnitude more in the CP and the costs of scale, security, and manageability is in the control-plane. Have you consider doing this?
> 
> Like I mentiioned to Satoru in West Palm Beach, without a level of indirection in routing, how do you scale the mobility part with SRv6? That is if you carry host routes, that can only scale so far, and it forces you to scope the host routes to small domains. That means, your mobility is limited within that domain. That tradeoff needs to be characterized. And I think most importantly what users need and want is mobility across WiFi and 3GPP based data-links. That is a problem that really needs to be solved since the RAN RF frequencies are just going to get more jammed with billions of IoT devices coming online. That means smaller cells, that means IP address changes, so IP address mobility is a huge problem coming that needs to be solved. 
> 
> Payment systems will be on phones. Payment systems will need to be secure. Payment identity will be hash of your public-key, your network address will need to be the same, that means you can change the IP address of a system when it roams ANYWHERE.
> 
> Some things to think about.
> 
> Cheers,
> Dino
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Begin forwarded message:
> 
> From: Dino Farinacci <[email-masked]>
> Subject: My comments to draft-hmm-dmm-5g-uplane-analysis-01 
> Date: August 30, 2018 at 4:17:09 PM PDT
> To: Dan Voyer <[email-masked]>, Satoru Matsushima <[email-masked]>, [email-masked]
> Cc: Shunsuke Homma <[email-masked]>, [email-masked], Dino Farinacci <[email-masked]>
> 
> Your draft text comes first and is indented and my comments follow.
> 
> Cheers,
> Dino
> 
> 
> In this section you should make it very clear where the tunnel terminates. It is not clear from the diagram on the N3 interface if it terminates on the eNodeB or the UE.
> 
> 
> Dan made this point verbally to me about unidirectional versus bidirectional tunnels. I don’t know if it matters because it all comes down to the amount of state that a router uses to terminate the tunnel. In cisco routers, we also used one “virtual interface” (called an idb) to support a tunnel. And we used it and all the accounting associated with it for both input and output counters, addressing, encap/decap functions, etc.
> 
> Dan made the comment that if tunnels were bidirectional, you reduce the tunnel state in half. That is only if an implementation used two data structues for different TEID values. But that would be a poor implementation where you could store a rec_teid and send_teid in one data structure not duplicating other parts of the data structure.
> 
>> 
> 
> All tunneling protocols should “copy inner to outer” fields like TOS, flow label, and TTL to outer header. That needs to be done by default and then later if there is marking policy, it happens at boundaries only in the outer header.
> 
> This won’t be an issue with IPv6/SRv6 but if one wants to preserve the original host values, and there is a rewrite at the boundaries, you will lose the information. People need to decide if this is a hard requirement.
> 
> 
> It just doesn’t matter. Link layer CRCs are very good these days and does a good job protecting bit errors and bad actor corruption of packet data.
> 
> 
> SRv6 will have this problem too. With just 1 SID can still put an user’s MTU sized packet over the effective path MTU.
> 
> 
> If this is the case, then the ECMP algorithm in GTP is broken. No packets should be reordered. So why are they?
> 
> 
> It was never clear to me and no one could ever explain exactly why a TEID is needed. I presumed for accounting reasons. But if there was a one-to-one mapping between tunnel and user, why couldn’t the inner addresses be used for accounting?
> 
> 
> How can packets be sent if the session is not setup. If the session is not setup, the encapsulator should have no state. And packets should be dropped locally and not go far to get an error back. This sounds architecturally broken.
> 
> 
> You should explain in summary form the model the control-plane uses. Does it use TCP for reliability, does it use multicast, is it like a routing protocol, is it like a management protocol. What are the failure modes, the state/bandwidth tradeoffs.
> 
> 
> 
> You should state if a UP/CP pair that is self contained with a slice. Or is there a CP that operates each UP in each slice. And how much fault isolation and resource reservation is obtained by slicing. I hope its more like containers than VRFs.  ;-)
> 
> 
> What I got from the CT4 meeting is that the LISP mapping system can be put in a box above and you could combine SMF, AMF, and PCF functions just in the mapping database data structure. I think more of those compoents could be combined, but I don’t understand what they all do yet. LOL.  ;-)
> 
> Note a holistic architecture would have the N6 interface terminate on a top-of-rack router in the DN data-center. 
> 
> 
> So I think you need a CP equivalent to this document. Because the complexity, even though present here in the UP, is order of magnitude more in the CP and the costs of scale, security, and manageability is in the control-plane. Have you consider doing this?
> 
> Like I mentiioned to Satoru in West Palm Beach, without a level of indirection in routing, how do you scale the mobility part with SRv6? That is if you carry host routes, that can only scale so far, and it forces you to scope the host routes to small domains. That means, your mobility is limited within that domain. That tradeoff needs to be characterized. And I think most importantly what users need and want is mobility across WiFi and 3GPP based data-links. That is a problem that really needs to be solved since the RAN RF frequencies are just going to get more jammed with billions of IoT devices coming online. That means smaller cells, that means IP address changes, so IP address mobility is a huge problem coming that needs to be solved. 
> 
> Payment systems will be on phones. Payment systems will need to be secure. Payment identity will be hash of your public-key, your network address will need to be the same, that means you can change the IP address of a system when it roams ANYWHERE.
> 
> Some things to think about.
> 
> Cheers,
> Dino
> 
> 
> 
> 
> 
> 
> 
> 
>

---

From: Sridhar Bhaskaran <[email-masked]>
Date: Wed, 5 Sep 2018 14:25:49 +0530

Dear Dino,

Some clarifications on your comments

>>> It was never clear to me and no one could ever explain exactly why a
TEID is needed. I presumed for accounting reasons. But if there was a
one-to-one mapping between tunnel and user, why couldn’t the inner
addresses be used for accounting?

[Sridhar] In EPC, each bearer has a GTPU tunnel. TEID identifies a tunnel
and hence consequently a bearer. Once the bearer context is identified the
QoS and charging policy applicable to the bearer is applied. So the purpose
of TEID is not just for accounting. Its for QoS treatment, charging and
bearer context identification.

In 5G, each PDU session has a GTPU tunnel. So TEID identifies the PDU
session whereas the QFI carried in GTPU extension header identifies the
flow. So in 5G TEID + QFI is used for QoS treatment and charging.

>>> How can packets be sent if the session is not setup. If the session is
not setup, the encapsulator should have no state. And packets should be
dropped locally and not go far to get an error back. This sounds
architecturally broken.

[Sridhar] The purpose of GTP-U error indication is to signal in band to the
sender that a GTP-U tunnel endpoint (TEID) at the receiving side is lost
for any reason. "No session exist" does not mean Session is not setup. "No
session exist" scenario after a session setup can happen due to local error
conditions, bearers released for administrative reasons etc.

>>> You should explain in summary form the model the control-plane uses.
Does it use TCP for reliability, does it use multicast, is it like a
routing protocol, is it like a management protocol. What are the failure
modes, the state/bandwidth tradeoffs.

[Sridhar] Explaining all these in IETF draft is simply reproducing what is
already there in TS 29.244. A reference to TS 29.244 should be enough. See
section 6.4 of TS 29.244 for reliable delivery of PFCP messages.

Thanks
Sridhar
3GPP CT4 Delegate

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 5 Sep 2018 09:16:34 -0700

> Some clarifications on your comments

I am going to use general terms here so we don’t get hung up in IETF and/or 3GPP terminology. Which doesn’t make things clear to anyone really. So we can stay on point.

> [Sridhar] In EPC, each bearer has a GTPU tunnel. TEID identifies a tunnel and hence consequently a bearer. Once the bearer context is identified the QoS and charging policy applicable to the bearer is applied. So the purpose of TEID is not just for accounting. Its for QoS treatment, charging and bearer context identification.

You told me what a TEID is but you didn’t say why you need to use it versus using the destination IP address for the tunnel.

> In 5G, each PDU session has a GTPU tunnel. So TEID identifies the PDU session whereas the QFI carried in GTPU extension header identifies the flow. So in 5G TEID + QFI is used for QoS treatment and charging.

When a packet is encapsulated in a tunnel, a packet has 4 addresses, which tells us (1) the UE, (2) the destination it is talking to, (3) the encapsualting node, and (4) the decapsulating node.

So again, why use more space in the packet, when you have sufficient information to identify a user, and therefore their packet policy?

> [Sridhar] The purpose of GTP-U error indication is to signal in band to the sender that a GTP-U tunnel endpoint (TEID) at the receiving side is lost for any reason. "No session exist" does not mean Session 

What does lost mean? You mean the path from encapsulator to decapsulator is not working? And since we are in a packet network, that path can be restored quite quickly.

> is not setup. "No session exist" scenario after a session setup can happen due to local error conditions, bearers released for administrative reasons etc.

So at the encapsulator, do you choose another decapsultor? Note that tunnels *usually* stay up since the topology that realizes the tunnel is robust and redundant.

> [Sridhar] Explaining all these in IETF draft is simply reproducing what is already there in TS 29.244. A reference to TS 29.244 should be enough. See section 6.4 of TS 29.244 for reliable delivery of PFCP messages.

Just pointing people to drafts doesn’t help in understanding. It requires people to go off, put in a lot of time where the odds are their question will not be answered.

The points I’m trying to make is not “what it is” you are designing but “why you did what you did” in the design. That is rarely in the specs.

Dino

---

From: Behcet Sarikaya <[email-masked]>
Date: Wed, 5 Sep 2018 11:34:44 -0500

On Wed, Sep 5, 2018 at 3:56 AM Sridhar Bhaskaran <
[email-masked]> wrote:

> [Sridhar] The purpose of GTP-U error indication is to signal in band to
> the sender that a GTP-U tunnel endpoint (TEID) at the receiving side is
> lost for any reason. "No session exist" does not mean Session is not setup.
> "No session exist" scenario after a session setup can happen due to local
> error conditions, bearers released for administrative reasons etc.
>
I checked in IETF IANA UDP assignments,

pfcp 8805 udp Destination Port number for PFCP [Kimmo_Kymalainen
<https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?&page=112#Kimmo_Kymalainen>
] [Kimmo_Kymalainen
<https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?&page=112#Kimmo_Kymalainen>
] 2017-05-08
 for packet forwarding control protocol. What is PFCP, is it GTP-U?

Behcet

> Thanks
> Sridhar
> 3GPP CT4 Delegate
> _______________________________________________
> dmm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dmm
>

---

From: Sridhar Bhaskaran <[email-masked]>
Date: Thu, 6 Sep 2018 15:54:05 +0530

My comments inline marked [SB]

> > [Sridhar] In EPC, each bearer has a GTPU tunnel. TEID identifies a
> tunnel and hence consequently a bearer. Once the bearer context is
> identified the QoS and charging policy applicable to the bearer is applied.
> So the purpose of TEID is not just for accounting. Its for QoS treatment,
> charging and bearer context identification.
>
> You told me what a TEID is but you didn’t say why you need to use it
> versus using the destination IP address for the tunnel.
>
>
> When a packet is encapsulated in a tunnel, a packet has 4 addresses, which
> tells us (1) the UE, (2) the destination it is talking to, (3) the
> encapsualting node, and (4) the decapsulating node.
>
> So again, why use more space in the packet, when you have sufficient
> information to identify a user, and therefore their packet policy?
>
> [SB] Lets say we only use UE IP address and no TEID. How will you identify
the bearer context the packet belongs? One UE may use multiple radio
bearers / QoS flows. DSCP in IPv4 and Flow Label in IPv6 is one option but
these are IP level markings which could be changed by any on path routers.
In order to uniquely identify the bearer / qos flow a particular packet for
a UE belongs, GTPU uses TEID.

Also the IP addresses (at least for IPv4) allocated to UE by PGW / SMF are
not always unique. The same IP pools can be shared across multiple PDNs /
DNs as long as these PDNs / DNs are separate autonomous networks. So just
relying on UE IP address alone will result in wrong context identification
for the uplink traffic. There is a clear one to one mapping of Radio bearer
to the EPS bearer / QoS flow required all the way upto the anchor node for
charging and QoS treatment. This comes from the requirements in stage 2
documents (c.f section 4.7 of TS 23.401 for EPC and 5.7 of TS 23.501 for
5G).

There are also requirements to support non-IP protocols like Ethernet PDU
and Unstructured PDU types in 5G.

>>
> > >>> How can packets be sent if the session is not setup. If the session
> is not setup, the encapsulator should have no state. And packets should be
> lost for any reason. "No session exist" does not mean Session
>
> What does lost mean? You mean the path from encapsulator to decapsulator
> is not working? And since we are in a packet network, that path can be
> restored quite quickly.
>

[SB] Lost here means - the "context" at the receiving entity is deleted.
For e.g due to administrative reasons, gNB or eNB removes the radio bearers
and correspondingly the GTPU context. If gNB or eNB loses a context for
known reasons, there could be signaling from gNB / eNB to AMF/MME and
corresponding removal of PDU session / EPS bearer at the core network. But
if the context is removed due to administrative reasons or unforeseen local
errors, signaling from gNB / eNB to AMF / MME may not happen. Hence the
GTPU error indication is an inband error detection mechanism.

Note TEID identifies a context at the receiving side. So all GTPU error
indication tells is that the receiver is not able to identify any context
for the received TEID.

> > is not setup. "No session exist" scenario after a session setup can
> happen due to local error conditions, bearers released for administrative
> reasons etc.
>
> So at the encapsulator, do you choose another decapsultor? Note that
> tunnels *usually* stay up since the topology that realizes the tunnel is
> robust and redundant.
>
> >>
> > [Sridhar] Explaining all these in IETF draft is simply reproducing what
> is already there in TS 29.244. A reference to TS 29.244 should be enough.
> See section 6.4 of TS 29.244 for reliable delivery of PFCP messages.
>
> Just pointing people to drafts doesn’t help in understanding. It requires
> people to go off, put in a lot of time where the odds are their question
> will not be answered.
>

[SB] TS 29.244 is not a draft but rather a full fledged technical
specification. The issue with repeating content from elsewhere instead of
just pointing is the risk of providing inaccurate information in IETF draft.

> The points I’m trying to make is not “what it is” you are designing but
> “why you did what you did” in the design. That is rarely in the specs.
>
> [SB] 3GPP SA/CT groups follow a waterfall model - service requirements are
in 22 series specs, system architecture in 23 series specs and protocol in
29 series specs. You could always trace back the reason for a particular
design aspect in a protocol to a corresponding architectural requirement in
23 series or a system requirement in 22 series specs.

For e.g as I explained above the reason for the existence of TEID - its the
one to one mapping of radio bearer to an EPS bearer / QOS flow that
demanded it.

---

From: Sridhar Bhaskaran <[email-masked]>
Date: Thu, 6 Sep 2018 15:57:33 +0530

Dear Behcet,

>>What is PFCP, is it GTP-U?

PFCP is a control plane protocol used between control plane function and
user plane function. In EPC, the PFCP protocol is used on the Sx interface.
In 5G its used on the N4 interface. It is used to set up the packet
classifiers and forwarding rules in the user plane function.

Thanks
Sridhar

---

From: Tom Herbert <[email-masked]>
Date: Thu, 6 Sep 2018 08:40:22 -0700

On Thu, Sep 6, 2018 at 3:24 AM, Sridhar Bhaskaran
<[email-masked]> wrote:
>> > >>> one-to-one mapping between tunnel and user, why couldn’t the inner addresses
>> > >>> be used for accounting?
>> >
> the bearer context the packet belongs? One UE may use multiple radio bearers
> / QoS flows. DSCP in IPv4 and Flow Label in IPv6 is one option but these are
> IP level markings which could be changed by any on path routers. In order to
> uniquely identify the bearer / qos flow a particular packet for a UE
> belongs, GTPU uses TEID.

Sridhar,

Couldn't the TEID be encoded in the outer IP address of an
encpasulation or network overlay in a similar way that VNIs are
encoded in IP addresses in virtual networking?

Tom

>> > the sender that a GTP-U tunnel endpoint (TEID) at the receiving side is lost
>> > for any reason. "No session exist" does not mean Session
>>
> [SB] Lost here means - the "context" at the receiving entity is deleted. For
> e.g due to administrative reasons, gNB or eNB removes the radio bearers and
> correspondingly the GTPU context. If gNB or eNB loses a context for known
> reasons, there could be signaling from gNB / eNB to AMF/MME and
> errors, signaling from gNB / eNB to AMF / MME may not happen. Hence the GTPU
> error indication is an inband error detection mechanism.
>
>> > >>> Does it use TCP for reliability, does it use multicast, is it like a routing
>> > >>> protocol, is it like a management protocol. What are the failure modes, the
>> > >>> state/bandwidth tradeoffs.
>> >
> one to one mapping of radio bearer to an EPS bearer / QOS flow that demanded
> it.
>
>
>
>

---

From: Behcet Sarikaya <[email-masked]>
Date: Thu, 6 Sep 2018 11:18:13 -0500

On Thu, Sep 6, 2018 at 10:40 AM Tom Herbert <[email-masked]> wrote:

> bearers
ILA if used would remove any need for tunneling and TEID is for tunneling.

Actually if you read 29.244 it is completely based on legacy protocols with
no IdLoc content at all, as Shunsuke mentioned.

Behcet

> For
> GTPU
> demanded

---

From: Tom Herbert <[email-masked]>
Date: Thu, 6 Sep 2018 09:21:54 -0700

> ILA if used would remove any need for tunneling and TEID is for tunneling.
Behcet,

I was thinking if TEID is need then that can be encoded in a locator
easily enough.

Tom

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 6 Sep 2018 09:38:30 -0700

Sridhar,

> [SB] Lets say we only use UE IP address and no TEID. How will you identify the bearer context the packet belongs? One UE may use multiple radio bearers / QoS flows. DSCP in IPv4 and Flow Label in IPv6 is one option but these are IP level markings which could be changed by any on path routers. In order to uniquely identify the bearer / qos flow a particular packet for a UE belongs, GTPU uses TEID. 
> 
> Also the IP addresses (at least for IPv4) allocated to UE by PGW / SMF are not always unique. The same IP pools can be shared across multiple PDNs / DNs as long as these PDNs / DNs are separate autonomous networks. So just relying on UE IP address alone will result in wrong context identification for the uplink traffic. There is a clear one to one mapping of Radio bearer to the EPS bearer / QoS flow required all the way upto the anchor node for charging and QoS treatment. This comes from the requirements in stage 2 documents (c.f section 4.7 of TS 23.401 for EPC and 5.7 of TS 23.501 for 5G). 
> 
> There are also requirements to support non-IP protocols like Ethernet PDU and Unstructured PDU types in 5G.

Then you have proven there is a lot of simplication opportunity. That is, use the IP header for charging, and for ethernet serivce encapsulate that at the UE so the eNodeB has a common and single way of classifying packets.

> [SB] Lost here means - the "context" at the receiving entity is deleted. For e.g due to administrative reasons, gNB or eNB removes the radio bearers and correspondingly the GTPU context. If gNB or eNB loses a context for known reasons, there could be signaling from gNB / eNB to AMF/MME and corresponding removal of PDU session / EPS bearer at the core network. But if the context is removed due to administrative reasons or unforeseen local errors, signaling from gNB / eNB to AMF / MME may not happen. Hence the GTPU error indication is an inband error detection mechanism.
> 
> Note TEID identifies a context at the receiving side. So all GTPU error indication tells is that the receiver is not able to identify any context for the received TEID.

Right, you are using “sessions” or “connections” at a packet layer. That is the fundamental problem. Why can't this 3GPP network be more like an IP network? You know that most of the critical traffic is IP traffic, even voice these days.

It looks like the dmm WG should propose a slice that is called an IP slice where we just do good ole datagram routing over it. Do charging with packet counters and strive to make the 3GPP network like an ethernet (or wifi network). 

I would like to hear arguments why this can’t be done. It is definitely a good place to start for 5G IMO. You do this and we can give users IP mobility from 3GPP to wifi more seamlessly.

IMO, if 3GPP does not solve the 3GPP to wifi mobility problem, it has failed to solve real world problems.

> [SB] TS 29.244 is not a draft but rather a full fledged technical specification. The issue with repeating content from elsewhere instead of just pointing is the risk of providing inaccurate information in IETF draft.

My point is that if you would simply summarize here in email, a lot of people could learn from you. That would be user friendly. Refering to a sequence of documents is not really helpfull.

> [SB] 3GPP SA/CT groups follow a waterfall model - service requirements are in 22 series specs, system architecture in 23 series specs and protocol in 29 series specs. You could always trace back the reason for a particular design aspect in a protocol to a corresponding architectural requirement in 23 series or a system requirement in 22 series specs.

Are you saying since there are so many specifications, you can’t in an email explain “why you did it this way”?

Thanks for the reply,
Dino

---

From: Marco Liebsch <[email-masked]>
Date: Thu, 6 Sep 2018 16:39:07 +0000

Tom, Behcet, I think TEID may still be needed in some cases, e.g. for mapping to a radio bearer
or to avoid superfluous packet classification if it has been done on the packet's path beforehand
already.

IMO, for non-encapsulation protocols, overloading of id-loc space seems interesting if the attribute
cannot be carried in extensions. What about encoding QoS class/flow IDs in that space as well ;-) ?

marco

-----Original Message-----
From: dmm [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Donnerstag, 6. September 2018 18:22
To: Behcet Sarikaya
Cc: [email-masked]; dmm
Subject: Re: [DMM] Comments to draft-hmm-dmm-5g-uplane-analysis-01

Behcet,

Tom

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 6 Sep 2018 09:40:20 -0700

> encoded in IP addresses in virtual networking?

There are lots of ways to do it. The point is, was an additional 32 bits necessary solely for this purpose. 

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 6 Sep 2018 09:44:30 -0700

> I was thinking if TEID is need then that can be encoded in a locator

Not if a locator is a PGW that is shared by many UEs. 

3GPP wants per bearer awareness so they need a specific ID, that could have been the UE’s IP address. And with IPv6 it can be unique and not the issue that Sridhar brought up.

If ILA was in use, just use the ILA-ID for this purpose.

Dino

---

From: Tom Herbert <[email-masked]>
Date: Thu, 6 Sep 2018 10:00:15 -0700

> cannot be carried in extensions. What about encoding QoS class/flow IDs in that space as well ;-) ?
Marco,

Yes, there is a lot of opportunity to encode different things in
addresses. The giantic address space of IPv6 almost begs us to do
that! :-) The amount of data is limited, so it's unlikely that we can
encapsulate all of the GTP-U extension headers. But with some careful
planning, we could probably encode the critical information that is
most common for fast data path. Address encoding is critical to ILA
and is what can eliminate the overhead of encapsulation for a fast,
low latency, data path.

Tom

---

From: Arashmid Akhavain <[email-masked]>
Date: Thu, 6 Sep 2018 20:57:38 +0000

Today TEID is a MUST in mobile core and as Sridhar indicated in his emails identifies a bearer or context.
I agree with Tom. A mechanism to encode the TEID should work.

Arashmid

From: dmm [mailto:[email-masked]] On Behalf Of Behcet Sarikaya
Sent: 06 September 2018 12:18
To: Tom Herbert <[email-masked]>
Cc: [email-masked]; dmm <[email-masked]>
Subject: Re: [DMM] Comments to draft-hmm-dmm-5g-uplane-analysis-01

On Thu, Sep 6, 2018 at 10:40 AM Tom Herbert <[email-masked]<mailto:[email-masked]>> wrote:
On Thu, Sep 6, 2018 at 3:24 AM, Sridhar Bhaskaran

Sridhar,

Tom

Actually if you read 29.244 it is completely based on legacy protocols with no IdLoc content at all, as Shunsuke mentioned.

Behcet
> [email-masked]<mailto:[email-masked]>

_______________________________________________
dmm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/dmm

---

From: Arashmid Akhavain <[email-masked]>
Date: Thu, 6 Sep 2018 21:49:27 +0000

Dino brought up a good point. Here is my two cents worth:

As it was explained by Sridhar,  each UE can have multiple contexts. For example, today some operators provide Data and VoLTE service to their customers. These two services are represented by separate GTP tunnels in the core with each tunnel tied up to a particular QoS.

IPv4 didn't fit the bill when GTP work was under way as it couldn't uniquely identify multiple UE sessions/context/bearer. So, GTP and TEID did the job. But I agree with Dino that IPv6 is much more versatile and is definitely worth looking at as an alternative.

A factor worth considering though is that the use of GTP and TEID in mobile core allows operators to deal with QoS on their own terms. The tunnels with specific operator-controlled QoS are established by the control plane between eNB, SGW, and PGW. UEs or applications sitting in the UEs have no say in this. Well at least till the packet exits operator's network.

Using the information in UE's IP packet header can jeopardise the above tight QoS control. I think going down this path, operators need proof that they will be still in the driving seat and QoS cannot be dictated/tampered by the UE or any application running in it. 

Now, here is an interesting question for the operators. Would any operator be interested in allowing QoS  to be set by the UE or by applications running in the UE and charged for by the network? "Yes" could potentially imply impacts on the air interface, UE resource block allocation and can make scheduling on the RAN side much more complex.

Arashmid

> From: dmm [mailto:[email-masked]] On Behalf Of Dino Farinacci
> Sent: 06 September 2018 12:45
> 3GPP wants per bearer awareness so they need a specific ID, that could have
> been the UE’s IP address. And with IPv6 it can be unique and not the issue
> that Sridhar brought up.
>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 6 Sep 2018 15:15:51 -0700

> Now, here is an interesting question for the operators. Would any operator be interested in allowing QoS  to be set by the UE or by applications running in the UE and charged for by the network? "Y…

Hi Arashmid,

I might pose the question a bit differently: Do operators want to
offer a rich set of services, e.g. QoS, and allow applications request
what services they want applied for their packets? I believe the
answer to that should be "yes" since it allows operators the chance
offer and monetize services that can benefit users. That is the easier
question to ask. The harder one is _how_ to do this in a secure,
generic, net neutrality compliant, practical, and fair manner that
doesn't require users to divulge the details of their content to the
provider or the whole Internet. Firewall and Service Tickets is being
proposed as one such mechanism to solve this (see
https://tools.ietf.org/html/draft-herbert-fast).

Tom

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 6 Sep 2018 15:59:13 -0700

> Dino brought up a good point. Here is my two cents worth:

Not sure which point.

> IPv4 didn't fit the bill when GTP work was under way as it couldn't uniquely identify multiple UE 

There is no reason why it shouldn’t. And IPv6, for this use-case doesn’t add anything new other than a 28 bit traffic-class/flow-label that can provide more bits for “new functionality”.

> sessions/context/bearer. So, GTP and TEID did the job. But I agree with Dino that IPv6 is much more versatile and is definitely worth looking at as an alternative.

That is not what I said. I said “IP could have solved this problem”. And “IP” means either IPv4 or IPv6, or both at the same time.

> A factor worth considering though is that the use of GTP and TEID in mobile core allows operators to deal with QoS on their own terms. The tunnels with specific operator-controlled QoS are establis…

The problem with one header, is that if you re-mark (known as PHB markign in the ole days) you lose the original value. Encapsulation is useful here because you can map the inner to outer and anywhere along the path you can PHB remark on the outer header. And then the destination can see the orignal source’s ToS/QoS/TC/flow-label whatever.

> Using the information in UE's IP packet header can jeopardise the above tight QoS control. I think going

Not if you encapsulate. But note with SRv6, you can possibly retain the original flow-label if the SID can retain those bits before overwriting the destination address from the option’s value.

Dino

>  down this path, operators need proof that they will be still in the driving seat and QoS cannot be dictated/tampered by the UE or any application running in it. 
>

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 6 Sep 2018 16:01:05 -0700

I’d ask the question another way:

Would users like to set QoS bits that may charge them more for service? Could they set bits to get cheaper service?

Let alone if the operator can deliver the service (in this net-neutrality-less era).

Dino

> On Sep 6, 2018, at 3:15 PM, Tom Herbert <[email-masked]> wrote:
>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 6 Sep 2018 17:21:30 -0700

> Would users like to set QoS bits that may charge them more for service? Could they set bits to get cheaper service?
Hey, if I could set some bits and save a few bucks (legally) on my
mobile phone bill each month that'd be great! How do I do that? ;-)

In lieu of those magic bits, what I'd settle for now as as a user is
better, more transparent accounting and pay-per-use for service. I
don't have a problem paying more to get better Internet service **if**
it's clear I'm getting money's worth. For instance, for the past few
months my home Internet provider is currently charging me for going
over my data limits each month, but they can't give me any detailed
break down of my Internet usage. They can't even tell me _how_ they're
doing accounting since they've outsourced it to a third party company.
Very irritating! And then, of course, there's the story of
firefighters fighting wild fires in California and having their data
rate throttled to 1/100th of the nominal rate even though they were on
the "unlimited" data plan. Not just irritating, but this put lives at
risk! Such problems can be fix with precise accounting and
transparency on services and service plans.

Tom

---

From: Alexandre Petrescu <[email-masked]>
Date: Fri, 7 Sep 2018 15:31:35 +0200

Le 04/09/2018 à 18:42, Dino Farinacci a écrit :
> I also attended 3GPP/CT4 a couple of weeks ago and can report on it in 
> Bangkok. Maybe Satoru can as well. There were two IETF-related 
> presentations given.
> 
> Thanks,
> Dino
> 
>> Begin forwarded message:
>>
>> *From: *Dino Farinacci <[email-masked] <mailto:[email-masked]>>
>> *Subject: **My comments to draft-hmm-dmm-5g-uplane-analysis-01 *
>> *Date: *August 30, 2018 at 4:17:09 PM PDT
>> *To: *Dan Voyer <[email-masked] <mailto:[email-masked]>>, 
>> Satoru Matsushima <[email-masked] 
>> <mailto:[email-masked]>>, 
>> [email-masked] 
>> <mailto:[email-masked]>
>> *Cc: *Shunsuke Homma <[email-masked] 
>> <mailto:[email-masked]>>, [email-masked] 
>> <mailto:[email-masked]>, Dino Farinacci 
>> <[email-masked] <mailto:[email-masked]>>
>>
>> In this section you should make it very clear where the tunnel 
>> terminates. It is not clear from the diagram on the N3 interface if it 
>> terminates on the eNodeB or the UE.

Even if it says it terminates at UE it must say what is the 'UE' for them.

Because some people may think 'UE' is the smartphone, but others can 
think it is the modem in the smartphone.

A typical smartphone has many processors and only some of them can be 
easily verifiable in a straightforward manner (open source, root access, 
etc.)

If the GTP-U terminates to a point that is hardly verifiable by average 
readers then its interest is very limited.

Alex

>>
>>
>> Dan made this point verbally to me about unidirectional versus 
>> bidirectional tunnels. I don’t know if it matters because it all comes 
>> down to the amount of state that a router uses to terminate the 
>> tunnel. In cisco routers, we also used one “virtual interface” (called 
>> an idb) to support a tunnel. And we used it and all the accounting 
>> associated with it for both input and output counters, addressing, 
>> encap/decap functions, etc.
>>
>> Dan made the comment that if tunnels were bidirectional, you reduce 
>> the tunnel state in half. That is only if an implementation used two 
>> data structues for different TEID values. But that would be a poor 
>> implementation where you could store a rec_teid and send_teid in one 
>> data structure not duplicating other parts of the data structure.
>>
>>>
>>
>> All tunneling protocols should “copy inner to outer” fields like TOS, 
>> flow label, and TTL to outer header. That needs to be done by default 
>> and then later if there is marking policy, it happens at boundaries 
>> only in the outer header.
>>
>> This won’t be an issue with IPv6/SRv6 but if one wants to preserve the 
>> original host values, and there is a rewrite at the boundaries, you 
>> will lose the information. People need to decide if this is a hard 
>> requirement.
>>
>>
>> It just doesn’t matter. Link layer CRCs are very good these days and 
>> does a good job protecting bit errors and bad actor corruption of 
>> packet data.
>>
>>
>> SRv6 will have this problem too. With just 1 SID can still put an 
>> user’s MTU sized packet over the effective path MTU.
>>
>>
>> If this is the case, then the ECMP algorithm in GTP is broken. No 
>> packets should be reordered. So why are they?
>>
>>
>> not setup, the encapsulator should have no state. And packets should 
>> be dropped locally and not go far to get an error back. This sounds 
>> architecturally broken.
>>
>>
>> routing protocol, is it like a management protocol. What are the 
>> failure modes, the state/bandwidth tradeoffs.
>>
>>
>>
>> You should state if a UP/CP pair that is self contained with a slice. 
>> Or is there a CP that operates each UP in each slice. And how much 
>> fault isolation and resource reservation is obtained by slicing. I 
>> hope its more like containers than VRFs.  ;-)
>>
>>
>> What I got from the CT4 meeting is that the LISP mapping system can be 
>> put in a box above and you could combine SMF, AMF, and PCF functions 
>> just in the mapping database data structure. I think more of those 
>> compoents could be combined, but I don’t understand what they all do 
>> yet. LOL.  ;-)
>>
>> Note a holistic architecture would have the N6 interface terminate on 
>> a top-of-rack router in the DN data-center.
>>
>>
>> So I think you need a CP equivalent to this document. Because the 
>> complexity, even though present here in the UP, is order of magnitude 
>> more in the CP and the costs of scale, security, and manageability is 
>> in the control-plane. Have you consider doing this?
>>
>> Like I mentiioned to Satoru in West Palm Beach, without a level of 
>> indirection in routing, how do you scale the mobility part with SRv6? 
>> That is if you carry host routes, that can only scale so far, and it 
>> forces you to scope the host routes to small domains. That means, your 
>> mobility is limited within that domain. That tradeoff needs to be 
>> characterized. And I think most importantly what users need and want 
>> is mobility across WiFi and 3GPP based data-links. That is a problem 
>> that really needs to be solved since the RAN RF frequencies are just 
>> going to get more jammed with billions of IoT devices coming online. 
>> That means smaller cells, that means IP address changes, so IP address 
>> mobility is a huge problem coming that needs to be solved.
>>
>> Payment systems will be on phones. Payment systems will need to be 
>> secure. Payment identity will be hash of your public-key, your network 
>> address will need to be the same, that means you can change the IP 
>> address of a system when it roams ANYWHERE.
>>
>> *From: *Dino Farinacci <[email-masked] <mailto:[email-masked]>>
>> *Subject: **My comments to draft-hmm-dmm-5g-uplane-analysis-01 *
>> *Date: *August 30, 2018 at 4:17:09 PM PDT
>> *To: *Dan Voyer <[email-masked] <mailto:[email-masked]>>, 
>> Satoru Matsushima <[email-masked] 
>> <mailto:[email-masked]>>, 
>> [email-masked] 
>> <mailto:[email-masked]>
>> *Cc: *Shunsuke Homma <[email-masked] 
>> <mailto:[email-masked]>>, [email-masked] 
>> <mailto:[email-masked]>, Dino Farinacci 
>> <[email-masked] <mailto:[email-masked]>>
>>
>> In this section you should make it very clear where the tunnel 
>> terminates. It is not clear from the diagram on the N3 interface if it 
>> terminates on the eNodeB or the UE.
>>
>>
>> Dan made this point verbally to me about unidirectional versus 
>> bidirectional tunnels. I don’t know if it matters because it all comes 
>> down to the amount of state that a router uses to terminate the 
>> tunnel. In cisco routers, we also used one “virtual interface” (called 
>> an idb) to support a tunnel. And we used it and all the accounting 
>> associated with it for both input and output counters, addressing, 
>> encap/decap functions, etc.
>>
>> Dan made the comment that if tunnels were bidirectional, you reduce 
>> the tunnel state in half. That is only if an implementation used two 
>> data structues for different TEID values. But that would be a poor 
>> implementation where you could store a rec_teid and send_teid in one 
>> data structure not duplicating other parts of the data structure.
>>
>>>
>>
>> All tunneling protocols should “copy inner to outer” fields like TOS, 
>> flow label, and TTL to outer header. That needs to be done by default 
>> and then later if there is marking policy, it happens at boundaries 
>> only in the outer header.
>>
>> This won’t be an issue with IPv6/SRv6 but if one wants to preserve the 
>> original host values, and there is a rewrite at the boundaries, you 
>> will lose the information. People need to decide if this is a hard 
>> requirement.
>>
>>
>> It just doesn’t matter. Link layer CRCs are very good these days and 
>> does a good job protecting bit errors and bad actor corruption of 
>> packet data.
>>
>>
>> SRv6 will have this problem too. With just 1 SID can still put an 
>> user’s MTU sized packet over the effective path MTU.
>>
>>
>> If this is the case, then the ECMP algorithm in GTP is broken. No 
>> packets should be reordered. So why are they?
>>
>>
>> not setup, the encapsulator should have no state. And packets should 
>> be dropped locally and not go far to get an error back. This sounds 
>> architecturally broken.
>>
>>
>> routing protocol, is it like a management protocol. What are the 
>> failure modes, the state/bandwidth tradeoffs.
>>
>>
>>
>> You should state if a UP/CP pair that is self contained with a slice. 
>> Or is there a CP that operates each UP in each slice. And how much 
>> fault isolation and resource reservation is obtained by slicing. I 
>> hope its more like containers than VRFs.  ;-)
>>
>>
>> What I got from the CT4 meeting is that the LISP mapping system can be 
>> put in a box above and you could combine SMF, AMF, and PCF functions 
>> just in the mapping database data structure. I think more of those 
>> compoents could be combined, but I don’t understand what they all do 
>> yet. LOL.  ;-)
>>
>> Note a holistic architecture would have the N6 interface terminate on 
>> a top-of-rack router in the DN data-center.
>>
>>
>> So I think you need a CP equivalent to this document. Because the 
>> complexity, even though present here in the UP, is order of magnitude 
>> more in the CP and the costs of scale, security, and manageability is 
>> in the control-plane. Have you consider doing this?
>>
>> Like I mentiioned to Satoru in West Palm Beach, without a level of 
>> indirection in routing, how do you scale the mobility part with SRv6? 
>> That is if you carry host routes, that can only scale so far, and it 
>> forces you to scope the host routes to small domains. That means, your 
>> mobility is limited within that domain. That tradeoff needs to be 
>> characterized. And I think most importantly what users need and want 
>> is mobility across WiFi and 3GPP based data-links. That is a problem 
>> that really needs to be solved since the RAN RF frequencies are just 
>> going to get more jammed with billions of IoT devices coming online. 
>> That means smaller cells, that means IP address changes, so IP address 
>> mobility is a huge problem coming that needs to be solved.
>>
>> Payment systems will be on phones. Payment systems will need to be 
>> secure. Payment identity will be hash of your public-key, your network 
>> address will need to be the same, that means you can change the IP 
>> address of a system when it roams ANYWHERE.
>>

---

From: Alexandre Petrescu <[email-masked]>
Date: Fri, 7 Sep 2018 15:32:51 +0200

Le 05/09/2018 à 10:55, Sridhar Bhaskaran a écrit :
[...]
> is already there in TS 29.244. A reference to TS 29.244 should be 
> enough. See section 6.4 of TS 29.244 for reliable delivery of PFCP messages.

No.

Rather: show a packet dump of the TS.  It's useless to talk about Word 
documents that are not backed by implementations.

Alex

---

From: Alexandre Petrescu <[email-masked]>
Date: Fri, 7 Sep 2018 15:34:14 +0200

Le 06/09/2018 à 12:27, Sridhar Bhaskaran a écrit :
> user plane function. In EPC, the PFCP protocol is used on the Sx 
> interface. In 5G its used on the N4 interface. It is used to set up the 
> packet classifiers and forwarding rules in the user plane function.

It's the first time I hear about this.

Where was discussed the allocation of this port number?

Alex

---

From: Behcet Sarikaya <[email-masked]>
Date: Fri, 7 Sep 2018 09:45:43 -0500

> Where was discussed the allocation of this port number?
Individuals can ask IANA to assign port numbers for their applications, I
think. In this case Kimmo asked in the name of 3GPP CT4
in 2017 as the recording shows.
I don't think there is any associated draft as PFCP which is a UDP
application was defined by CT4 much like GTP-U.
If you remember Cameron used to say that are a lot of protocols used in
practice defined elsewhere not in IETF.

Behcet

---

From: Behcet Sarikaya <[email-masked]>
Date: Fri, 7 Sep 2018 10:00:18 -0500

Hi Marco,

> or to avoid superfluous packet classification if it has been done on the
 So it should be treat as a special case.
My suggestion is let's solve the main problem first, i.e. privacy issues,
make idloc deployable in large scale.

IMO, for non-encapsulation protocols, overloading of id-loc space seems
> interesting if the attribute
Sure.

Behcet

> >> > / QoS flows. DSCP in IPv4 and Flow Label in IPv6 is one option but
> these
> >> > are
> >> > not always unique. The same IP pools can be shared across multiple
> PDNs
> >> > /
> >> >> > >>> is not setup, the encapsulator should have no state. And
> packets
> >> >> > >>> should be
> >> >> > [Sridhar] The purpose of GTP-U error indication is to signal in
> band
> >> >> > to
> >> >> > the sender that a GTP-U tunnel endpoint (TEID) at the receiving
> side
> >> >> > is lost
> >> > e.g due to administrative reasons, gNB or eNB removes the radio
> bearers
> >> > and
> >> >> > >>> protocol, is it like a management protocol. What are the
> failure
> >> >> > >>> modes, the
> >> >> > enough.
> >> > in 22 series specs, system architecture in 23 series specs and
> protocol
> >> > in
> >> > design aspect in a protocol to a corresponding architectural
> requirement
> >> > in
> >> > For e.g as I explained above the reason for the existence of TEID -
> its
> >> > the

---

From: Arashmid Akhavain <[email-masked]>
Date: Fri, 7 Sep 2018 15:01:12 +0000

> From: Dino Farinacci [mailto:[email-masked]]
> Sent: 06 September 2018 18:59
> To: Arashmid Akhavain <[email-masked]>
> Cc: Tom Herbert <[email-masked]>; [email-masked];
> dmm <[email-masked]>
> > As it was explained by Sridhar,  each UE can have multiple contexts. For
> example, today some operators provide Data and VoLTE service to their
> customers. These two services are represented by separate GTP tunnels in
> the core with each tunnel tied up to a particular QoS.
> >
> There is no reason why it shouldn’t. And IPv6, for this use-case doesn’t add
> anything new other than a 28 bit traffic-class/flow-label that can provide
> more bits for “new functionality”.

[Arashmid]  And that's what I meant. Having a flow label is handy. We can perhaps use it to identify
different UE sessions.

> 
> > sessions/context/bearer. So, GTP and TEID did the job. But I agree with
> Dino that IPv6 is much more versatile and is definitely worth looking at as an
> alternative.
> 
> That is not what I said. I said “IP could have solved this problem”. And “IP”
> means either IPv4 or IPv6, or both at the same time.

[Arashmid] 
How would we employ IPv4 to distinguish between different UE sessions. TOS?
Or you mean using encapsulation?

> 
> > A factor worth considering though is that the use of GTP and TEID in mobile
> core allows operators to deal with QoS on their own terms. The tunnels with
> specific operator-controlled QoS are established by the control plane
> between eNB, SGW, and PGW. UEs or applications sitting in the UEs have no
> say in this. Well at least till the packet exits operator's network.
> 
> The problem with one header, is that if you re-mark (known as PHB markign
> in the ole days) you lose the original value. Encapsulation is useful here
> because you can map the inner to outer and anywhere along the path you
> can PHB remark on the outer header. And then the destination can see the
> orignal source’s ToS/QoS/TC/flow-label whatever.

[Arashmid] Yes, I agree. The original value is lost with PHB. Encapsulation certainly
makes things easier and the inner to outer mapping trick has been widely used
in IP and MPLS(multiple labels like service and transport)

> Not if you encapsulate. But note with SRv6, you can possibly retain the
> original flow-label if the SID can retain those bits before overwriting the
> destination address from the option’s value.

[Arashmid] Agree. Encapsulation does the trick again. That's why GTP has worked
well and served the purpos in the mobile back-haul so far.

> 
> Dino
> 
> >  down this path, operators need proof that they will be still in the driving
> seat and QoS cannot be dictated/tampered by the UE or any application
> running in it.
> >
> > Now, here is an interesting question for the operators. Would any operator
> be interested in allowing QoS  to be set by the UE or by applications running
> in the UE and charged for by the network? "Yes" could potentially imply
> impacts on the air interface, UE resource block allocation and can make
> scheduling on the RAN side much more complex.
> >
> > Arashmid
> >
> >
> >> 3GPP wants per bearer awareness so they need a specific ID, that
> >> could have been the UE’s IP address. And with IPv6 it can be unique
> >> and not the issue that Sridhar brought up.
> >>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 7 Sep 2018 08:12:39 -0700

> [Arashmid]  And that's what I meant. Having a flow label is handy. We can perhaps use it to identify
Careful if you use the flow label to identify flows. It should be
considered "soft identification" since it might not always be correct
(it can be changed en route, isn't protected by any checksum, anyone
can set it however they want, etc.). It's useful for things like ECMP
that don't require 100% accuracy in identifying flow. The flow label
was briefly considered for holding VNIs in network virtualization, but
we talked them out of that.

Tom

---

From: Arashmid Akhavain <[email-masked]>
Date: Fri, 7 Sep 2018 15:14:53 +0000

Hi Tom and Dino,

Yes, I agree that these are the type of services that operator might want to look into to extend their revenue. Tom put it nicely: "it allows operators the chance to offer and monetize services that can benefit users".

I doubt that operators would like to allow dynamic QoS control to shift to UEs. But I think a mechanism that would allow QoS request to sit on the UE 
and the control to be done at the operator side would work well. This is value that the slicing concept in 5G brings to the table. It allows the operators
to setup preconfigured slices with different QoS, delay, jitter, etc behaviour. UE flows then will be mapped to appropriate 5G slice.

The complexity of the scheduler on the RAN side is a different story. But certainly use of the slicing concept makes the scheduler job less complicated.

Arashmid

> Sent: 06 September 2018 19:01
> Cc: Arashmid Akhavain <[email-masked]>; ta-
> [email-masked]; dmm <[email-masked]>
> Would users like to set QoS bits that may charge them more for service?
> Could they set bits to get cheaper service?
> 
> >> IPv4 didn't fit the bill when GTP work was under way as it couldn't
> uniquely identify multiple UE sessions/context/bearer. So, GTP and TEID did
> the job. But I agree with Dino that IPv6 is much more versatile and is
> definitely worth looking at as an alternative.
> >>
> >> A factor worth considering though is that the use of GTP and TEID in
> mobile core allows operators to deal with QoS on their own terms. The
> tunnels with specific operator-controlled QoS are established by the control
> plane between eNB, SGW, and PGW. UEs or applications sitting in the UEs
> have no say in this. Well at least till the packet exits operator's network.
> >>
> >> Using the information in UE's IP packet header can jeopardise the above
> tight QoS control. I think going down this path, operators need proof that
> they will be still in the driving seat and QoS cannot be dictated/tampered by
> the UE or any application running in it.
> >>
> >> Now, here is an interesting question for the operators. Would any
> operator be interested in allowing QoS  to be set by the UE or by applications
> running in the UE and charged for by the network? "Yes" could potentially
> imply impacts on the air interface, UE resource block allocation and can make
> >>>> I was thinking if TEID is need then that can be encoded in a
> >>>> locator easily enough.
> >>>>
> >>>> Tom
> >>>

---

From: Arashmid Akhavain <[email-masked]>
Date: Fri, 7 Sep 2018 15:26:55 +0000

Correct, flow labels can change along the path. That's why I like the slicing concept.
UEs can request services with different attributes, operators control how service request are mapped into slices. I should look into the air side of the business and see what happens there.

> From: Tom Herbert [mailto:[email-masked]]
> Sent: 07 September 2018 11:13
> Cc: Dino Farinacci <[email-masked]>; [email-masked];
> dmm <[email-masked]>
> >> customers. These two services are represented by separate GTP tunnels
> >> in the core with each tunnel tied up to a particular QoS.
> >> >
> >> > uniquely identify multiple UE
> >>
> >> There is no reason why it shouldn’t. And IPv6, for this use-case
> >> doesn’t add anything new other than a 28 bit traffic-class/flow-label
> >> that can provide more bits for “new functionality”.
> >
> >
> > [Arashmid]  And that's what I meant. Having a flow label is handy. We
> > can perhaps use it to identify different UE sessions.
> >
> Careful if you use the flow label to identify flows. It should be considered
> "soft identification" since it might not always be correct (it can be changed en
> route, isn't protected by any checksum, anyone can set it however they want,
> etc.). It's useful for things like ECMP that don't require 100% accuracy in
> identifying flow. The flow label was briefly considered for holding VNIs in
> network virtualization, but we talked them out of that.
> 
> Tom
> 
> >>
> >> Dino that IPv6 is much more versatile and is definitely worth looking
> >> at as an alternative.
> >>
> >> core allows operators to deal with QoS on their own terms. The
> >> tunnels with specific operator-controlled QoS are established by the
> >> control plane between eNB, SGW, and PGW. UEs or applications sitting
> >> in the UEs have no say in this. Well at least till the packet exits operator's
> network.
> >>
> >> The problem with one header, is that if you re-mark (known as PHB
> >> markign in the ole days) you lose the original value. Encapsulation
> >> is useful here because you can map the inner to outer and anywhere
> >> along the path you can PHB remark on the outer header. And then the
> >> destination can see the orignal source’s ToS/QoS/TC/flow-label whatever.
> >
> >
> > [Arashmid] Yes, I agree. The original value is lost with PHB.
> > Encapsulation certainly makes things easier and the inner to outer
> > mapping trick has been widely used in IP and MPLS(multiple labels like
> > service and transport)
> >
> >>
> >> Not if you encapsulate. But note with SRv6, you can possibly retain
> >> the original flow-label if the SID can retain those bits before
> >> overwriting the destination address from the option’s value.
> >
> > [Arashmid] Agree. Encapsulation does the trick again. That's why GTP
> > has worked well and served the purpos in the mobile back-haul so far.
> >
> >>
> >> Dino
> >>
> >> > operator
> >> be interested in allowing QoS  to be set by the UE or by applications
> >> running in the UE and charged for by the network? "Yes" could
> >> potentially imply impacts on the air interface, UE resource block
> >> allocation and can make scheduling on the RAN side much more complex.
> >> >
> >> > Arashmid
> >> >
> >> >
> >> >> could have been the UE’s IP address. And with IPv6 it can be
> >> >> unique and not the issue that Sridhar brought up.
> >> >>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 7 Sep 2018 08:50:53 -0700

> UEs can request services with different attributes, operators control how service request are mapped into slices. I should look into the air side of the business and see what happens there.

Arashmid,

Yes, slices are a good model. The key question is how does a user's
packets get mapped to the right slice. Presumably, this is done at an
ingress point into the network. There are two ingress points to be
considered, one from UE into the network and one from Internet into
network (return path). If the UE sets bits in the packet to get
service in the forward path, we somehow need to have those bits
available on packets in the return path to map incoming packets. In
lieu or requiring the network to maintain a whole bunch of complex
flow state, FAST arranges that the remote server reflects the bits in
response packets.

Tom

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 7 Sep 2018 10:08:29 -0700

I think you’ll still have the PHB re-marking issues I mentioned in previous emails. The question is, should the network touch/change any header bits of the packet the source has built. The answer should probably be no.

Having said that, GTP did it the right way, even though it cost in header length.

Dino

> On Sep 7, 2018, at 8:26 AM, Arashmid Akhavain <[email-masked]> wrote:
>

---

From: Arashmid Akhavain <[email-masked]>
Date: Fri, 7 Sep 2018 17:45:02 +0000

I understand your point, but there is no guarantee for a precise QoS without using some sort of encapsulation being it GTP, RSVP, etc. Even with tunnels, there is no guarantee that all nodes along the path have the same hardware capability and can provide the same QoS treatment.  

For example, the code points in routers need to be configured to correctly handle the EXP bits in MPLS labels. But there is no guarantee that all routers can support all values. The EXP values get mapped to code points but the mapping is not always one to one.  3-bit EXPs can map to 4 code points on those routers with less capable H/W.

Slicing is almost the same. It allows user traffic to be mapped to what the operator provides.
I agree with you that network should not touch/change original header bits. GTP or any other encapsulation easily allow for this. The question is whether we can provide for this without using encapsulation. IPv6 might be the answer. But as Tom pointed out, flow labels can still change in the middle. Is there any room for improvement. SIDs might present an opportunity.

> Sent: 07 September 2018 13:08
> I think you’ll still have the PHB re-marking issues I mentioned in previous
> emails. The question is, should the network touch/change any header bits of
> the packet the source has built. The answer should probably be no.
> 
> > UEs can request services with different attributes, operators control how
> service request are mapped into slices. I should look into the air side of the
> business and see what happens there.
> >
> >
> >
> >> Cc: Dino Farinacci <[email-masked]>;
> >> [email-masked]; dmm <[email-masked]>
> >>>> example, today some operators provide Data and VoLTE service to
> >>>> their customers. These two services are represented by separate GTP
> >>>> tunnels in the core with each tunnel tied up to a particular QoS.
> >>>>>
> >>>> doesn’t add anything new other than a 28 bit
> >>>> traffic-class/flow-label that can provide more bits for “new
> functionality”.
> >>>
> >>>
> >>> [Arashmid]  And that's what I meant. Having a flow label is handy.
> >>> We can perhaps use it to identify different UE sessions.
> >>>
> >> was briefly considered for holding VNIs in network virtualization, but we
> talked them out of that.
> >>
> >> Tom
> >>
> >>>>
> >>>> Dino that IPv6 is much more versatile and is definitely worth
> >>>> looking at as an alternative.
> >>>>
> >>>> tunnels with specific operator-controlled QoS are established by
> >>>> the control plane between eNB, SGW, and PGW. UEs or applications
> >>>> sitting in the UEs have no say in this. Well at least till the
> >>>> packet exits operator's
> >> network.
> >>>>
> >>> mapping trick has been widely used in IP and MPLS(multiple labels
> >>> like service and transport)
> >>>
> >>>>
> >>>> seat and QoS cannot be dictated/tampered by the UE or any
> >>>> application running in it.
> >>>>>
> >>>> be interested in allowing QoS  to be set by the UE or by
> >>>> applications running in the UE and charged for by the network?
> >>>> "Yes" could potentially imply impacts on the air interface, UE
> >>>> resource block allocation and can make scheduling on the RAN side
> much more complex.
> >>>>>
> >>>>> Arashmid
> >>>>>
> >>>>>
> >>>>>> Subject: Re: [DMM] Comments to draft-hmm-dmm-5g-uplane-
> analysis-
> >> 01
> >>>>>>
> >>>>>>> Behcet,
> >>>>>>>

---

From: Arashmid Akhavain <[email-masked]>
Date: Fri, 7 Sep 2018 17:46:57 +0000

100% Agree with you Tom. And these are type of questions we need to answer when we talk about GTP replacement. The control plane in today's mobile core sets up the tunnel and ensure the uniformity of QoS in
both forward and backward path. 

Can you elaborate on using FAST?  FAST on UPFs?

Arashmid

> Sent: 07 September 2018 11:51
> Yes, slices are a good model. The key question is how does a user's packets
> get mapped to the right slice. Presumably, this is done at an ingress point
> into the network. There are two ingress points to be considered, one from
> UE into the network and one from Internet into network (return path). If the
> UE sets bits in the packet to get service in the forward path, we somehow
> need to have those bits available on packets in the return path to map
> incoming packets. In lieu or requiring the network to maintain a whole bunch
> of complex flow state, FAST arranges that the remote server reflects the bits
> in response packets.
> 
> Tom
> 
> >
> >
> >
> >> >> Cc: Tom Herbert <[email-masked]>;
> >> >> their customers. These two services are represented by separate
> >> >> GTP tunnels in the core with each tunnel tied up to a particular QoS.
> >> >> >
> >> > How would we employ IPv4 to distinguish between different UE
> sessions.
> >> TOS?
> >> >> markign in the ole days) you lose the original value.
> >> >> Encapsulation is useful here because you can map the inner to
> >> >> outer and anywhere along the path you can PHB remark on the outer
> >> >> header. And then the destination can see the orignal source’s
> ToS/QoS/TC/flow-label whatever.
> >> >
> >> >
> >> >> > Using the information in UE's IP packet header can jeopardise
> >> >> > the above tight QoS control. I think going
> >> >>
> >> >> Not if you encapsulate. But note with SRv6, you can possibly
> >> >> retain the original flow-label if the SID can retain those bits
> >> >> before overwriting the destination address from the option’s value.
> >> >
> >> > [Arashmid] Agree. Encapsulation does the trick again. That's why
> >> > GTP has worked well and served the purpos in the mobile back-haul so
> far.
> >> >
> >> >>
> >> >> Dino
> >> >>
> >> >> >> Subject: Re: [DMM] Comments to
> >> >> >> draft-hmm-dmm-5g-uplane-analysis-
> >> 01
> >> >> >>
> >> >> >>> Behcet,
> >> >> >>>
> >> >> >> 3GPP wants per bearer awareness so they need a specific ID,
> >> >> >> that could have been the UE’s IP address. And with IPv6 it can
> >> >> >> be unique and not the issue that Sridhar brought up.
> >> >> >>

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 7 Sep 2018 11:28:47 -0700

> I understand your point, but there is no guarantee for a precise QoS without using some sort of encapsulation being it GTP, RSVP, etc. Even with tunnels, there is no guarantee that all nodes along…

There is existing hardware where the encapsulator copies inner QoS to outer QoS. All routers along the path just process the outer QoS, no changes to or new processing requirements for them.

> For example, the code points in routers need to be configured to correctly handle the EXP bits in MPLS labels. But there is no guarantee that all routers can support all values. The EXP values get…

That is a completely different matter. The discussion is about remarking. And if one remarks to what the path cannot support, well things don’t work as expected.

> I agree with you that network should not touch/change original header bits. GTP or any other encapsulation easily allow for this. The question is whether we can provide for this without using encap…

Not if they are encapsulated and routers don’t touch packets inside.

Dino

---

From: Tom Herbert <[email-masked]>
Date: Fri, 7 Sep 2018 11:40:17 -0700

Hi Arashmid,

FAST should be orthogonal to the issue of GTP replacement and is
agnostic to underlying RAN protocols. It's implemented at the IP layer
and the network services it conveys can be supported by a network in
an arbitrary fashion.

> Can you elaborate on using FAST?  FAST on UPFs?

The basic idea of FAST is that hosts (UE) requests tickets from a
"ticket agent" in their local network. The request is an abstract
description of desired services (like low latency, low jitter for
1080p video chat). The ticket agent sends back a "ticket" that encodes
the services the network has granted. The ticket is opaque to the
application and anyone else outside of the network provider. The
application attaches the ticket to its packets (in IPv6 HBH options)
and sends them normally otherwise. At the first hop node in the
network, the ticket is decoded by the network. Appropriate services
are applied (e.g. set DSCP, set QoS, send into a tunnel or slice,
etc.). The packet makes it's way to the destination with the ticket
attached. The destination saves the ticket in it's context for
communication. When the remote server replies, it attaches saved
tickets as a "reflected tickets". When the return packet enters the
client's network of the client, a node decodes the ticket and maps it
to the right services for transit the provider networks back the UE.

There are lot of details: like how to keep overhead low, how to make
this secure ans stateless, how to limit scope of sensitive
information, how to deal with paths that block HBH options. Please
take a look at the FAST draft
https://tools.ietf.org/html/draft-herbert-fast and white paper
https://github.com/quantonium/papers/blob/master/FAST.pdf

Tom

---

From: Satoru Matsushima <[email-masked]>
Date: Sat, 8 Sep 2018 11:04:51 +0900

Thanks Sridhar for your followups.

> [SB] TS 29.244 is not a draft but rather a full fledged technical specification. The issue with repeating content from elsewhere instead of just pointing is the risk of providing inaccurate informa…

I don’t believe that 3GPP forbid anyone to publish any books, papers to describe what the 3GPP system and the protocols and how it works.

And it was one of our reasons to introduce the user plane analysis draft that introduces 3GPP specifications in familiar way for IETF people which could help to bridge two SDOs. 

Best regards,
--satoru

> 2018/09/06 19:24、Sridhar Bhaskaran <[email-masked]>のメール:
> 
> For e.g as I explained above the reason for the existence of TEID - its the one to one mapping of radio bearer to an EPS bearer / QOS flow that demanded it. 
> 
> 
>

---

From: Behcet Sarikaya <[email-masked]>
Date: Mon, 10 Sep 2018 10:21:03 -0500

Folks, let's not forget that most of these things you are talking about are
session layer issues.
According to OSI layering, the session layer is Layer 5, above Layer 4
which is transport layer.
3GPP is using UDP which is Layer 4 for this session layer coding. I think
probably it is a better layer than the Internet layer which would be the
case with IdLoc protocols, I think even with LISP which uses UDP
encapsulation.

Behcet

---

From: Shunsuke Homma <[email-masked]>
Date: Mon, 1 Oct 2018 12:50:30 +0900

Hi all,
# Sorry for my late response...

Thank you for your lively discussion. It is very helpful for 
understanding points which need supplemental explanation and more 
consideration.

Following the discussion, we're planning to update the I-D for covering 
the points below:

- termination points of GTP-U
   (RANs and UPFs terminate GTP-U in 5GS.)
- setting QoS parameter of outer IP header
   (Note that it's not just copy of inner to outer.)
- problems related to IP connectivity (e.g., MTU in IPv6 networks, IPv4 
address duplication)
- summary of network slicing in 5GS
   (E.g., "slice is composed of SMF and UPFs. AMF selects SMF depending 
on NSSAI sent from UE, and SMF indicates to the UE the UPF that it is 
allocated.")
- case studies on UPF selection
   (E.g., parameters used for deciding destination UPF)
# Optimizing forwarding paths solution might be realized with UPF 
selection mechanism in 3GPP architecture. (ID-LOC may be applied as such 
mechanism.)

If you have any request for us on this updating, please let us know.

Best regards,

Shunsuke

-- 
----------------------------------
Shunsuke Homma
<[email-masked]>
TEL: +81 422 59 3486
FAX: +81 422 60 7460

NTT Network Service Systems Labs.
Musashino city, Tokyo, Japan
----------------------------------