Subject: [Spud] Possible WG-forming follow-on to SPUD BoF

From: Brian Trammell <[email-masked]>
Date: Thu, 19 May 2016 17:04:50 +0200

Greetings, all,

We propose to hold a working-group forming BoF in Berlin as a follow-on to the SPUD work, to define a common substrate protocol for encrypted transports based on the requirements derived from experimentation with the SPUD prototype.

First, note that since the acronym "SPUD" refers primarily to the prototype itself, any follow-on working group should have a different name. We're using the derived requirements, not starting for the prototype. The name is a subject for discussion, and suggestions are welcome. To have something to put in the proposed charter, we'd propose "Path Layer UDP Substrate" (PLUS) as a starting point.

The proposed charter appears below. We're interested in hearing initial feedback on the proposed charter in preparation for a BoF proposal (the cutoff date is two weeks from tomorrow, on Friday 3 June). Is there work to do here within the IETF? Is the scope of the proposed charter appropriate? Is there energy to do this work?

Thanks, cheers,

Brian and Mirja

Path Layer UDP Substrate (PLUS)
===============================

The PLUS working group’s goal is to enable the deployment of new, encrypted
transport protocols, while providing a transport-independent method to signal
flow semantics under transport and application control.

The current Internet protocol stack has no layer for explicit signaling of
flow semantics and characteristics to network elements, nor an integrated
signaling mechanism from network elements to back to endpoints and
applications. This layer never evolved within the stack, because middleboxes
and other devices on path could simply inspect and modify headers and payload
of unencrypted traffic at every layer. This implicit use of information from
the transport and application layers is a key origin of the ossification that
makes it hard or impossible to deploy new protocols.

In order to support more ubiquitous deployment of encryption, explicit
signaling must be added to the stack, and it must be transport protocol
independent. While IP would seem to be the natural home for this facility,
both IPv4 and IPv6 options and extensions have deployment problems on their
own, which makes it hard to include any additional information in these
protocols.  Additionally, a feedback channel that provides information from
on-path devices back to endpoints and applications, e.g. for error handling,
is essential for the deployment and success of an explicit cooperation
approach.

The PLUS working group will specify a new protocol as a Path Layer User
Substrate (PLUS), to support experimental deployment of explicit cooperation
between endpoints and devices on path, with the following goals:

- enable ubiquitous deployment of encrypted higher layer protocols by providing exposure of similar semantics to existing protocols (e.g. TCP) to devices on path (e.g. NATs and firewalls)

- allow applications and transport protocols to explicitly provide limited information to devices on path

- allow devices on path to provide feedback and information about the path to sending endpoints, under sending endpoint control

- allow devices on path to provide information about the path to receiving endpoints, with feedback to the sending endpoint, under sending endpoint control

Note that this approach explicitly gives the control of information exposure
back the application and/or transport layer protocol on the end host. It is
the goal of PLUS to minimize the information exposed at the level of detail
that is useful for the network, while encrypting everything else. This is
important to avoid future implicit treatment and the resulting ossification,
as well as to leverage the principle of least exposure to minimize privacy
risks presented by explicit cooperation.

Given that the primary goal of PLUS is to enable the deployment of arbitrary,
fully encrypted transport protocols, we assume that the higher layer protocol
can provide an encryption context that can be used by PLUS to provide
authentication, integrity, and encryption where needed. The primary threat
model to defend against will be modification or deletion of exposed
information by middleboxes and other devices on path, by allowing a remote
endpoint to detect modifications.

The working group will start with an initial set of use cases (see
draft-kuehlewind-spud-use-cases) and requirements (see draft-trammell-spud-req),
taken from experience with the Substrate Protocol for User Datagrams prototype.
The working group's main output will be an experimental protocol specification,
together with an initial registry of types of information that can be exposed
using PLUS, clearly aligned to these use cases and requirements. The working
group will close if it is not able to come to consensus on a protocol design
to meet these requirements.

The working group will additionally aim to identify other working groups that
could or should address parts of these requirements within existing protocols,
e.g. by specifying new protocol extensions or as input for on-going
standardization work. It will aim to work with working groups defining
encryption protocols (e.g. TLS) which could be used for encryption of
transport protocols running over PLUS.

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Thu, 19 May 2016 17:12:03 +0000

Does the sentence ...

  The current Internet protocol stack has no layer for explicit signaling of flow semantics and characteristics to network elements ...
 
actually wants to say something like

  The current Internet protocol stack has no layer for explicit *in-band* signaling of flow semantics and characteristics ... ?

I may be wrong, but there is no shortage of IETF and non-IETF protocols that can signal flow semantics and characteristics out-of-band, either path-coupled or path-decoupled, e.g., in the control or management plane. As a result, there are protocol "layers" for that signaling, but they are not in-band. There may also be other ways to fix this problem (e.g., by adding "end-to-end" or "inter-domain" instead of "in-band").

Michael

-----Original Message-----
From: Stackevo-discuss [mailto:[email-masked]] On Behalf Of Brian Trammell
Sent: Thursday, May 19, 2016 5:07 PM
To: [email-masked]
Subject: [Stackevo-discuss] Fwd: Possible WG-forming follow-on to SPUD BoF

Greetings, all,

Forwarding this to the correct address for this mailing list (oops). Please discuss at [email-masked].

Thanks, cheers,

Brian

> Begin forwarded message:
> 
> From: Brian Trammell <[email-masked]>
> Subject: Possible WG-forming follow-on to SPUD BoF
> Date: 19 May 2016 at 17:04:50 GMT+2
> To: [email-masked]
> Cc: [email-masked], [email-masked], "Mirja Kuehlewind 
> (IETF)" <[email-masked]>, Spencer Dawkins at IETF 
> <[email-masked]>
> 
> Greetings, all,
> 
> The PLUS working group’s goal is to enable the deployment of new, 
> encrypted transport protocols, while providing a transport-independent 
> method to signal flow semantics under transport and application control.
> 
> The current Internet protocol stack has no layer for explicit 
> signaling of flow semantics and characteristics to network elements, 
> nor an integrated signaling mechanism from network elements to back to 
> endpoints and applications. This layer never evolved within the stack, 
> because middleboxes and other devices on path could simply inspect and 
> modify headers and payload of unencrypted traffic at every layer. This 
> implicit use of information from the transport and application layers 
> is a key origin of the ossification that makes it hard or impossible to deploy new protocols.
> 
> signaling must be added to the stack, and it must be transport 
> protocol independent. While IP would seem to be the natural home for 
> this facility, both IPv4 and IPv6 options and extensions have 
> deployment problems on their own, which makes it hard to include any 
> additional information in these protocols.  Additionally, a feedback 
> channel that provides information from on-path devices back to 
> endpoints and applications, e.g. for error handling, is essential for 
> the deployment and success of an explicit cooperation approach.
> 
> The PLUS working group will specify a new protocol as a Path Layer 
> User Substrate (PLUS), to support experimental deployment of explicit 
> cooperation between endpoints and devices on path, with the following goals:
> 
> - enable ubiquitous deployment of encrypted higher layer protocols by 
> providing exposure of similar semantics to existing protocols (e.g. 
> TCP) to devices on path (e.g. NATs and firewalls)
> 
> - allow devices on path to provide feedback and information about the 
> path to sending endpoints, under sending endpoint control
> 
> - allow devices on path to provide information about the path to 
> receiving endpoints, with feedback to the sending endpoint, under 
> sending endpoint control
> 
> Note that this approach explicitly gives the control of information 
> exposure back the application and/or transport layer protocol on the 
> end host. It is the goal of PLUS to minimize the information exposed 
> at the level of detail that is useful for the network, while 
> encrypting everything else. This is important to avoid future implicit 
> treatment and the resulting ossification, as well as to leverage the 
> principle of least exposure to minimize privacy risks presented by explicit cooperation.
> 
> Given that the primary goal of PLUS is to enable the deployment of 
> arbitrary, fully encrypted transport protocols, we assume that the 
> higher layer protocol can provide an encryption context that can be 
> used by PLUS to provide authentication, integrity, and encryption 
> where needed. The primary threat model to defend against will be 
> modification or deletion of exposed information by middleboxes and 
> other devices on path, by allowing a remote endpoint to detect modifications.
> 
> draft-kuehlewind-spud-use-cases) and requirements (see 
> draft-trammell-spud-req), taken from experience with the Substrate Protocol for User Datagrams prototype.
> The working group's main output will be an experimental protocol 
> specification, together with an initial registry of types of 
> information that can be exposed using PLUS, clearly aligned to these 
> use cases and requirements. The working group will close if it is not 
> able to come to consensus on a protocol design to meet these requirements.
> 
> The working group will additionally aim to identify other working 
> groups that could or should address parts of these requirements within 
> existing protocols, e.g. by specifying new protocol extensions or as 
> input for on-going standardization work. It will aim to work with 
> working groups defining encryption protocols (e.g. TLS) which could be 
> used for encryption of transport protocols running over PLUS.
>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 19 May 2016 10:50:21 -0700

> I may be wrong, but there is no shortage of IETF and non-IETF protocols that can signal flow semantics and characteristics out-of-band, either path-coupled or path-decoupled, e.g., in the control o…
Not to mention that TOS and diff-serv are in-band signaling mechanisms
to the network, ECN is explicitly signaling from the network. Given
that they've been around for many years but have never been widely
deployed on the Internet makes me wonder if any in-band signaling
mechanism will ever catch on. The use of out of band signaling should
be considered, especially considered that in many cases like NAT and
firewall the network elements that need signaling are likely in local
networks of one side of the connection.

I have a similar concern about the statement that IP options/EH and
extension headers are being left off the table as part of a solution.
The fact that neither have these caught on in the Internet and that
tells me two things 1) No IP option is needed for the Internet to
operate and be successful which makes me wonder if signaling between
network and hosts is even required 2) If the extensibility mechanisms
of IP are chronically not deployable, I wonder why any alternative
that does not yet exist but intends to solve the same problem would be
any more deployable.

Tom

> _______________________________________________
> Spud mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spud

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 19 May 2016 11:27:01 -0700

Tom, Michael

I am reading your positions and i am confused:

DSCP is very successfull in enterprise networks. SPUD does not have to
be successfull "on the Internet". To me it would already be a great success if it would
help to improve controlled network services (like enterprise or 4G/5G networks). And
their transit traffic across the Internet (without changing anything on the Internet
itself).

AFAIK, TOS/DSCP/ECN are not per-flow, but per-packet mechanisms. Even if we often think about
them as per-flow and ould maybe like them to be restrict to per-flow. At least i don't know
any normative reference that defines them to be per-flow.

Wrt to out-of-band/out-of-path: The whole reason for SPUD is the analysis that all those
mechanisms fail to provide an easily deplopyale framework in the face of middleboxes. Aka: their
presence and mostly non-success should be an argument FOR working on spud. Not against it.

If a NAT/FW does support any flow signaling, its most likely inband. Every NAT/firwall
supports flow recognition via the explicit signaling of flow-based transport protoocols
like TCP. They do try to do the same for UDP and because that has no explicit flow signaling,
thats where the problem starts. NAT/FW do deep packet inspection which has all these problems
we know. SPUD sets out to solve/reduce these NAT/FW issues.

Non-support of anything like IP options is IMHO primarily driven by the absence of
unform cross-OS/Dev-language APIs for them. Yes. It may suck architecturally by designing
protocol stacks based on reality like: applications will only be able to use UDP and TCP,
so lets do new work on top of them, but to me the most important goal is to find the best
deployable solution to a problem. Not the architecturally cleanest one. If you give me
research money to work outside of reality i will happily take it though and represent a
different perspective.

Please, i do not want to dismiss your opposition, i just don't understand how the
arguments you make are working out. Maybe you can explain to me where i am going wrong
in my interpretations above.

Cheers
   Toerless

On Thu, May 19, 2016 at 10:50:21AM -0700, Tom Herbert wrote:
> >> The PLUS working group???s goal is to enable the deployment of new,

-- 
---
Toerless Eckert, [email-masked]

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Thu, 19 May 2016 18:56:34 +0000

My comment is *only* that the sentence "The current Internet protocol stack has no layer for explicit signaling of flow semantics and characteristics to network elements" needs to be improved, in my personal point of view.
 
For instance, for NAT/FW control, there are many existing out-of-band protocols such as PCP, MEGACO/H.248, NSIS, MIDCOM (and SIMCO), COPS, etc., not to mention all the SDN protocols, which are typically all flow-based. They are all out-of-band but e.g. NSIS would have been end-to-end. The intention and deployment model of these protocols may be very different to what SPUD aims for, and none of them is deployed end-to-end in the Internet. But I think this sentence has to be slightly more specific how to differentiate e.g. from control plane solutions.

Michael

-----Original Message-----
From: Toerless Eckert [mailto:[email-masked]] 
Sent: Thursday, May 19, 2016 8:27 PM
To: Tom Herbert
Cc: Scharf, Michael (Nokia - DE); Brian Trammell; [email-masked]
Subject: Re: [Spud] [Stackevo-discuss] Fwd: Possible WG-forming follow-on to SPUD BoF

Tom, Michael

DSCP is very successfull in enterprise networks. SPUD does not have to be successfull "on the Internet". To me it would already be a great success if it would help to improve controlled network services (like enterprise or 4G/5G networks). And their transit traffic across the Internet (without changing anything on the Internet itself).

AFAIK, TOS/DSCP/ECN are not per-flow, but per-packet mechanisms. Even if we often think about them as per-flow and ould maybe like them to be restrict to per-flow. At least i don't know any normative reference that defines them to be per-flow.

Wrt to out-of-band/out-of-path: The whole reason for SPUD is the analysis that all those mechanisms fail to provide an easily deplopyale framework in the face of middleboxes. Aka: their presence and mostly non-success should be an argument FOR working on spud. Not against it.

If a NAT/FW does support any flow signaling, its most likely inband. Every NAT/firwall supports flow recognition via the explicit signaling of flow-based transport protoocols like TCP. They do try to do the same for UDP and because that has no explicit flow signaling, thats where the problem starts. NAT/FW do deep packet inspection which has all these problems we know. SPUD sets out to solve/reduce these NAT/FW issues.

Non-support of anything like IP options is IMHO primarily driven by the absence of unform cross-OS/Dev-language APIs for them. Yes. It may suck architecturally by designing protocol stacks based on reality like: applications will only be able to use UDP and TCP, so lets do new work on top of them, but to me the most important goal is to find the best deployable solution to a problem. Not the architecturally cleanest one. If you give me research money to work outside of reality i will happily take it though and represent a different perspective.

Please, i do not want to dismiss your opposition, i just don't understand how the arguments you make are working out. Maybe you can explain to me where i am going wrong in my interpretations above.

Cheers
   Toerless

On Thu, May 19, 2016 at 10:50:21AM -0700, Tom Herbert wrote:
> >> encrypted transport protocols, while providing a 
> >> transport-independent method to signal flow semantics under transport and application control.
> >>
> >> signaling of flow semantics and characteristics to network 
> >> elements, nor an integrated signaling mechanism from network 
> >> elements to back to endpoints and applications. This layer never 
> >> evolved within the stack, because middleboxes and other devices on 
> >> path could simply inspect and modify headers and payload of 
> >> unencrypted traffic at every layer. This implicit use of 
> >> information from the transport and application layers is a key origin of the ossification that makes it hard or impossible to deploy new protocols.
> >>
> >> In order to support more ubiquitous deployment of encryption, 
> >> explicit signaling must be added to the stack, and it must be 
> >> transport protocol independent. While IP would seem to be the 
> >> natural home for this facility, both IPv4 and IPv6 options and 
> >> extensions have deployment problems on their own, which makes it 
> >> hard to include any additional information in these protocols.  
> >> Additionally, a feedback channel that provides information from 
> >> on-path devices back to endpoints and applications, e.g. for error 
> >> handling, is essential for the deployment and success of an explicit cooperation approach.
> >>
> >> User Substrate (PLUS), to support experimental deployment of 
> >> explicit cooperation between endpoints and devices on path, with the following goals:
> >>
> >> - enable ubiquitous deployment of encrypted higher layer protocols 
> >> by providing exposure of similar semantics to existing protocols (e.g.
> >> - allow devices on path to provide feedback and information about 
> >> the path to sending endpoints, under sending endpoint control
> >>
> >> exposure back the application and/or transport layer protocol on 
> >> the end host. It is the goal of PLUS to minimize the information 
> >> exposed at the level of detail that is useful for the network, 
> >> while encrypting everything else. This is important to avoid future 
> >> implicit treatment and the resulting ossification, as well as to 
> >> leverage the principle of least exposure to minimize privacy risks presented by explicit cooperation.
> >>
> >> information that can be exposed using PLUS, clearly aligned to 
> >> these use cases and requirements. The working group will close if 
> >> it is not able to come to consensus on a protocol design to meet these requirements.
> >>
> >> groups that could or should address parts of these requirements 
> >> within existing protocols, e.g. by specifying new protocol 
> >> extensions or as input for on-going standardization work. It will 
> >> aim to work with working groups defining encryption protocols (e.g. 
> >> TLS) which could be used for encryption of transport protocols running over PLUS.
> >>
> >

---

From: Tom Herbert <[email-masked]>
Date: Thu, 19 May 2016 12:17:30 -0700

> their transit traffic across the Internet (without changing anything on the Internet
Yes, that is my point about localizing the problem. A client device
should know what network it's on an be able to signal to that network
to get the services it wants. A server on the other side of the planet
that the node is talking to really shouldn't be concerned with this
and not required to implement some complex protocol to accommodate it.

> we know. SPUD sets out to solve/reduce these NAT/FW issues.
There seems to be a lot of UDP going through NAT/FW today without any
sort of in-band signaling, and protocols like QUIC are already seeing
deployment without that (according to their slide deck "About half of
Google to Chrome"). Considering that there is no deployment of a SPUD
aware firewall and we can't want to wait for every device on the
Internet to upgrade, SPUD will have to work with NAT/FW as is or we
couldn't consider it. TBH I'm really having a hard time seeing that
any extra signaling is required in the Internet other than what is in
the IP header, at best it seems like an optimization for certain
networks or configurations.

> research money to work outside of reality i will happily take it though and represent a

There has been a lot of discussion about this in v6ops, many networks
are simply dropping packets with extension headers. Please look at
draft-ietf-v6ops-ipv6-ehs-in-real-world-02 and
draft-gont-v6ops-ipv6-ehs-packet-drops-03. Because of this and the
fact that there hasn't been any interesting options defined worth
sending to date means that we (speaking for applications) have no use
to send them (btw all the major OSes support options AFAIK). If some
interesting options did come about, then we would probably apply a
happy eyeballs approach to deploy.

With something like SPUD (basically anything not simple TCP/IPv4) we
are always faced with the same problem that some networks on the
Internet may arbitrarily drop or otherwise restrict packets (we know
for instance that some network operators choose to severely rate limit
all UDP because of DOS concerns).  So even if we do manage to start
deploying transport protocols over UDP we are going have TCP as a
fallback for indefinite future, i.e. more happy eyeballs.

Tom

---

From: Brian Trammell <[email-masked]>
Date: Fri, 20 May 2016 10:46:10 +0200

hi Michael,

> On 19 May 2016, at 19:12, Scharf, Michael (Nokia - DE) <[email-masked]> wrote:
> 
> Does the sentence ...
> 

Yes. One could make the argument that DSCP is an in-band signaling mechanism at the network layer, but as designed really does work at the network layer (i.e., it is explicitly applied per hop).

> I may be wrong, but there is no shortage of IETF and non-IETF protocols that can signal flow semantics and characteristics out-of-band, either path-coupled or path-decoupled, e.g., in the control o…

"Interdomain, in-band" would work for me here and be more accurate. "End-to-end" would imply to my ears that we only want the endpoints to participate, for which we don't need a layer -- that's transport's job.

Note that the current requirements document is not explicitly limited to in-band approaches, but the fact that the signaling has to get the same NAT and forwarding treatment would seem to be an implicit limit.

Cheers,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Fri, 20 May 2016 12:08:04 +0200

hi Tom,

> On 19 May 2016, at 19:50, Tom Herbert <[email-masked]> wrote:
> 
> to the network, ECN is explicitly signaling from the network.

DSCP is properly network-layer. If you believe in a "path layer" (which I'm pretty sure I do), you can argue that ECN is the first path layer protocol.

> Given
> deployed on the Internet

DCSP was defined in such a way that it makes little sense in an interdomain context, and its widespread intradomain deployment means that it will never deploy across domains in the general case.

I still hold out a (potentially irrational) hope for ECN's deployment.

> makes me wonder if any in-band signaling

As noted, out of band signaling is not explicitly out of scope in the current

> extension headers are being left off the table as part of a solution.

I don't think they are off the table; a protocol such as what we propose to develop in PLUS could certainly use IPv6 extension headers, provided they can be implemented such that applications actually get access to those headers. IPv4 options are more of a stretch, simply due to questions about how well the path treats them.

> that does not yet exist but intends to solve the same problem would be

I'd assert that encapsulating things in UDP is certainly much more deployable *at the endpoints*. The open question is about adoption by on-path devices.

Cheers,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Fri, 20 May 2016 12:32:55 +0200

> On 19 May 2016, at 21:17, Tom Herbert <[email-masked]> wrote:
> 

There's a couple of problems here which could (IMO should) be handled better in a transport protocol independent way:

(1) Generic indication of flow start/end. Firewalls use TCP semantics, for the most part, and new protocols should expose at least these to get better treatment. Yes, this would have most of the same problems than SYNs, RSTs, and FINs do in TCP right now, but we can do better than the status quo. One of the things we looked at with the SPUD header is using a tube ID in a sparse identifier space to at least provide better protection against RST/FIN with guessed sequence numbers (which outside BGP is still a threat to deployed TCP, see [1]).

(2) Indication of NAT/FW state timeout and timeout expectation, to minimize unproductive keepalive traffic. Even with generic flow state indication you still need timers, and always will, but exposing information makes the average case performance better.

We could consider adding these as key use cases in the protocol, but I still think we want to have the broader use case discussion within any eventual PLUS WG.

If we *don't* care about the general deployability of new transport protocols, if we think that QUIC is the Last Transport We'll Ever Need (i.e., what SCTP was supposed to be), then we can just make sure QUIC exposes enough to make this possible, and go home. While QUIC is a definite improvement for the application semantics it's meant to support, and a widely deployable SCTP would be a improvement for other media applications, I'm not convinced that The Last Transport We'll Ever Need is a thing that exists, or that we should want to exist.

> happy eyeballs approach to deploy.

There's an additional problem in that the way v6 EH are defined makes it difficult to handle them on path: building silicon to handle arbitrary header chains is expensive. So one of the functional requirements for PLUS should be "information that forwarding devices need to see needs to (usually) to be not very far into the packet, at a constate offset if possible; see [2].

> fallback for indefinite future, i.e. more happy eyeballs.

Yes, and if we do manage to start deploying transport protocols over IPv6, we are going to have IPv4 fallback for an indefinite future. I share your concern about happy eyeballs state explosion (a phenomenon we refer to around the office as "angry eyeballs" -> eight orthogonal network/transport layer choices to test at startup means that every flow begins with a 256 packet burst)... but the benefit of doing this to transport flexibility is potentially worth the cost, which is the fundamental question we want to address in PLUS.

Cheers,

Brian

[1] Luckie et al "Resilience of Deployed TCP to Blind Attacks", IMC 2015, http://conferences.sigcomm.org/imc/2015/papers/p13.pdf

[2] Scudder "How a router actually works" IEPG at IETF 94, http://www.iepg.org/2015-11-01-ietf94/IEPG-RouterArchitecture-jgs.pdf

---

From: Brian Trammell <[email-masked]>
Date: Fri, 20 May 2016 13:02:00 +0200

Hi, all,

I've heard one concrete charter suggestion so far; the edited draft charter (changed paragraph 2) is below. If others would like to see points clarified to answer their questions, please send suggested edits. :)

I take the fact that we're diving into technical discussion as an indication that those participating are at least interested in the BoF, if not (yet ;) ) convinced of the need for a WG at this space?

Thanks, cheers,

Brian

The PLUS working group's goal is to enable the deployment of new,
encrypted transport protocols, while providing a
transport-independent method to signal flow semantics under
transport and application control.

The current Internet protocol stack has no layer for explicit,
in-band signaling of flow semantics and characteristics to network
elements in an interdomain context, nor an integrated signaling
mechanism from network elements to back to endpoints and applications.
This layer never evolved within the stack, because middleboxes and
other devices on path could simply inspect and modify headers and
payload of unencrypted traffic at every layer. This implicit use of
information from the transport and application layers is a
key cause of the ossification that makes it hard or impossible
to deploy new protocols.

handling, is essential for the deployment and success of an explicit
cooperation approach.

- allow applications and transport protocols to explicitly provide
limited information to devices on path

Note that this approach explicitly gives the control of information
exposure back the application and/or transport layer protocol on
the end host. It is the goal of PLUS to minimize the information
exposed at the level of detail that is useful for the network,
while encrypting everything else. This is important to avoid future
implicit treatment and the resulting ossification, as well as to
leverage the principle of least exposure to minimize privacy risks
presented by explicit cooperation.

The working group will start with an initial set of use cases (see
draft-kuehlewind-spud-use-cases) and requirements (see
draft-trammell-spud-req), taken from experience with the Substrate
Protocol for User Datagrams (SPUD) prototype.
The working group's main output will be an experimental protocol
specification, together with an initial registry of types of
information that can be exposed using PLUS, clearly aligned to
these use cases and requirements. The working group will close if
it is not able to come to consensus on a protocol design to meet
these requirements.

TLS) which could be used for encryption of transport protocols
running over PLUS.

> On 19 May 2016, at 20:56, Scharf, Michael (Nokia - DE) <[email-masked]> wrote:
>

---

From: Dave Dolson <[email-masked]>
Date: Fri, 20 May 2016 11:23:46 +0000

Brian,
I see no explicit mention of congestion avoidance or requirements on the sender to back off under congestion.

Arguably it is implicit; does it deserve mention in the charter?

(one of the hazards of placing control in the application is having varying ideas about how to do this, which won't compete fairly with each other. Figuring out how to deal with cheaters is another thing...)

-Dave

  Original Message
From: Brian Trammell
Sent: Friday, May 20, 2016 7:02 AM
To: [email-masked]
Subject: Re: [Spud] [Stackevo-discuss] Fwd: Possible WG-forming follow-on to SPUD BoF

Hi, all,

Thanks, cheers,

Brian

---

From: Ian Swett <[email-masked]>
Date: Fri, 20 May 2016 08:18:47 -0400

On Fri, May 20, 2016 at 6:32 AM, Brian Trammell <[email-masked]> wrote:

> There's a couple of problems here which could (IMO should) be handled
> better in a transport protocol independent way:
>
> (1) Generic indication of flow start/end. Firewalls use TCP semantics, for
> the most part, and new protocols should expose at least these to get better
> treatment. Yes, this would have most of the same problems than SYNs, RSTs,
> and FINs do in TCP right now, but we can do better than the status quo. One
> of the things we looked at with the SPUD header is using a tube ID in a
> sparse identifier space to at least provide better protection against
> RST/FIN with guessed sequence numbers (which outside BGP is still a threat
> to deployed TCP, see [1]).
>
> (2) Indication of NAT/FW state timeout and timeout expectation, to
> minimize unproductive keepalive traffic. Even with generic flow state
> indication you still need timers, and always will, but exposing information
> makes the average case performance better.
>
> We could consider adding these as key use cases in the protocol, but I
> still think we want to have the broader use case discussion within any
> eventual PLUS WG.
>
> If we *don't* care about the general deployability of new transport
> protocols, if we think that QUIC is the Last Transport We'll Ever Need
> (i.e., what SCTP was supposed to be), then we can just make sure QUIC
> exposes enough to make this possible, and go home. While QUIC is a definite
> improvement for the application semantics it's meant to support, and a
> widely deployable SCTP would be a improvement for other media applications,
> I'm not convinced that The Last Transport We'll Ever Need is a thing that
> exists, or that we should want to exist.
>

I think QUIC is great, but it's certainly not "The Last Transport We'll
Ever Need.".  I don't think such a thing exists.

> There's an additional problem in that the way v6 EH are defined makes it
> difficult to handle them on path: building silicon to handle arbitrary
> header chains is expensive. So one of the functional requirements for PLUS
> should be "information that forwarding devices need to see needs to
> (usually) to be not very far into the packet, at a constate offset if
> possible; see [2].
>
> Yes, and if we do manage to start deploying transport protocols over IPv6,
> we are going to have IPv4 fallback for an indefinite future. I share your
> concern about happy eyeballs state explosion (a phenomenon we refer to
> around the office as "angry eyeballs" -> eight orthogonal network/transport
> layer choices to test at startup means that every flow begins with a 256
> packet burst)... but the benefit of doing this to transport flexibility is
> potentially worth the cost, which is the fundamental question we want to
> address in PLUS.
>
> Cheers,
>
> Brian
>
> [1] Luckie et al "Resilience of Deployed TCP to Blind Attacks", IMC 2015,
> http://conferences.sigcomm.org/imc/2015/papers/p13.pdf
>
> [2] Scudder "How a router actually works" IEPG at IETF 94,
> http://www.iepg.org/2015-11-01-ietf94/IEPG-RouterArchitecture-jgs.pdf
>
>

---

From: Brian Trammell <[email-masked]>
Date: Fri, 20 May 2016 16:06:51 +0200

hi Ian,

> I think QUIC is great, but it's certainly not "The Last Transport We'll Ever Need.".  I don't think such a thing exists.

To be clear, I agree with every part of this statement. :)

There is an alternate approach that might be worth discussion here: Instead of defining an encapsulation layer for handling these two core use cases, we can define a set of behaviors along with expected bits and offsets in headers under the UDP header that can be used for state and (maybe) timer management purposes. If a QUIC header and a PLUS header look the same to the path for these purposes, then we can still solve this problem once for both.

Of course a set of behaviors and bits at given offsets is a "header", so the distinction is probably academic.

Cheers,

Brian

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 20 May 2016 16:25:59 +0200

Hi Dave,

PLUS is a new path (shim) layer mainly focused on signaling. However, 
congestion control/avoidance is clearly a function of the transport layer 
itself and has to stay there. So I would say it's actually not relevant for 
PLUS except, you'd like to integrate any additional signaling mechanisms 
here. Some use cases listed in draft-kuehlewind-spud-use-cases are kind of 
related to congestion avoidance but I don't think that this must be mentioned 
separately in the charter (as it is just one function of many in the 
transport layer).

If your goal is to make it easier to use the same congestion control in 
different transports, that's another topic and more related to the taps 
working group than PLUS/SPUD but I believe also not in scope for them (yet).

Mirja

---

From: Ian Swett <[email-masked]>
Date: Fri, 20 May 2016 10:33:46 -0400

You were clear, I just wanted to voice my support on that point.

As you imply, there are a variety of ways to define a 'header', and as long
as the peers and the on-path elements can interpret it correctly, it
accomplishes the desired goal.  That could be a PLUS header fully
encapsulating another protocol, or as you mention, it could be defining a
way to integrate PLUS into new and existing UDP protocols.

On Fri, May 20, 2016 at 10:06 AM, Brian Trammell <[email-masked]> wrote:

> >> (1) Generic indication of flow start/end. Firewalls use TCP semantics,
> for the most part, and new protocols should expose at least these to get
> better treatment. Yes, this would have most of the same problems than SYNs,
> RSTs, and FINs do in TCP right now, but we can do better than the status
> quo. One of the things we looked at with the SPUD header is using a tube ID
> in a sparse identifier space to at least provide better protection against
> There is an alternate approach that might be worth discussion here:
> Instead of defining an encapsulation layer for handling these two core use
> cases, we can define a set of behaviors along with expected bits and
> offsets in headers under the UDP header that can be used for state and
> (maybe) timer management purposes. If a QUIC header and a PLUS header look
> the same to the path for these purposes, then we can still solve this
> problem once for both.
>

---

From: Dave Dolson <[email-masked]>
Date: Fri, 20 May 2016 15:02:22 +0000

Mirja,

Thanks for clarifying.

Now that I read the proposed charter again, it is obvious that PLUS it not 
itself a transport protocol.

Now I understand that the congestion control is the responsibility of
protocol riding on top of PLUS.  E.g., in "TCP-like-transport over PLUS" protocol.

-Dave

-----Original Message-----
From: Mirja Kühlewind [mailto:[email-masked]] 
Sent: Friday, May 20, 2016 10:26 AM
To: Dave Dolson; Brian Trammell; [email-masked]
Subject: Re: [Spud] [Stackevo-discuss] Fwd: Possible WG-forming follow-on to SPUD BoF

Hi Dave,

Mirja

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 20 May 2016 08:09:36 -0700

> Path Layer UDP Substrate (PLUS)

Don't like don't understand this paragraph.
- What if we do not develope new transprt protocols but recipes
  how to modify existing transport protocols to run in PLUS ?
- "We are going to cook stew while juggling 4 balls". The relationship
  between the two is not really clear from the sentence. 
- What does "under transport and application control" mean ?
- Does not cover the key aspect of being able to implement these protocols
  as part of application code.

The goal of the PLUS working group is to overcome ossification of
deployments of new transport layer work. PLUS will develop a new
transport layer approach with the following properties:

- The transport protocol is encrypted.
- A transport protocol independent inband layer is used for signaling of
  flow semantics.
- The transport layer can be implemented in application layer code.

> key cause of the ossification that makes it hard or impossible

First paragraph wrong if you take STUN/ICE/MALICE into account.
Second paragraph unfair because it's hitting only one side. Give
the IETF some friendly slapping too, please.

Proposed text:

Traditionally, the Internet protocols did not provide explicit,
transport protocol independent signaling of flow characteristics to
on-path network devices. In return, the industry developed the practice
of inspecting protocol headers all the way from the transport layer
ports to application payload. This practice is called DPI (Deep Packet
Inspection. Later on, middleboxes started to filter or modify deep
into packets as well, for example to fix up IP addresses in application
layer protocols in NAT middleboxes. This was all possible because
historically all these layers where (and partially are) unencrypted. 

Later IETF standards track onpath signaling work such as RSVP or NSIS could not
overcome this practice because it could not not solely be implemented by
applications interested in it because these protocols had to be implemented by
OS vendors. Investmenting in these protocols was unattractive too, because
by default middleboxes that do not support these protocols would simply
drop them: The practice of unspecified, vendor specific DPI had lead to a
"security" model in middleboxes where it was impossible to deploy new
transport protocols that could be recognized and supported by pre-existing
middleboxes without new work on these middleboxes. This lead to ossification
in deployment of new transport work. 

In recent years, STUN/ICE and extensions to it started to introduce
the ability for onpath inband explicit signaling to middleboxes, but
being limited to be used purely with existing UDP based transport options,
they do not address how to solve the problem for applications using
non-UDP transport.

> transport protocol independent.

> While IP would seem to be the

Better:

The natural layer for flow information is between Internet and Transport
protocols - IP/IPv6 options/extensions or a common transport sublayer.
Implementation in the IP/IPv6 layer makes it difficult o adopt though
because it would not allow for implementation in application code.

> - enable ubiquitous deployment of encrypted higher layer protocols
                                            ^^^ transport and
> receiving endpoints, with feedback to the sending endpoint, under

So.. The biggest discussion point i guess will be around the "limited",
where the privacy advocacy segment of our population will argue
"nothing", and the evil-network-doer will argue "everything".

What i really want to see is this:

- Allow the application to learn from the network devices what information
  it must present in the clear to have its traffic be passed through
  and/or receive additional treatment by the network.

Aka: "Security guard: Sure, you can go through this door if you show me your <credential> else not"

Then the app can decide if it wants to show <credential>. Otherwise the
apps have to operate like corporate drones, carrying around their corp-ID
in the open all the time hoping that thats the right credential.

Or to rephrase: if we do not even attempt to overcome TCP port numbers as
credentials then we're lame and will not be successful.

> the end host.

remove "and/or transport layer protocols". We MUST have auditability
of what is being sent in the clear by the application. We can not
have transport protocols inserting information in the clear without
the app knowing.

> It is the goal of PLUS to minimize the information

How about a next paragraph about this:

Text:
In cases where middleboxes require auditability/unencrypted access to
the encrypted information of a flow, PLUS can still be used, but the
encryption keys need to be shared between application and middleboxes
(or backend thereof). PLUS itself will not consider any mechanisms
for such encryption key sharing, but instead expects this to be done
by a higher layer in the application.
End of text.

I would like to see something to this extend, because to me there
are two classes of key sharing and they have a crucial difference:

One where the application is actually knowledgable about this and wants
to support this key sharing, such as in banking applications for auditing middleboxes
(money laundering rules etc..). And PLUS can be perfectly be used. the app/
middleboxes will just have to write up their key sharing as another layer in the app
and middlebox backend.

The other class of course is (L)I where the premise is that the app is unaware.
I want to make it as clear as possible that there is nothing in PLUS or
the encrypted transports it works on that enables this model. If somebody
wants to make this model work, he has to define another secret key sharing
layer in the application cocde or other layers, but PLUS needs to stay clean

> where needed. The primary threat model to defend against will be
"threat model" == "attack vector" ??
> other devices on path, by allowing a remote endpoint to detect modifications.

Detection of messups is a pretty low bar. I do not need encryption for
that. A protected checksum will suffice. The "encryption where needed" is
also i think a basic question - eg: would be nice we we had clarity during
charter:

I as assuming in my above comments that we would ALWAYS have encryption,
and that we would need application layer key sharing. Is that the model
we want ?? If not, what are the criteria for not having encryption ? Do
we think we can agree on criteria during charter ?

> TLS) which could be used for encryption of transport protocols

Cheers
    Teorless

---

From: Tom Herbert <[email-masked]>
Date: Fri, 20 May 2016 09:03:49 -0700

> DSCP is properly network-layer. If you believe in a "path layer" (which I'm pretty sure I do), you can argue that ECN is the first path layer protocol.
I'm not sure what "path layer" is. Yes every packet takes some path,
but the Internet is packet-switched not circuit-switched, so not all
packets for a flow are required to always take the same path.
Maintaining flow state in intermediate nodes is only best effort. In
the presence of multi-homing and mobility intermediate devices that
track flow state will inevitably have it wrong. Anyone who has ever
tried to build a front-end L4 load balancer understands this problem
all too well :-)

> DCSP was defined in such a way that it makes little sense in an interdomain context, and its widespread intradomain deployment means that it will never deploy across domains in the general case.
Sure, but can you give a specific example of information exposure that
is useful in interdomain context, does not diminish end user security,
and does have any detrimental effects if the information is
fabricated?

> As noted, out of band signaling is not explicitly out of scope in the current
Great.

> I don't think they are off the table; a protocol such as what we propose to develop in PLUS could certainly use IPv6 extension headers, provided they can be implemented such that applications actua…
Considering that there's already talk of sunsetting IPv4 I think it's
less of a concern-- IPv6 is the future of the Internet. In fact,
optimizing SPUD for IPv6 seems perfectly inline with encouraging
adoption wider adoption of IPv6.

> I'd assert that encapsulating things in UDP is certainly much more deployable *at the endpoints*. The open question is about adoption by on-path devices.
I'm not sure in this specific case that UDP necessarily make things
more deployable. For instance, there's the pesky problem of
fragmentation. If L3 information is put into UDP and that packet is
fragmented, only the first fragment contains the L3 information the
others don't. Either the application needs to pick some minimal MTU
and hope that packets make it through, or needs to implement PMTU
discovery (rather or not ICMP PTB is delivered to the application is
another story, in Linux it is at least). IPv6 nicely great here since
minimum MTU is 1280, IPv4 might be harder to nail down.

Tom

---

From: Dave Dolson <[email-masked]>
Date: Fri, 20 May 2016 16:17:28 +0000

Brian,
The text of the proposed charter doesn't mention "UDP"; it's just in the title.

If the intent is to require the use of UDP, it should be spelled out, I'd think.

-Dave

-----Original Message-----
From: Spud [mailto:[email-masked]] On Behalf Of Brian Trammell
Sent: Friday, May 20, 2016 7:02 AM
To: [email-masked]
Subject: Re: [Spud] [Stackevo-discuss] Fwd: Possible WG-forming follow-on to SPUD BoF

Hi, all,

Thanks, cheers,

Brian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 20 May 2016 12:43:54 -0400

The tension we always get caught up in here is 'make it work in the
existing Internet' and 'write a proper transport layer'.

Here is why I don't think the tension exists: You have to do the first as a
precondition for the second.

Adding features to TCP or developing new protocols that are transport layer
but not organized as a single stream has a small design cost and an
enormous deployment overhead. The fact that we can get fifty people to work
on such protocols in IETF says absolutely nothing about the viability of
the deployment of that protocol.

When I did particle physics, one of the profs once told me 'you don't need
five thousand people to do a billion dollar experiment but you need that
and more to get the money to do a billion dollar experiment'. It is the
same with protocol design. You don't need the hundreds of people who were
involved to write HTML5 but you need thousands of developers bought in to
move that ecosystem.

What we need to focus on is deployment. Riding on UDP doesn't look as clean
as riding on raw IP but it is much easier to get deployment because you
only need buy in from a much smaller community. You don't need Microsoft or
Apple or Linus to add your new protocol to their sockets implementation.
And that is absolutely essential because a TCP implementation that has 20+
years of optimization is always going to outperform any transport protocol
that abstracts to a single stream of data.

If you are going to do anything interesting with SPUD, the API has to
expand beyond BSD sockets so that you have the ability to provide the
application layer programmer with abstractions such as separating the
control stream from the data stream, how fast data can be sent before it
starts being buffered and all the other capabilities that people use the
UDP layer and multiplex schemes built on HTTP to provide.

The WebSockets API is now implemented in Windows and in most other network
stacks. It didn't get there because people thought the spec was a good idea
and implemented it. It got there after WebSockets was being widely used in
applications, then a specification was written and only later did the
implementations appear.

So for me, the core of what I want from SPUD is to tell me 'what
information should I expose on my UDP packets to tell middleboxes the
information they need to process it successfully in accordance with their
network acceptable use policy'.

SPUD can't provide a means of treating network gateways as damage and
routing round them, nor for that matter can any standard because that
particular trope only applies to the internetwork, the network between the
networks, the network of networks.

My network, my rules. And my rules say that my network devices don't decide
which packets flow, I do. To do otherwise is to invite my devices to spend
much of their time recruited to botnets, uploading confidential data from
my local network to the cloud and worse.

Right now we have a big problem in that the network gateways perform poorly
because they can only apply heuristics to guess how traffic should be
managed. The result is more false positives and more false negatives.

---

From: Michael Welzl <[email-masked]>
Date: Sun, 22 May 2016 21:34:20 +0200

Hi,

I’m sorry for derailing this a little - it isn’t so relevant to the BoF, but I’d really like to answer this part of the discussion:

> On 20. mai 2016, at 12.32, Brian Trammell <[email-masked]> wrote:
> 
> 

(snip)

> Yes, and if we do manage to start deploying transport protocols over IPv6, we are going to have IPv4 fallback for an indefinite future. I share your concern about happy eyeballs state explosion (a…

I have a problem with this “angry eyeballs” thing: it’s assuming a very simplistic happy eyeballing algorithm and, by calling it names, gives the impression as if the described problem was universal to all possible happy eyeballing algorithms - which isn’t the case.

For example: we often surf to the same websites, which are often on CDNs, on a path that doesn’t change very much. So if my computer would happy eyeball slowly and store information in its cache, that information is potentially valid and useful for a LONG time. Then, why not use the favored transport *only* as long as the cache hasn’t expired, and then try fall-backs if things fail?  Also the multiplicative effect that you’re pointing at can be circumvented. E.g., if TCP doesn’t work over IPv6, it doesn’t make much sense to try a different transport over v6… so the number of tests can be gradually reduced.

I’m just giving examples to say that much more reasonable mechanisms than the one that you sketch here could be devised. Happy Eyeballing doesn’t *need* to be “angry”.

Cheers,
Michael

---

From: Brian Trammell <[email-masked]>
Date: Sun, 22 May 2016 23:41:31 +0200

> On 22 May 2016, at 21:34, Michael Welzl <[email-masked]> wrote:
> 
> Hi,
> 

I agree with you here. There are lots and lots of ways to keep the state manageable. For example, quite a few impairments you might want to happy-eyeball around seem to be linked to access networks, as opposed to impairments along the path. For these, caching information on a per-interface / per-connection-profile basis amortizes the measurement overhead quite well.

I think it *is* our responsibility, though, in standardizing things that might not work everywhere, to carefully specify how the measurement of multiple features should work, because the naive implementation of these measurements done in an ad-hoc way, or designs which do not carefully consider the tradeoff between cache management overhead and network overhead, lead to the worst case, which does deserve the name.

Cheers,

Brian

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 22 May 2016 15:07:26 -0700

Not sure how close we are in standardizing things, but certainly it would be helped
to have more writeups of experiences and experimentation. I am a bit concerned if
things learned on different tracks is mutually reviewed to come to best common
conclusions. Eg: v4/v6 happy eyeballs, transport happy eyeballs, figuring out redundant
multipaths in MP-TCP/SCTP, ICE optimizations in RTCWEB/RTP, etc. pp.

All problems may be slightly different from each other, but effectively, the complexity
of a host transport stack will be to be able to do all of it, and i am sure there is
a lot of overlap, and if we're lucky we figure out common recommendations if not standards.

On Sun, May 22, 2016 at 11:41:31PM +0200, Brian Trammell wrote:
> > I???m sorry for derailing this a little - it isn???t so relevant to the BoF, but I???d really like to answer this part of the discussion:
> > 
> > I have a problem with this ???angry eyeballs??? thing: it???s assuming a very simplistic happy eyeballing algorithm and, by calling it names, gives the impression as if the described problem was universal to all possible happy eyeballing algorithms - which isn???t the case.
> > 
> > For example: we often surf to the same websites, which are often on CDNs, on a path that doesn???t change very much. So if my computer would happy eyeball slowly and store information in its cache, that information is potentially valid and useful for a LONG time. Then, why not use the favored transport *only* as long as the cache hasn???t expired, and then try fall-backs if things fail?  Also the multiplicative effect that you???re pointing at can be circumvented. E.g., if TCP doesn???t work over IPv6, it doesn???t make much sense to try a different transport over v6??? so the number of tests can be gradually reduced.
> > 
> > I???m just giving examples to say that much more reasonable mechanisms than the one that you sketch here could be devised. Happy Eyeballing doesn???t *need* to be ???angry???.
>

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 23 May 2016 12:07:17 +1000

Hi Brian,

At the BoF, I and several other people highlighted concerns about privacy and SPUD:
   http://www.ietf.org/proceedings/92/minutes/minutes-92-spud  (under "Open Discussion")

The response at that time was that these considerations were important, and that work on them could happen in parallel with the SPUD experiment.

Unfortunately, there doesn't appear to have been much obvious progress in list discussion, and draft-trammell-spud-req is quite brief on the topic:

"""
SPUD must allow endpoints to control the amount of information exposed to middleboxes, with the default being the minimum necessary for correct functioning.  This includes the cryptographic protection of transport layer headers from inspection by devices on path, in order to prevent ossification of these headers.
"""

However, I can see that some thought along these lines has been put into the proposed charter below. So, a few questions.

* In the proposed approach, will it be possible for a network operator to use PLUS mechanisms to insert arbitrary metadata (e.g., a user identifier) in the server-bound stream of a client-server protocol?  

* In the proposed approach, will it be possible for a network operator to require a client to insert a particular kind of metadata into a stream, e.g., to gain network access?

* In the proposed approach, will it be possible for a network to offer a different quality of service to servers that insert specific metadata into their streams (thereby affecting network neutrality)?

See also: http://mnot.github.io/I-D/transport-metadata-impact/

Cheers,

> On 20 May 2016, at 9:02 PM, Brian Trammell <[email-masked]> wrote:
> 
> Hi, all,
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Michael Welzl <[email-masked]>
Date: Mon, 23 May 2016 08:53:41 +0200

I agree with both of you.

> On 23 May 2016, at 00:07, Toerless Eckert <[email-masked]> wrote:
>

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 23 May 2016 12:33:14 +0200

Hi Toerless,

thanks for the detailed feedback. Find selectively a few comments inline for now. Brian and I will be working on updating the charter and then come back to you and the list.

> Am 20.05.2016 um 17:09 schrieb Toerless Eckert <[email-masked]>:
> 

Sorry, that’s actually seems to be easily miss understandable. What we wanted to say it, that PLUS will make it easier to deploy new protocol as well as encrypted protocols. However, in the wg we do not plan to design any new protocol and as a starting point we will implement PLUS for the use with existing protocols of course. We will clarify this sentences.

> - What does "under transport and application control" mean ?

… vs. middlebox control. The big goals is to give the control back to the owner which is the some layer in the endhost.

>  as part of application code.

Yes, for us that is not the main goals. However. you do find this in the requirements doc (draft-trammels-spud-req). Of course we do not list all requirements in the charter. For me this is rather a question of deployability (than functionality). However, I know how important this point is for application designers. Still providing PLUS functionality should be possible in user space as well as in the kernel. I’m sure we will have this discussion on-going in a potential wg. I don’t think it not to be emphasized that much in the charter. For it is just one aspect/benefit of many.

> the IETF some friendly slapping too, please.

We already added the work in-band here.

> they do not address how to solve the problem for applications using

This is very much focused to the application view. This is also right but we tried to keep the charter short and focused and therefore this not further discuss these (as well as other) aspect in detail. Brian and I will have another look at this, but as I said I don’t think it is need to  elaborate (only) this aspect that much.

> because it would not allow for implementation in application code.

Your last sentence is not the point we were trying to make here. Again your view is very application-centric (which is not bad; just saying that other people have other views on the problem). We simply wanted to shortly note that there are deployment issues with IP option, e.g. slow path processing of IPv6 options.

>> - enable ubiquitous deployment of encrypted higher layer protocols

Here our view on higher layer is everything above PLUS (higher than PLUS) therefore including transport.

> "nothing", and the evil-network-doer will argue "everything“.

Yes, that discussion need to take place in the wg.
> credentials then we're lame and will not be successful.

Okay… I first of all have to simply say no here. I don’t want the middleboxes to require anything here, because why would a middlebox not just say „either you show me all you content in plain or I don’t transport you data“. I believe that’s something we all don’t want and building a mechanism that could lead to this, it the last thing I want. I know what you intention is here, but we have to be more than careful here and the way you’ve just phrased it, I can just disagree. However, I would propose to have this discussion in a separate thread.

> have transport protocols inserting information in the clear without

Also, again there is not only the application view here. iI e.g. TCP is used over PLUS, it’s the transport that ‚tells‘ PLUS, please expose the equivalent to my SYN such that a firewall knows this is the first packet I’ve sent. More general, a transport will decide about exposing transport semantics and the application might expose traffic characteristics because only the applications know how the traffic should be treated.

> by a higher layer in the application.

So also again, I think this point needs further discussion in a wg. However, for me this is not the primary use case (and therefore not mentioned explicitly in the charter; even though e’ve previously discussed this point). My take is that given the higher layer needs to set up the encryption context anyway, it is the responsibility of the higher who should be allowed to decrypt something. I can see a point that it might makes sense to encryption different part of the PLUS header maybe with different keys. But that’s again just one requirement that should be discussed in a potential wg.

> we think we can agree on criteria during charter ?

Okay, here we are talking about different encryption and we should and will clarify this in the charter (while I thought this was clear). We would envisions that all higher layer protocols will encrypt everything, incl. the transport header; however that’s the responsibility of the higher layer protocol(s) and independent of what PLUS does. However, PLUS will not negotiate an own encryption context, while it will use the context from the higher protocol to encrypt potentially some data in the PLUS header. Of course we will not encrypt the whole PLUS header given the goal is to expose information to middleboxes (even when we don’t have a trust relationship with them). However, there might be other cases where we need even to encrypt something in the PLUS header, e.g. an encrypted feedback channel might be useful. If we don’t have any encryption context from the higher layer, because even unencrypted traffic could decide to use PLUS, we however maybe cannot use all PLUS features.

Mirja

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 23 May 2016 12:40:29 +0200

Hi Mark,

just a very quick reply for now: None of the three points listed below is something that we want to support, but is it really hard to prevent those cases, isn’t it? You can always add additional data on any layer.

Our big goals is it to enable more encryption and only expose selectively under the control of the endpoint (instead of simply exposing everything by default as unencrypted traffic is doing it today).

I think the more important part to consider is draft-trammels-spud-req is the following:

„… the use of SPUD for standard signaling must use a common
   vocabulary with registered codepoints allocated under relatively
   restrictive policy.“

We want control about which information should be exposed including a full spec explaining how those information should be used (and also given good reason why it’s useful).

I hope that’s helpful for now. More later.

Mirja

> Am 23.05.2016 um 04:07 schrieb Mark Nottingham <[email-masked]>:
> 
> Hi Brian,
>

---

From: Brian Trammell <[email-masked]>
Date: Mon, 23 May 2016 15:44:09 +0200

> On 20 May 2016, at 18:17, Dave Dolson <[email-masked]> wrote:
> 
> Brian,

I think the situation requires the use of UDP... especially, as Toerless says, if we make it clear that userspace implementability is a hard requirement (which IMO it is; longer reply to Toerless' charter comments to follow) in the charter.

So yes, this should be explicit in the charter.

Thanks, cheers,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Mon, 23 May 2016 15:44:25 +0200

> On 20 May 2016, at 18:03, Tom Herbert <[email-masked]> wrote:
> 

Your L4 load balancer is a path layer device. The fact that the architecture (and operational practice built around it) don't work well with it is a consequence of the architecture ignoring the existence of path layer devices.

In short (and I need a better definition than this), the "path layer" is a virtual layer in the stack comprised of those behaviors of on-path devices that keep state derived from the data plane of the network layer and/or use transport layer information (and therefore have to keep state to operate properly, as the transport layer maintains end-to-end state beyond the network interface at each endpoint).

> and does have any detrimental effects if the information is

SYN/FIN/RST, with higher probability than TCP affords that the thing that sent the FIN or RST is at least on path.

Very limited information about the transport protocol's tolerances: loss and reordering tolerance come to mind.

I'm a fan of timestamps with echoes (as with TSOPT, but with some way to indicate local delta) for diagnostics, but given that clock drift provides a few bits of fingerprint entropy, they might not meet the "does not diminish end user security" requirement if your attacker model includes people trying to track you via fingerprinting.

Note that all of these are things you get on-path with TLS over TCP, that you won't get with transport protocols that encrypt their headers, so these are examples of the "giving back" of information taken away when transport headers are encrypted, and/or the transport isn't TCP.

<snip>

> adoption wider adoption of IPv6.

I'm extremely sympathetic to this approach. "Optimize for v6" can't mean "v6 only", though, unless we don't think it will deploy until v4 is on the way out anyway.

> minimum MTU is 1280, IPv4 might be harder to nail down.

Agreed. One of the things that works really well in an all-PLUS world is you can actually have native signaling of path MTU. Until then, PLPMTUD is your friend, and can live in the PLUS shim.

Cheers,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Mon, 23 May 2016 15:43:55 +0200

hi Mark,

Thanks for the questions. We have indeed spent a fair amount of time thinking about how to address the threats to privacy discussed at the Dallas BoF, and I'd like to get text into the draft charter that scopes the work appropriately to address them.

So let me answer them in terms of how we're thinking the PLUS protocol will handle them; of course, we'll actually do the engineering in the WG, should it be chartered:

> On 23 May 2016, at 04:07, Mark Nottingham <[email-masked]> wrote:
> 
> Hi Brian,
> 

Short answer: No.

Detailed answer: The "under endpoint control" language in the spud-req and PLUS charters is meant to describe something like the following:

A sending endpoint (here, the client) controls the type and content of information exposed to the path (as TLV, CBOR, whatever; conceptually, it's a key-value mapping). Each of the keys in this mapping has a meaning. Some of the keys are "application-to-path" keys, and are set by the sender and protected by a MAC using a secret derived from the superstrate session key. Some of the keys are "path-to-application" keys, or requests for information from the path to be sent to the far endpoint (which can then echo them back, encrypted, in a subsequent packet). These are set by the sender to a fixed-length array of 0x00 bytes, to be filled in by anything along the path which understands them. The presence of the key and the length of its data are protected by the MAC, but *not* the content. 

In this case, if everyone is following the protocol, the client has to create a key ("path-to-application-user-id") to let the access network put its user ID in it. As Mirja points out, I doubt that an IETF following a restrictive key allocation policy would standardize such a thing, given BCP 188. And if so, a client could simply fail to create this key, and there would be no space in the PLUS header for the user ID. (To answer your what I suspect your next question would be, "what if the access network requires me to put the user ID path-to-app key in my packets", see the answer to the questions below). 

Of course, MUSTS and SHOULDS are nice, but what if an access network gateway abuses the header? Two possibilities here: it could co-opt an existing path-to-application key, or wedge new information into the header underneath the MTU and tear it back off. In the first case, a complicit server endpoint wouldn't even have to ignore a MAC failure, and could echo back 0, signifying "nobody on path understood this". (There might be ways to put a nonce here to defend against this, but I'm not sure what the details would look like). One way to prevent this being used for large identifier spaces would be to make most such accumulators use only a few bits, and most of the ones we're thinking of (delay accumulators, path MTU) have this property.

The second case isn't really any different from today: any endpoint can stick a shim header between TCP and the (TLS-protected) payload if it knows something on the far end will rip it off, given room under the path MTU. And if there's no room under the path MTU, then you can't steal bits from PLUS either. Indeed, if PLUS makes good use of PLPMTUD, or supported explicit PMTU in a future with PLUS-enabled routers, then the amount of space available for such shenanigans could be reduced. There are also countless out-of-band ways to associate user IDs with flow identifiers that PLUS can't do anything about, either.

> * In the proposed approach, will it be possible for a network to offer a different quality of service to servers that insert specific metadata into their streams (thereby affecting network neutrali…

I think these are two different shades of the same threat: "tell me something I want to hear, or I drop your packets", "tell me something I want to hear, or I'll give you crap service", where "something I want to hear" is either some secret verifying you've done something the operator wants you to do (paid money, watched an ad) or information about the traffic the operator could independently verify but doesn't want to have to (otherwise, you could just lie).

Short answer: I do not think it is possible to prevent this kind of behavior, even in the current architecture, because it isn't really a technical problem. But it would be possible with PLUS to make it transparent (which is the reason I don't see PLUS as enabling this behavior).

Long answer: Here, the threat is above layer 7: the application is "forced" to hand over a token in order to get connectivity of acceptable performance. This token might represent information that could currently be taken through DPI, but can't when that information is protected via TLS. 

Sadly, I can see no technical solution to this problem at all. An operator could today force an to place information in bits under its control (IPv6 source and destination address, flow label; TCP source port) for such a thing, or leverage the relative ease of five-tuple routing plus the ease of standing up HTTPS servers in random bits of the cloud to use out-of-band mechanisms ("give me your token and your five-tuple via a TLS-protected API with a server somewhere on a CDN network that you can't easily tell is Your Friendly Local ISP, and you get to ride in the fast lane.") So assuming such a mechanism is deployed without standardization, at least the presence of the keys in the PLUS header would make this behavior visible to *everyone* on the network. That alone is IMO a reason it wouldn't be deployed in this way -- the relatively vigorous opposition to this kind of thing from customers who care means operators would rather hide it.

Thanks, cheers,

Brian

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 23 May 2016 07:03:25 -0700

> Long answer: Here, the threat is above layer 7: the application is "forced" to hand over a token in order to get connectivity of acceptable performance. This token might represent information that…

I like that answer much better than what Mirjan sent in before. But when you are saying
"PLUS to make it transparent" - that was exactly  what i was trying to get to, and
i don't think thats understood so far: How would the application know what it needs
to send to get through. Right now it does not know (existing firewalls). 

So, how in PLUS would an app know what it needs to send to get through ?

And then even more importantly: How can we make sure that anything that needs to be exposed
to the firewall is known to be exposed only to the "trusted" firewall of eg: your enterprise,
but not potentially all along the path to any other observers whom the app does not
want to show its credential.

Btw: there are good and IMHO easy answers to these questions, i just haven't seen
clear evidence whether PLUS intends to provide them.

Cheers
    Toerless

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 23 May 2016 16:33:52 +0200

I don’t think we can deny the existence of stateful network work devices anymore. Even though your load balancer does not work (well), there are other things such as firewall which need to keep state for a good reason.

Mirja

> Am 23.05.2016 um 15:44 schrieb Brian Trammell <[email-masked]>:
>

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 23 May 2016 16:47:51 +0200

Hi Toerless,

see below.

> Am 23.05.2016 um 16:03 schrieb Toerless Eckert <[email-masked]>:
> 

Okay, that’s a good point. However, from the PLUS point of view that not the important question for me because if you are willing to provide certain information because they are not privacy-sensitive and there simply might be a chance that by sending the information you get the service that you’d like to get/need, why not just providing these information…? Simple example again: don’t really see a reason to hind the TCP SYN/not expose the first packet of a flow in PLUS, so just do it.

I think , you again have a little bit more the application view here and there the question is e.g. if you would like to use quic (as an example): how do I know it works? Here I think you still need some happy eyeball or fallback mechanism in the transport layer (or higher) (see taps wg). However, SPUD could help you with a more meaningful error message why your initial packet was dropped and therefore assist the higher layer to implement the fallback appropriately e.g. speeding up the fallback because you don’t have to wait for a timeout.

However, I’m not sure if there is really a point in discussing this at a general level. I think this has to be discussed separately for each piece of information that we decide to support/provide in PLUS.

Mirja

---

From: Joe Touch <[email-masked]>
Date: Mon, 23 May 2016 08:56:41 -0700

As I've pointed out repeatedly, OS implementation issues should not
drive network architecture.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 23 May 2016 09:00:54 -0700

On 5/23/2016 6:44 AM, Brian Trammell wrote:
>> ...
> Your L4 load balancer is a path layer device. 

The only notion the Internet has of a path is a flow label.

Transport IDs identify services and connections within an endpoint, and
are not universally available or meaningful except at the endpoints.

All attempts by intermediate devices to rely on those identifiers makes
assumptions about visibility and correlation to desired shared-path
behavior that are baseless.

This is the key reason why MP-TCP is very badly misnamed. It is
multi-endpoint, not multipath, and always has been.

Joe

---

From: Brian Trammell <[email-masked]>
Date: Mon, 23 May 2016 18:03:43 +0200

> On 23 May 2016, at 17:56, Joe Touch <[email-masked]> wrote:
> 
> As I've pointed out repeatedly, OS implementation issues should not drive network architecture.

Indeed.

A very large part of the problem here is the narrowness of the interface from the operating system down to TCP; attempts to work around this either have to sit on top of it (abstracting away some of the ugliness of socket options to provide a closer-to-DWIM interface to the programmer), or implement entirely new ways of doing things with little interoperability across libraries. You're also quite right in pointing out that *this* is not the IETF's problem.

IMO, leveraging the work done in making userspace stacks work fast gives us a way to sidestep the API in userspace. But assuming it is... in this world, userspace processes do direct DMA off the NIC, but we still need a transport header so the kernel knows which process gets to access which region of memory. If the transport is TCP, and has semantics that are TCP-enough that the network won't choke on them, then this header should be a TCP header. If it isn't, then we need a header that won't set up an expectation of TCP semantics on the wire; that leaves us with UDP if we acknowledge the existence of NATs and the expansion of five-tuple forwarding in general.

So it's not OS implementation details that are driving architecture, in this case, but the built environment.

Cheers,

Brian

>>> From: Spud [
>>> mailto:[email-masked]
>>> ] On Behalf Of Brian Trammell
>>> To:
>>> [email-masked]
>>>> From: Toerless Eckert [
>>>> mailto:[email-masked]
>>>> ]
>>>>>> From: Stackevo-discuss [
>>>>>> mailto:[email-masked]
>>>>>> ] On
>>>>>> Behalf Of Brian Trammell
>>>>>> Forwarding this to the correct address for this mailing list (oops). Please discuss at
>>>>>> [email-masked]
>>>>>> .
>>>>>> 
>>>>>> Thanks, cheers,
>>>>>> 
>>>>>> Brian
>>>>>> 
>>>>>> 
>>>>>>> Begin forwarded message:
>>>>>>> 
>>>>>>> From: Brian Trammell
>>>>>>> <[email-masked]>
>>>>>>> 
>>>>>>> To:
>>>>>>> [email-masked]
>>>>>>> Cc:
>>>>>>> [email-masked], [email-masked]
>>>>>>> , "Mirja Kuehlewind
>>>>>>> (IETF)"
>>>>>>> <[email-masked]>
>>>>>>> , Spencer Dawkins at IETF
>>>>>>>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 23 May 2016 09:09:53 -0700

> Your L4 load balancer is a path layer device. The fact that the architecture (and operational practice built around it) don't work well with it is a consequence of the architecture ignoring the exi…

But the architecture here is that of the Internet which is inherently
packet-switched not circuit switched. I don't see how "path layer"
fits into that architecture, and we can't change the core Internet
protocols to accommodate it. Any intermediate device that creates
state presumes that flows follow a persistent and invariant path will
eventually, at some point, be wrong about the state. This is why
signaling connection state can only ever be considered best effort, in
the network we cannot operational rely on it to always be available or
correct. For instance, consider in PLUS a FIN is defined for end of
flow. Now suppose that some connection followed five different paths
during its lifetime. That means when the FIN is sent only the devices
on one of those paths will see it and know to destroy flow state.
State in devices on the other four paths is left hanging indefinitely
so they still need to implement some state eviction method like
timeout.

We solve this problem in out L4 load balancers by having them
cooperate with each other to perform identical consistent hashing or
out of band sharing of flow information. I don't believe such
cooperative methods can be used in the Internet between arbitrary
third party devices.

> Very limited information about the transport protocol's tolerances: loss and reordering tolerance come to mind.
Don't we already have diff-serv for such things?

> Agreed. One of the things that works really well in an all-PLUS world is you can actually have native signaling of path MTU. Until then, PLPMTUD is your friend, and can live in the PLUS shim.
We already have PMTU discovery (RFC1191). The fact that it's
considered unusable because many network devices don't implement it is
is not a good rationale to create a new mechanism that provides the
same information but requires other new implementation in those same
devices.

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 23 May 2016 09:15:07 -0700

Mirja:

I am sure privacy advocats will rightfully argue that anything beyond whats
necessary is bad to disclose.

What i wouldlike to have is the ability of PLUS to mange whats signaled:
As an app, i'd be happy to signal relevant data up to the point (eg: firewall)
where it leaves my local trust-domain. To do this, i'd either need to encrypt
the info so if it leaks beyond that trust edge, its useless, or PLUS should be
abble to remove the info, and in any case, the sender would need a mechanism
to figure out if three will be a trust-edge removing the info before the
flow enters the untrusted Internet.

Cheers
    Toerless

On Mon, May 23, 2016 at 04:47:51PM +0200, Mirja K?hlewind wrote:
> Okay, that???s a good point. However, from the PLUS point of view that not the important question for me because if you are willing to provide certain information because they are not privacy-sensitive and there simply might be a chance that by sending the information you get the service that you???d like to get/need, why not just providing these information???? Simple example again: don???t really see a reason to hind the TCP SYN/not expose the first packet of a flow in PLUS, so just do it.
> 
> I think , you again have a little bit more the application view here and there the question is e.g. if you would like to use quic (as an example): how do I know it works? Here I think you still need some happy eyeball or fallback mechanism in the transport layer (or higher) (see taps wg). However, SPUD could help you with a more meaningful error message why your initial packet was dropped and therefore assist the higher layer to implement the fallback appropriately e.g. speeding up the fallback because you don???t have to wait for a timeout.
> 
> However, I???m not sure if there is really a point in discussing this at a general level. I think this has to be discussed separately for each piece of information that we decide to support/provide in PLUS.
> 
> Mirja
> 
> 
> >

---

From: Brian Trammell <[email-masked]>
Date: Mon, 23 May 2016 18:15:39 +0200

> On 23 May 2016, at 18:00, Joe Touch <[email-masked]> wrote:
> 
> 
> 

But those intermediate devices *do indeed* route those packets based an N-tuple, where N is generally 5. One can ignore this fact and be surprised by what the network does, or resign oneself to it and leverage it.

> multi-endpoint, not multipath, and always has been.

You're quite right, but the current routing architecture gives us no way to influence paths except by choosing endpoints. And even then, we can know very little about how the endpoint identifiers map to paths.

Cheers,

Brian

---

From: Joe Touch <[email-masked]>
Date: Mon, 23 May 2016 09:17:20 -0700

> IMO, leveraging the work done in making userspace stacks work fast gives us a way to sidestep the API in userspace. But assuming it is... in this world, userspace processes do direct DMA off the NI…

You need a transport header to define multiple protocol machines inside
an endpoint, period. The rest is an OS implementation issue.

However, once you accept that header as a demux point, the payload has
to be sufficiently complete. TCP in UDP isn't.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 23 May 2016 09:19:03 -0700

> But those intermediate devices *do indeed* route those packets based an N-tuple, where N is generally 5. One can ignore this fact and be surprised by what the network does, or resign oneself to it…

Or we can encrypt the transport and disable the network's ability to do
this altogether.

> You're quite right, but the current routing architecture gives us no way to influence paths except by choosing endpoints.
We have routing headers for *exactly* this purpose.

>  And even then, we can know very little about how the endpoint identifiers map to paths.
Yes; that's why we have routing headers.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Mon, 23 May 2016 09:20:26 -0700

> But those intermediate devices *do indeed* route those packets based an N-tuple, where N is generally 5. One can ignore this fact and be surprised by what the network does, or resign oneself to it…
If you're referring to ECMP that does not require state to be created
it operates fine being stateless. Stateful devices are not required to
route packets.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Mon, 23 May 2016 10:17:02 -0700

Mirja,

There was a great presentation in Buenos Aires on UDP usability
(https://www.ietf.org/proceedings/95/slides/slides-95-maprg-3.pdf).
Unless I'm reading it incorrectly, the conclusion is that UDP is
mostly usable on the Internet as is, stateful devices can already deal
with it even though UDP has no concept of explicit signaling for
connection creation/tear down. Also, judging from the comments from
network operators that block UDP, it seems they are more inclined to
assume all UDP is bad and I don't think we've seen any indication that
adding explicit state signaling would change that view.

I suppose it's a natural inclination to try to mimic TCP semantics
because of its pervasiveness, but if things already work well enough
then I don't see what problem it solves. The addition of such
signaling increases the complexity of the protocols, the networking
stack, network devices, and creates yet another opportunity for
ossification. So, yes, we can acknowledge firewalls and other stateful
devices do exist, but if existing protocols already work I'm not
seeing why we have to do anything differently.

Tom

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 24 May 2016 10:59:55 +1000

Hi Brian,

> In this case, if everyone is following the protocol, the client has to create a key ("path-to-application-user-id") to let the access network put its user ID in it. As Mirja points out, I doubt tha…

That sounds good, as long as something (ideally, a forwarding node, but at least the receiving endpoint) breaks in a fairly ugly fashion when the MAC isn't correct. This needs to be specified behaviour, and actually implemented by at least a sizeable population, so that it sticks.

Question: is application-to-path information exposed to the other endpoint, and if so, is its delivery there guaranteed by the protocol? 

In other words, if my phone inserts metadata to the path, can the Web site on the other end reliably detect its presence?

From your answer below, it sounds like that's the case, just wanted to be sure.

> Of course, MUSTS and SHOULDS are nice, but what if an access network gateway abuses the header? Two possibilities here: it could co-opt an existing path-to-application key, or wedge new information…

That sounds like the start of text for security considerations + registry requirements...

> The second case isn't really any different from today: any endpoint can stick a shim header between TCP and the (TLS-protected) payload if it knows something on the far end will rip it off, given r…

*nod*

> Sadly, I can see no technical solution to this problem at all. An operator could today force an to place information in bits under its control (IPv6 source and destination address, flow label; TCP…

I don't disagree on any particular point. What concerns me are situations where people are asked to install a particular piece of software, or configure their machines a particular way, to be able to access the Internet; it's already a problem with things like browser extensions and CA certs, and adding more surface area doesn't seem like a good direction to go in.

Thanks!

---

From: Brian Trammell <[email-masked]>
Date: Tue, 24 May 2016 09:35:43 +0200

> On 24 May 2016, at 02:59, Mark Nottingham <[email-masked]> wrote:
> 
> Hi Brian,
> 
> That sounds good, as long as something (ideally, a forwarding node, but at least the receiving endpoint)

Receiving endpoint is trivially easy. Forwarding node is more difficult and requires more overhead, but possible. But I think the details of this requirement and the specification are IMO post-BoF work.

> breaks in a fairly ugly fashion when the MAC isn't correct.

Yep, this would be a rage quit: regardless of whether a forwarding node assists in detection, the PLUS layer on the receiving endpoint tells the superstrate transport that the association is hard reset.

> This needs to be specified behaviour, and actually implemented by at least a sizeable population, so that it sticks.

Agree completely.

> Question: is application-to-path information exposed to the other endpoint, and if so, is its delivery there guaranteed by the protocol?

Yes, with a caveat on "guaranteed": if the packet gets delivered to the other endpoint (subject to the reliability requirements and mechanisms of the superstrate), then *all* the information

One major change in our thinking from what's in the SPUD prototype, which came from experience with it, is that a2p and p2a data aren't really all that different from an API standpoint. There's simply information bound to a tube whose content is integrity protected, and which on-path devices must not modify or drop, and information bound to a tube whose presence and length are integrity protected, and which on-path devices may modify. This is a property of the information element's identifier, and the identifier itself is probably encoded in a way that the integrity code and easily generate and verify MACs without doing a bunch of registry lookups. The sending endpoint has complete control over the information elements it sends and their content, and the receiving endpoint has access to all of them.

(There's an API requirement hiding in there.)

> In other words, if my phone inserts metadata to the path, can the Web site on the other end reliably detect its presence?

In the absence of misbehavior of the second case, below, yes.

> That sounds like the start of text for security considerations + registry requirements...

yep. thanks for asking the question. :)

> I don't disagree on any particular point. What concerns me are situations where people are asked to install a particular piece of software, or configure their machines a particular way, to be able…

I share this concern. I think we have different estimations of how much additional surface area PLUS practically presents.

Thanks, cheers,

Brian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 24 May 2016 17:40:25 +1000

> On 24 May 2016, at 5:35 PM, Brian Trammell <[email-masked]> wrote:
> 

I don't have an estimation yet. :)

Thanks for taking the time to give detailed answers!

Cheers,

---

From: Brian Trammell <[email-masked]>
Date: Tue, 24 May 2016 09:57:22 +0200

hi Tom,

> On 23 May 2016, at 19:17, Tom Herbert <[email-masked]> wrote:
> 
> connection creation/tear down.

Speaking as the presenter of that talk, this wasn't what we measured. Abusing UDP traceroute on the RIPE Atlas probes supports only a basic connectivity test - "does the first packet make it through, and how much slower/faster is it than TCP?" is the question we asked.

Measuring the state timeout spectrum, and differences for UDP and TCP, is near-future work. It's not something we can leverage Atlas for, so getting access to a whole lot of access networks will be hard. We presume that it's still a problem.

> Also, judging from the comments from

Adding explicit state signaling won't change that view. Adding productive, well-behaved, congestion-controlled traffic with an IP protocol number of 17 to the Internet (i.e. QUIC, PLUS, WebRTC) will.

"Block/rate limit UDP" is not, as far as I can tell, a universally accepted and applied policy in the Internet. Gaming, for instance, seems to work in the Internet. Networks that don't have a lot of gamers on them (enterprises, mobiles in many markets) can afford to use heuristics to limit harmful UDP traffic. Other networks can't, unless they want all their gamers to go find other providers.

> seeing why we have to do anything differently.

If it just works, I agree. I don't think that's the case, though. Data, hopefully, to follow.

Cheers,

Brian

---

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 24 May 2016 10:13:30 +0200

Hi Tom,

> Am 24.05.2016 um 09:57 schrieb Brian Trammell <[email-masked]>:
> 
>> ossification.

The problem is it only works for what we have right now. It works less well for new things which is a pity… it’s ossified. Yes, adding more stuff will make it more complex; I guess that’s a general case but I don’t think is an argument to stop trying to solve the problem. And also yes, it might get ossified again, but if we do it right this time, I want it to get ossified and settle as a common base for new protocols to get (more quickly and at all) deployed.

Mirja

---

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 24 May 2016 10:47:30 +0200

One quick comment below.

> Am 23.05.2016 um 18:09 schrieb Tom Herbert <[email-masked]>:
> 

Yes, that’s a very valid agreement. I don’t think we can get fully rid of these mechanisms, also not all traffic will support PLUS. However, if it is supported and you get the information is can help you to make the write decision earlier. And the more PLUS deployment we see, the more it will actually free memory and computation resources...

---

From: Tom Herbert <[email-masked]>
Date: Wed, 25 May 2016 07:46:39 -0700

Mirja,

I will argue that what we have does not work right now. If a NAT state
is evicted in a network device then that means we lose the TCP
connection often without any notification. This has forced widespread
use of NAT keepalives which are basically junk packets sent into the
Internet that serve no other purpose than keep NAT state fresh. This
is a real problem especially on mobile devices where we waking up a
transmitter to send a keepalives is drain on the battery.

A proposed solution to this is what I'm calling "disassociated
location" (described in draft-herbert-transports-over-udp-00 and
borrowed from QUIC). The idea is that the endpoints do not use the IP
addresses and port numbers to identify a connection, they instead use
connection identifiers that are negotiated between the two endpoints
to be unique the other connections in the nodes. With this method, I
believe that connections can survive a NAT eviction. So if a
connection goes idle we no longer have to worry about maintaining NAT
state (presuming the client is the one that initiates requests). And
when an idle client wakes up it can assume that the NAT state has
timed so it may as well use new port numbers and new flow identifiers;
to the network this looks like a completely new flow (good for
security), but to the transport endpoints it is a continuation of an
established connection.

Similarly this should resolve the multi-homing/mobility issue.
Currently, if the path of a connection changes and packets are routed
through a different NAT device we lose the connection. With a
transport over UDP and disassociated location the connection remains
viable in that scenario.

This model only assumes typical UDP-NAT behavior in the network (like
in RFC4787) and does not require any new "path layer" protocol. I
don't see how exposing connection state can help here, and honestly I
wouldn't even know how to set it-- during the lifetime of one
connection it would be seen by the network to be different flows at
different times and (hopefully) would be unable deduce that these
refer to the same transport layer connection.

Tom