Subject: Re: [Spud] [Stackevo-discuss] Fwd: Possible WG-forming follow-on to SPUD BoF

From: Dave Dolson <[email-masked]>
Date: Fri, 20 May 2016 02:29:58 +0000

Please also think about how a network operator can monitor how well things are working, and how DDoS traffic may be discriminated from non-attack traffic.
Today,
- TCP packet loss can be measured by observing sequence numbers
- round-trip times can be measured ‎from TCP sequence/ack numbers
- non-attack traffic properly completes connections, whereas attack traffic often does not
- some fair-queuing approaches apply fairness by TCP connections‎
- ECMP and load-balancers try to make individual sessions follow a single path

If packets become completely opaque, I think ‎in general the internet would operate less well.
It would be good for SPUD and QUIC ‎to provide enough information for monitoring devices to measure progress, handshakes, round-trip times, as can be done with TCP carrying TCP.

-Dave
‎

  Original Message  
From: Tom Herbert
Sent: Thursday, May 19, 2016 3:17 PM
To: Toerless Eckert
Cc: Brian Trammell; Scharf, Michael (Nokia - DE); [email-masked]
Subject: Re: [Spud] [Stackevo-discuss] Fwd: Possible WG-forming follow-on to SPUD BoF

On Thu, May 19, 2016 at 11:27 AM, Toerless Eckert <[email-masked]> wrote:
> Tom, Michael
>
> I am reading your positions and i am confused:
>
> DSCP is very successfull in enterprise networks. SPUD does not have to
> be successfull "on the Internet". To me it would already be a great success if it would
> help to improve controlled network services (like enterprise or 4G/5G networks). And
> their transit traffic across the Internet (without changing anything on the Internet
> itself).
>
Yes, that is my point about localizing the problem. A client device
should know what network it's on an be able to signal to that network
to get the services it wants. A server on the other side of the planet
that the node is talking to really shouldn't be concerned with this
and not required to implement some complex protocol to accommodate it.

> AFAIK, TOS/DSCP/ECN are not per-flow, but per-packet mechanisms. Even if we often think about
> them as per-flow and ould maybe like them to be restrict to per-flow. At least i don't know
> any normative reference that defines them to be per-flow.
>
> Wrt to out-of-band/out-of-path: The whole reason for SPUD is the analysis that all those
> mechanisms fail to provide an easily deplopyale framework in the face of middleboxes. Aka: their
> presence and mostly non-success should be an argument FOR working on spud. Not against it.
>
> If a NAT/FW does support any flow signaling, its most likely inband. Every NAT/firwall
> supports flow recognition via the explicit signaling of flow-based transport protoocols
> like TCP. They do try to do the same for UDP and because that has no explicit flow signaling,
> thats where the problem starts. NAT/FW do deep packet inspection which has all these problems
> we know. SPUD sets out to solve/reduce these NAT/FW issues.
>
There seems to be a lot of UDP going through NAT/FW today without any
sort of in-band signaling, and protocols like QUIC are already seeing
deployment without that (according to their slide deck "About half of
Google to Chrome"). Considering that there is no deployment of a SPUD
aware firewall and we can't want to wait for every device on the
Internet to upgrade, SPUD will have to work with NAT/FW as is or we
couldn't consider it. TBH I'm really having a hard time seeing that
any extra signaling is required in the Internet other than what is in
the IP header, at best it seems like an optimization for certain
networks or configurations.

> Non-support of anything like IP options is IMHO primarily driven by the absence of
> unform cross-OS/Dev-language APIs for them. Yes. It may suck architecturally by designing
> protocol stacks based on reality like: applications will only be able to use UDP and TCP,
> so lets do new work on top of them, but to me the most important goal is to find the best
> deployable solution to a problem. Not the architecturally cleanest one. If you give me
> research money to work outside of reality i will happily take it though and represent a
> different perspective.
>

There has been a lot of discussion about this in v6ops, many networks
are simply dropping packets with extension headers. Please look at
draft-ietf-v6ops-ipv6-ehs-in-real-world-02 and
draft-gont-v6ops-ipv6-ehs-packet-drops-03. Because of this and the
fact that there hasn't been any interesting options defined worth
sending to date means that we (speaking for applications) have no use
to send them (btw all the major OSes support options AFAIK). If some
interesting options did come about, then we would probably apply a
happy eyeballs approach to deploy.

With something like SPUD (basically anything not simple TCP/IPv4) we
are always faced with the same problem that some networks on the
Internet may arbitrarily drop or otherwise restrict packets (we know
for instance that some network operators choose to severely rate limit
all UDP because of DOS concerns). So even if we do manage to start
deploying transport protocols over UDP we are going have TCP as a
fallback for indefinite future, i.e. more happy eyeballs.

Tom

> Please, i do not want to dismiss your opposition, i just don't understand how the
> arguments you make are working out. Maybe you can explain to me where i am going wrong
> in my interpretations above.
>
> Cheers
> Toerless
>
> On Thu, May 19, 2016 at 10:50:21AM -0700, Tom Herbert wrote:
>> On Thu, May 19, 2016 at 10:12 AM, Scharf, Michael (Nokia - DE)
>> <[email-masked]> wrote:
>> > Does the sentence ...
>> >
>> > The current Internet protocol stack has no layer for explicit signaling of flow semantics and characteristics to network elements ...
>> >
>> > actually wants to say something like
>> >
>> > The current Internet protocol stack has no layer for explicit *in-band* signaling of flow semantics and characteristics ... ?
>> >
>> > I may be wrong, but there is no shortage of IETF and non-IETF protocols that can signal flow semantics and characteristics out-of-band, either path-coupled or path-decoupled, e.g., in the control or management plane. As a result, there are protocol "layers" for that signaling, but they are not in-band. There may also be other ways to fix this problem (e.g., by adding "end-to-end" or "inter-domain" instead of "in-band").
>> >
>> Not to mention that TOS and diff-serv are in-band signaling mechanisms
>> to the network, ECN is explicitly signaling from the network. Given
>> that they've been around for many years but have never been widely
>> deployed on the Internet makes me wonder if any in-band signaling
>> mechanism will ever catch on. The use of out of band signaling should
>> be considered, especially considered that in many cases like NAT and
>> firewall the network elements that need signaling are likely in local
>> networks of one side of the connection.
>>
>> I have a similar concern about the statement that IP options/EH and
>> extension headers are being left off the table as part of a solution.
>> The fact that neither have these caught on in the Internet and that
>> tells me two things 1) No IP option is needed for the Internet to
>> operate and be successful which makes me wonder if signaling between
>> network and hosts is even required 2) If the extensibility mechanisms
>> of IP are chronically not deployable, I wonder why any alternative
>> that does not yet exist but intends to solve the same problem would be
>> any more deployable.
>>
>> Tom
>>
>> > Michael
>> >
>> >
>> > -----Original Message-----
>> > From: Stackevo-discuss [mailto:[email-masked]] On Behalf Of Brian Trammell
>> > Sent: Thursday, May 19, 2016 5:07 PM
>> > To: [email-masked]
>> > Subject: [Stackevo-discuss] Fwd: Possible WG-forming follow-on to SPUD BoF
>> >
>> > Greetings, all,
>> >
>> > Forwarding this to the correct address for this mailing list (oops). Please discuss at [email-masked].
>> >
>> > Thanks, cheers,
>> >
>> > Brian
>> >
>> >> Begin forwarded message:
>> >>
>> >> From: Brian Trammell <[email-masked]>
>> >> Subject: Possible WG-forming follow-on to SPUD BoF
>> >> Date: 19 May 2016 at 17:04:50 GMT+2
>> >> To: [email-masked]
>> >> Cc: [email-masked], [email-masked], "Mirja Kuehlewind
>> >> (IETF)" <[email-masked]>, Spencer Dawkins at IETF
>> >> <[email-masked]>
>> >>
>> >> Greetings, all,
>> >>
>> >> We propose to hold a working-group forming BoF in Berlin as a follow-on to the SPUD work, to define a common substrate protocol for encrypted transports based on the requirements derived from experimentation with the SPUD prototype.
>> >>
>> >> First, note that since the acronym "SPUD" refers primarily to the prototype itself, any follow-on working group should have a different name. We're using the derived requirements, not starting for the prototype. The name is a subject for discussion, and suggestions are welcome. To have something to put in the proposed charter, we'd propose "Path Layer UDP Substrate" (PLUS) as a starting point.
>> >>
>> >> The proposed charter appears below. We're interested in hearing initial feedback on the proposed charter in preparation for a BoF proposal (the cutoff date is two weeks from tomorrow, on Friday 3 June). Is there work to do here within the IETF? Is the scope of the proposed charter appropriate? Is there energy to do this work?
>> >>
>> >> Thanks, cheers,
>> >>
>> >> Brian and Mirja
>> >>
>> >>
>> >> Path Layer UDP Substrate (PLUS)
>> >> ===============================
>> >>
>> >> The PLUS working group???s goal is to enable the deployment of new,
>> >> encrypted transport protocols, while providing a transport-independent
>> >> method to signal flow semantics under transport and application control.
>> >>
>> >> The current Internet protocol stack has no layer for explicit
>> >> signaling of flow semantics and characteristics to network elements,
>> >> nor an integrated signaling mechanism from network elements to back to
>> >> endpoints and applications. This layer never evolved within the stack,
>> >> because middleboxes and other devices on path could simply inspect and
>> >> modify headers and payload of unencrypted traffic at every layer. This
>> >> implicit use of information from the transport and application layers
>> >> is a key origin of the ossification that makes it hard or impossible to deploy new protocols.
>> >>
>> >> In order to support more ubiquitous deployment of encryption, explicit
>> >> signaling must be added to the stack, and it must be transport
>> >> protocol independent. While IP would seem to be the natural home for
>> >> this facility, both IPv4 and IPv6 options and extensions have
>> >> deployment problems on their own, which makes it hard to include any
>> >> additional information in these protocols. Additionally, a feedback
>> >> channel that provides information from on-path devices back to
>> >> endpoints and applications, e.g. for error handling, is essential for
>> >> the deployment and success of an explicit cooperation approach.
>> >>
>> >> The PLUS working group will specify a new protocol as a Path Layer
>> >> User Substrate (PLUS), to support experimental deployment of explicit
>> >> cooperation between endpoints and devices on path, with the following goals:
>> >>
>> >> - enable ubiquitous deployment of encrypted higher layer protocols by
>> >> providing exposure of similar semantics to existing protocols (e.g.
>> >> TCP) to devices on path (e.g. NATs and firewalls)
>> >>
>> >> - allow applications and transport protocols to explicitly provide
>> >> limited information to devices on path
>> >>
>> >> - allow devices on path to provide feedback and information about the
>> >> path to sending endpoints, under sending endpoint control
>> >>
>> >> - allow devices on path to provide information about the path to
>> >> receiving endpoints, with feedback to the sending endpoint, under
>> >> sending endpoint control
>> >>
>> >> Note that this approach explicitly gives the control of information
>> >> exposure back the application and/or transport layer protocol on the
>> >> end host. It is the goal of PLUS to minimize the information exposed
>> >> at the level of detail that is useful for the network, while
>> >> encrypting everything else. This is important to avoid future implicit
>> >> treatment and the resulting ossification, as well as to leverage the
>> >> principle of least exposure to minimize privacy risks presented by explicit cooperation.
>> >>
>> >> Given that the primary goal of PLUS is to enable the deployment of
>> >> arbitrary, fully encrypted transport protocols, we assume that the
>> >> higher layer protocol can provide an encryption context that can be
>> >> used by PLUS to provide authentication, integrity, and encryption
>> >> where needed. The primary threat model to defend against will be
>> >> modification or deletion of exposed information by middleboxes and
>> >> other devices on path, by allowing a remote endpoint to detect modifications.
>> >>
>> >> The working group will start with an initial set of use cases (see
>> >> draft-kuehlewind-spud-use-cases) and requirements (see
>> >> draft-trammell-spud-req), taken from experience with the Substrate Protocol for User Datagrams prototype.
>> >> The working group's main output will be an experimental protocol
>> >> specification, together with an initial registry of types of
>> >> information that can be exposed using PLUS, clearly aligned to these
>> >> use cases and requirements. The working group will close if it is not
>> >> able to come to consensus on a protocol design to meet these requirements.
>> >>
>> >> The working group will additionally aim to identify other working
>> >> groups that could or should address parts of these requirements within
>> >> existing protocols, e.g. by specifying new protocol extensions or as
>> >> input for on-going standardization work. It will aim to work with
>> >> working groups defining encryption protocols (e.g. TLS) which could be
>> >> used for encryption of transport protocols running over PLUS.
>> >>
>> >
>> > _______________________________________________
>> > Spud mailing list
>> > [email-masked]
>> > https://www.ietf.org/mailman/listinfo/spud
>>
>> _______________________________________________
>> Spud mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/spud
>
> --
> ---
> Toerless Eckert, [email-masked]

_______________________________________________
Spud mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/spud

---

From: Dave Dolson <[email-masked]>
Date: Fri, 20 May 2016 02:31:17 +0000

Er, make that "TCP carrying TLS" in the last sentence.

  Original Message
From: Dave Dolson
Sent: Thursday, May 19, 2016 10:30 PM
To: Tom Herbert; Toerless Eckert
Cc: Brian Trammell; Scharf, Michael (Nokia - DE); [email-masked]
Subject: Re: [Spud] [Stackevo-discuss] Fwd: Possible WG-forming follow-on to SPUD BoF

-Dave
‎

  Original Message
From: Tom Herbert

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 19 May 2016 19:42:37 -0700

+1

a) The ECMP/fair-queueing issues should be resolved if SPUD as i hope it will simply
   maps UDP/SPUD ports 1:1 to the transport protocols inside of it. No aggregation.
b) SPUD is meant to add connection setup/close signaling even if the original protocol did
   not have it. And having a single con setup/teardown signaling to network elements would
   allow new transport protocols to come up with whatever they want without having to
   bother with network devices not understanding them.
c) Wrt to measurements: I don't think you can measure RTT from sequence/ack.
   I do like the idea of measuring loss, but if i want to enable agility of transport protocol
   designs then its a bad idea to rely on headers that are transport specific. In the same
   way as SPUD would have common setup/teardown signaling, it could have an optional
   measurement header. Make it kinda RTP-lite to allow east quality measurements.
   Sequence-number and timestamp (timestamps are great for delay/jitter measurements).

On Fri, May 20, 2016 at 02:29:58AM +0000, Dave Dolson wrote:
> - round-trip times can be measured ???from TCP sequence/ack numbers
> - some fair-queuing approaches apply fairness by TCP connections???
> If packets become completely opaque, I think ???in general the internet would operate less well.
> It would be good for SPUD and QUIC ???to provide enough information for monitoring devices to measure progress, handshakes, round-trip times, as can be done with TCP carrying TCP.
> 
> -Dave
> ???
> 
>   Original Message  
> From: Tom Herbert

---

From: Brian Trammell <[email-masked]>
Date: Fri, 20 May 2016 12:45:59 +0200

hi Dave,

> On 20 May 2016, at 04:29, Dave Dolson <[email-masked]> wrote:
> 
> Please also think about how a network operator can monitor how well things are working,

This is one of the aims of section 9 "In-band Measurement" of draft-kuehlewind-spud-use-cases.

> and how DDoS traffic may be discriminated from non-attack traffic.

Good point; this is a bit harder in the general case. One feature of the SPUD prototype, which we abstract out in section 6.6 "Protection against trivial abuse" of draft-trammell-spud-req, also provided by draft-herbert-udp-magic-numbers, is that by carefully designing the UDP payload of an encapsulation layer, and making that encapsulation layer reflection-resistent, you can make it easy to discriminate reflection-amplification traffic from encapsulated traffic.

> - TCP packet loss can be measured by observing sequence numbers

...and making some assumptions that usually hold, and being very careful about where your observation point is located with respect to sender and receiver, yes;

> - round-trip times can be measured ‎from TCP sequence/ack numbers

...if you're not terribly concerned about accuracy (see [1]), but yes;

> - non-attack traffic properly completes connections, whereas attack traffic often does not

...an assumption which mostly holds, and will probably remain valid for one class of attacks;

> - ECMP and load-balancers try to make individual sessions follow a single path

...the universal utility of both of which is questionable: it might be nice to expose more information which would allow senders to indicate better partitions of their traffic than "per-flow fairness", or to indicate a future application/transport that has much better reordering tolerance to make ECMP trunking irrelevant, both of which would be potential applications for the type of think we're talking about in PLUS.

> It would be good for SPUD and QUIC ‎to provide enough information for monitoring devices to measure progress, handshakes, round-trip times, as can be done with TCP carrying TCP.

This opinion led to the initial work here; I think we should have a discussion about how opaque we can get while having everything still work, where the ideal is "as opaque as possible while retaining acceptable performance and operational manageability".

Cheers,

Brian

[1] Ding and Rabinovich, "TCP Stretch Acknowledgements and Timestamps: Findings and Implications for Passive RTT Measurement", ACM CCR July 2015

---

From: Tom Herbert <[email-masked]>
Date: Fri, 20 May 2016 08:33:36 -0700

> It would be good for SPUD and QUIC ‎to provide enough information for monitoring devices to measure progress, handshakes, round-trip times, as can be done with TCP carrying TCP.
Dave,

I agree that we need to consider the needs of network operators, but
we also have to consider the perspective of the users. For instance,
here at Facebook we operate on nearly every public network on the
planet except in those places where we are prohibited. That means we
need to deal with the policies and constraints across all network
operators-- that quickly leads to application of the robustness
principle in that we are going to be conservative in what we send to
achieve maximum reachability. So my question regarding exposure of L4
information in SPUD is what are the _requirements_ for what we send?

In order to establish that any information is a requirement, like that
in the list you proposed, I think we need at least three things:

1) A specification as to what the required information is. This should
include a description of how the network is allowed to use the
information. Also the information should be common across all
protocols and cannot be inferred by other means. For instance,
exposing sequence numbers may make sense for TCP, but other protocols
may not have a usable concept of sequence numbers or they may have
multiple sequence numbers like in sub-streams. The ECMP problem is
solved by IPv6 flow label which eliminating the need to otherwise
expose L4 information for that purpose. If the information is used as
part of maintaining flow state, like in tracking sequence numbers for
a connection, then there must be a consideration for when the state is
inaccurate or incomplete-- flow state tracking is inherently best
effort since the Internet is packet-switched not circuit-switched.
2) A description of the effects if the information is misused by some
node in the network, rather intentionally of not.  Misuse must not be
detrimental to the users. Unfortunately, I don't believe there's a lot
of latitude in this one. For instance, if we exposure of a piece of
information gives a performance benefit to 99% of our users but is
security risk for 1%, that puts sixteen million of our users at risk.
It's not a tradeoff we can easily make.
3) A description of the effects if end node fabricate the exposed
information. Hosts cannot be allowed to gain an unfair advantage
simply by lying about what is carried in the payload. This is the
"setting priority" problem on the Internet, where if end users are
allowed to indicate priority they will all say their packets are high
priority. Generally, there is no trust relationship established
between the end hosts and the network, so anything the network tells
the host and vice versa cannot be implicitly trusted to be true.

If the discussions in accord about exposing information beyond L4 are
a bellwether, I think getting consensus here is going to be difficult,
but this does seem like an important discussion to have in the context
of SPUD.

Tom