Subject: Re: [Udp35] Trying to learn about udp35

From: Jana Iyengar <[email-masked]>
Date: Tue, 24 Jun 2014 19:26:45 -0700

Brian, all,

I'm reviving this thread after a while -- apologies for responding so late.
Was there a kickoff call (or a doodle poll to set up the kickoff call?) I
may have missed it entirely, and apologies if I did.

Transport definitely needs architectural reimagining, and I am very happy
to see this conversation begin.

There's advise for future transport work that can come out of conversations
here, and this could influence the nature of work done at the IETF. In
terms of transport protocol work, TSV currently largely focusses on TCP
(RMCAT is a departure, and a good one; we finally have an app driving
design, however frustrating that may seem.) There may be some good from
thinking about general transport mechanisms -- Brian suggested
reordering-resiliency, and that is a great example of a transport problem
that could totally be chased in a general manner, independent of TCP. One
of the hard problems in the past, IMO, has been separating mechanisms from
actual implementations in protocols, and *maybe* one outcome of this
conversation is general guidelines that can help new transports not have to
translate everything out of TCP-ese.

Outside of just transport mechanisms, there's deployment issues that we've
faced in the past. We've still not solved that problem, but I think we may
be better equipped to attack that question now than we were in the past...
at any rate, I'm very glad to sit and discuss these and various other
relevant questions.

(FWIW: There's overlap with other areas that we cannot ignore -- for
instance, there's substantially more overlap of transport with security,
IMO, than we currently let ourselves think about at the IETF.)

Thanks for starting this effort, and I look forward to brainstorming. Do we
have a time yet? I still haven't booked my flight, and I'd like to make
sure I don't get in too late.
- jana

On Thu, May 22, 2014 at 7:24 AM, Michael Welzl <[email-masked]> wrote:

>
> On 22. mai 2014, at 14:09, Brian Trammell wrote:
>
> > hi Michael, all,
> >
> > (Crossing threads here, another reply to follow; in the meantime,
> inline...)
> >
> > On 22 May 2014, at 10:27, Michael Welzl <[email-masked]> wrote:
> >
> >> Hi,
> >>
> >> Thanks a lot for your response, I greatly appreciate that!  In line:
> >
> > <snip>
> >
> >>>> If this means having a WG-forming BoF in Toronto, then I guess it's
> about time to get this organized.
> >>>
> >>>
> >>> While it's clear that the TAPS charter as formulated addresses the
> same problem, and that most if not all of the various technical pieces
> remain the same, it's not clear to me personally that binding the solutions
> to these problems to a (single) IETF-defined API is the best way to ensure
> this work sees wide implementation and adoption. There are other paths:
> providing building blocks to API designers, just accepting the fact that
> people are already designing APIs and providing them guidance (especially
> on how not to break things, e.g. CC), doing one of these latter things and
> building an API on top of that, etc... and this is part of what we want to
> talk about in Toronto.
> >>
> >> So this is really the crux of the debate, and the reason why I said
> "let's please not rinse and repeat":
> >
> > I think what we have here is a somewhat misunderstood case of emphatic
> agreement. :)
> >
> >> - at the Bar BOF in Vancouver, we got feedback saying "this shouldn't
> be one API, in one place. There should only be services defined, and they
> can then be supported in various ways; anyway people aren't using the
> socket interface much, anymore"
> >
> > (This bit is true, and it is false. There aren't a lot of application
> programmers writing things to the sockets API anymore. But if you're a
> platform/library person writing an API implementation for app developers,
> and you want to use kernel TCP -- i.e. you don't own the kernel you're
> writing to -- you're still limited by it.  But this is a detail.)
> >
> >> - then, we incorporated this in the charter (
> https://sites.google.com/site/transportprotocolservices/charter-proposal-before-bof
> ), by writing that we'll specify services, and only *describe* at least one
> *example* API with one possible implementation.
> >>
> >> - because the London BoF was non-WG-forming, the charter wasn't
> presented; I think this led to some misunderstanding of Gorry's
> presentation, which showed *one possible implementation* of this - folks
> took it to mean that this is the only architecture we're proposing, we want
> to do one single API, and some ended up saying the same things that were
> already said at the Bar BOF.
> >
> > So here's where we run into the problem, and this is more about
> _framing_ than content, and something we identified as the reason the
> London BoF missed an opportunity to get more feedback. In my opinion even
> speaking of these things in terms of APIs is risky, because the effect of
> doing so within the IETF will be that people show up and debate the pros
> and cons of this engineering detail of this API, versus that engineering
> detail of that API, versus whether we should be in the API business at all,
> and the core work gets lost in the noise.
>
> Ok. I understand, and I agree.
>
>
> >> So now we're already sitting with this charter that, *before* London,
> already incorporated much of the feedback that we received in London again.
> And now you write "not sure that a (single) IETF-defined API is the best
> way ..."   :-)    => see. That shows me that my "square one" concern was at
> least not completely unfounded.
> >
> > At least as it reads to me, the charter on the transportservices site
> (this is the latest, yes)? reads as if the goal is to survey application
> APIs then specify a meta-API for all transport services APIs; given the
> derail in London, I'm afraid the example API will be an expensive
> distraction in the best case.
>
> Yes it's the latest; yes I see, and agree. We can remove it.
>
>
> >>>> I can imagine that we want to do more than what we have written into
> our charter, but what I'm suggesting is not to wait with piece A of the
> puzzle just because we in fact want pieces A, B and perhaps C too, and we
> have to still find out if/how to do pieces B and C. I'm having a hard time
> imagining that piece A just won't fit, whatever B and C may look like, so
> why not get A done?!
> >>>
> >>> The charter, especially its formulation of the problem statement, is a
> good starting point; to borrow your terminology, I'll try to follow up
> tomorrow or Friday with specific suggestions on how we might (lightly)
> tweak A to better fit aspects of the problem space we didn't really get
> into either in our premeeting or during the BoF in London.
> >>
> >> Great, thanks!
> >
> > (see comment below, more to follow...)
> >
> >>
> >>> I note that the first concrete deliverable we've identified is
> essentially item 1 on the proposed TAPS charter; this analysis work should
> absolutely continue regardless of progress in forming any future WG --
> indeed, formation of a working group would be likely to slow this work
> down, as opposed to speeding it up. Such an item could be published on the
> IAB stream if a WG isn't formed in the meantime.
> >>
> >> Okay... indeed that was my thinking - that at least some of the outputs
> in the TAPS charter should be done anyway, no matter what the bigger plans
> are.
> >
> > We absolutely need the first deliverable (Informational RFC summarizing
> the services provided by IETF transport protocols and congestion control
> mechanisms as well as the requirements of common APIs) soon, regardless of
> venue/stream. Personally I wouldn't limit it to the interfaces provided by
> common APIs, if only because (1) a lot of these APIs provide additional
> services above and beyond what a transport layer should and (2) being
> limited to the transports currently available, they may miss dimensions
> where innovation could be possible.
> >
> > A concrete example of (2): One of the comments from the routing ADs
> during the retreat is that a transport that was extremely
> reordering-tolerant (which to some extent requires either reordering or
> latency tolerance on the application's part) would enable new services
> based on maximizing intra-(micro-)flow path diversity -- think a
> "spread-spectrum Internet" to increase pervasive surveillance resistance in
> the core.
>
> Well.... so if you open up for "whatever applications want, based on use
> cases we can describe", you find yourself in an endless sea of
> possibilities. I agree that this is where things get really useful and
> exciting, but this is also where it's going to be hard to go ahead with
> people from one area, and harder to imagine a WG with a clear scope and
> end. THIS could be the IRTF bit IMO.
>
>
> > (Side note: the fact that this came from routing -- an area that appears
> at first glance to consider transport irrelevant, and certainly has a
> relatively small contributorship overlap with TSV working groups -- means
> that there's scope to get good feedback across areas that we'll miss if we
> do this as a TSV-area WG with mostly the usual suspects in the room. Which
> is one reason we started running with this as an IAB activity post-London.)
> >
> > I also think it's unwise to take "signaling improvement" out of scope
> for the eventual full effort (though it's entirely possible that it's
> necessary to separate this from the service-taxonomy and service
> recommendation work). I suspect that the reason to specifically exclude
> this and QoS is that those two taken together are AEON things.
>
> No, the reason was just to avoid ratholing into a signaling debate (but
> indeed AEON nicely complements all this). Signaling is also one of the
> points I'm least sure about, as it could be used to make the inevitable
> Happy Eyeballing more scalable.
>
>
> > But one of the real risks I see is that continued middlebox path
> impairment makes it harder and harder for anything other than straight TCP
> (or worse, straight HTTP) to work end-to-end, such that we'll need (1) a
> "NAT/middlebox-friendly" shim layer (which would itself have to be based on
> TCP or UDP) to support further innovation in transport plus (2) a way to do
> raw sockets in userspace without privilege, (which pretty much means UDP),
> in order to support explicitly non-TCP transports (i.e., what QUIC has
> done). Okay, not really signaling, but layering. And again, this might be a
> separate effort under this general umbrella, coordinated with TAPS, as
> opposed to something TAPS itself must address.
>
> Yes, separate, and it also sounds like you're making this too complicated,
> frankly (I think best effort + happy eyeballing + fall-backs can go a long
> way). Anyway, separate please.
>
>
> >> As for a WG slowing things down, I wasn't aware of that being possible
> but that's just me not knowing as much about IETF internals as (all of) you.
> >
> > A better way to put this might be that it's better to bring a fully
> formed idea into a WG to flesh out the ideas and engineering details than
> it is to do green-field engineering from first principles in a new WG.
> While TAPS is much closer to the former than the latter at this point, I
> think there's still enough disgreement/confusion about the scope of all
> this post-London that I think it would be very useful for a smaller group
> to sit down and draw boxes around the uncertainties
> >
> > Hence the meeting in Toronto. :)
> >
> > On which: if there's anyone who can't make Saturday afternoon/evening,
> we can try to reschedule, but we'd strongly prefer to be able to do this
> before the official festivities start.
> >
> > Cheers,
> >
> > Brian
>
>
>
> Cheers,
> Michael
>
>

---

From: Brian Trammell <[email-masked]>
Date: Wed, 25 Jun 2014 08:59:17 +0200

hi Jana, all,

On 25 Jun 2014, at 04:26, Jana Iyengar <[email-masked]> wrote:

> Brian, all,
> 
> I'm reviving this thread after a while -- apologies for responding so late. Was there a kickoff call (or a doodle poll to set up the kickoff call?) I may have missed it entirely, and apologies if I did.

We had a call on 2 June. Lots of the focus since them has been on chartering TAPS, as the identifying-dimensions-of-transport work had already started there, and will continue there as well.

> Transport definitely needs architectural reimagining, and I am very happy to see this conversation begin. 
> 
> There's advise for future transport work that can come out of conversations here, and this could influence the nature of work done at the IETF. In terms of transport protocol work, TSV currently largely focusses on TCP (RMCAT is a departure, and a good one; we finally have an app driving design, however frustrating that may seem.) There may be some good from thinking about general transport mechanisms -- Brian suggested reordering-resiliency, and that is a great example of a transport problem that could totally be chased in a general manner, independent of TCP. One of the hard problems in the past, IMO, has been separating mechanisms from actual implementations in protocols, and *maybe* one outcome of this conversation is general guidelines that can help new transports not have to translate everything out of TCP-ese.

This is our intention, yes... We've talked in terms of "mix-ins" -- guideline documents / source code / whatever that allow new transports to be constructed in userspace without having to completely re-invent the wheel every time.

> Outside of just transport mechanisms, there's deployment issues that we've faced in the past. We've still not solved that problem, but I think we may be better equipped to attack that question now than we were in the past... at any rate, I'm very glad to sit and discuss these and various other relevant questions.

One approach to _this_ problem would be a sub-(userspace-)transport session layer, which would allow middleboxes to interact with the common semantics of these layer-4(.5) transports (mainly here we're talking about firewalls applying policy to the first packet of a flow) without breaking the transport-specific semantics.

Hence "udp35" :)

> (FWIW: There's overlap with other areas that we cannot ignore -- for instance, there's substantially more overlap of transport with security, IMO, than we currently let ourselves think about at the IETF.)

+many. 

One thing that came up in Cancun is there's also more overlap with routing than we let ourselves think. (The routing ADs discussed route-diversity-based approaches to reducing pervasive surveillance exposure at the cable-landing-site surface in the topology, but assumed that wouldn't fly based on assumptions about how much reordering tolerance the transport layer has for each (micro-)flow)

> Thanks for starting this effort, and I look forward to brainstorming. Do we have a time yet? I still haven't booked my flight, and I'd like to make sure I don't get in too late.

We're planning to meet from 17:00 EDT on Saturday. See you there!

Cheers,

Brian

> > (Crossing threads here, another reply to follow; in the meantime, inline...)
> >
> >>>> If this means having a WG-forming BoF in Toronto, then I guess it's about time to get this organized.
> >>>
> >>>
> >>> While it's clear that the TAPS charter as formulated addresses the same problem, and that most if not all of the various technical pieces remain the same, it's not clear to me personally that binding the solutions to these problems to a (single) IETF-defined API is the best way to ensure this work sees wide implementation and adoption. There are other paths: providing building blocks to API designers, just accepting the fact that people are already designing APIs and providing them guidance (especially on how not to break things, e.g. CC), doing one of these latter things and building an API on top of that, etc... and this is part of what we want to talk about in Toronto.
> >>
> >> So this is really the crux of the debate, and the reason why I said "let's please not rinse and repeat":
> >
> > I think what we have here is a somewhat misunderstood case of emphatic agreement. :)
> >
> >> - at the Bar BOF in Vancouver, we got feedback saying "this shouldn't be one API, in one place. There should only be services defined, and they can then be supported in various ways; anyway people aren't using the socket interface much, anymore"
> >
> > (This bit is true, and it is false. There aren't a lot of application programmers writing things to the sockets API anymore. But if you're a platform/library person writing an API implementation for app developers, and you want to use kernel TCP -- i.e. you don't own the kernel you're writing to -- you're still limited by it.  But this is a detail.)
> >
> >> - then, we incorporated this in the charter ( https://sites.google.com/site/transportprotocolservices/charter-proposal-before-bof ), by writing that we'll specify services, and only *describe* at least one *example* API with one possible implementation.
> >>
> >> - because the London BoF was non-WG-forming, the charter wasn't presented; I think this led to some misunderstanding of Gorry's presentation, which showed *one possible implementation* of this - folks took it to mean that this is the only architecture we're proposing, we want to do one single API, and some ended up saying the same things that were already said at the Bar BOF.
> >
> > So here's where we run into the problem, and this is more about _framing_ than content, and something we identified as the reason the London BoF missed an opportunity to get more feedback. In my opinion even speaking of these things in terms of APIs is risky, because the effect of doing so within the IETF will be that people show up and debate the pros and cons of this engineering detail of this API, versus that engineering detail of that API, versus whether we should be in the API business at all, and the core work gets lost in the noise.
> 
> >> So now we're already sitting with this charter that, *before* London, already incorporated much of the feedback that we received in London again. And now you write "not sure that a (single) IETF-defined API is the best way ..."   :-)    => see. That shows me that my "square one" concern was at least not completely unfounded.
> >
> > At least as it reads to me, the charter on the transportservices site (this is the latest, yes)? reads as if the goal is to survey application APIs then specify a meta-API for all transport services APIs; given the derail in London, I'm afraid the example API will be an expensive distraction in the best case.
> 
> >>>> I can imagine that we want to do more than what we have written into our charter, but what I'm suggesting is not to wait with piece A of the puzzle just because we in fact want pieces A, B and perhaps C too, and we have to still find out if/how to do pieces B and C. I'm having a hard time imagining that piece A just won't fit, whatever B and C may look like, so why not get A done?!
> >>>
> >>> The charter, especially its formulation of the problem statement, is a good starting point; to borrow your terminology, I'll try to follow up tomorrow or Friday with specific suggestions on how we might (lightly) tweak A to better fit aspects of the problem space we didn't really get into either in our premeeting or during the BoF in London.
> >>
> >> Great, thanks!
> >
> >>> I note that the first concrete deliverable we've identified is essentially item 1 on the proposed TAPS charter; this analysis work should absolutely continue regardless of progress in forming any future WG -- indeed, formation of a working group would be likely to slow this work down, as opposed to speeding it up. Such an item could be published on the IAB stream if a WG isn't formed in the meantime.
> >>
> >> Okay... indeed that was my thinking - that at least some of the outputs in the TAPS charter should be done anyway, no matter what the bigger plans are.
> >
> > We absolutely need the first deliverable (Informational RFC summarizing the services provided by IETF transport protocols and congestion control mechanisms as well as the requirements of common APIs) soon, regardless of venue/stream. Personally I wouldn't limit it to the interfaces provided by common APIs, if only because (1) a lot of these APIs provide additional services above and beyond what a transport layer should and (2) being limited to the transports currently available, they may miss dimensions where innovation could be possible.
> >
> > A concrete example of (2): One of the comments from the routing ADs during the retreat is that a transport that was extremely reordering-tolerant (which to some extent requires either reordering or latency tolerance on the application's part) would enable new services based on maximizing intra-(micro-)flow path diversity -- think a "spread-spectrum Internet" to increase pervasive surveillance resistance in the core.
> 
> Well.... so if you open up for "whatever applications want, based on use cases we can describe", you find yourself in an endless sea of possibilities. I agree that this is where things get really useful and exciting, but this is also where it's going to be hard to go ahead with people from one area, and harder to imagine a WG with a clear scope and end. THIS could be the IRTF bit IMO.
> 
> 
> > (Side note: the fact that this came from routing -- an area that appears at first glance to consider transport irrelevant, and certainly has a relatively small contributorship overlap with TSV working groups -- means that there's scope to get good feedback across areas that we'll miss if we do this as a TSV-area WG with mostly the usual suspects in the room. Which is one reason we started running with this as an IAB activity post-London.)
> >
> > I also think it's unwise to take "signaling improvement" out of scope for the eventual full effort (though it's entirely possible that it's necessary to separate this from the service-taxonomy and service recommendation work). I suspect that the reason to specifically exclude this and QoS is that those two taken together are AEON things.
> 
> No, the reason was just to avoid ratholing into a signaling debate (but indeed AEON nicely complements all this). Signaling is also one of the points I'm least sure about, as it could be used to make the inevitable Happy Eyeballing more scalable.
> 
> 
> > But one of the real risks I see is that continued middlebox path impairment makes it harder and harder for anything other than straight TCP (or worse, straight HTTP) to work end-to-end, such that we'll need (1) a "NAT/middlebox-friendly" shim layer (which would itself have to be based on TCP or UDP) to support further innovation in transport plus (2) a way to do raw sockets in userspace without privilege, (which pretty much means UDP), in order to support explicitly non-TCP transports (i.e., what QUIC has done). Okay, not really signaling, but layering. And again, this might be a separate effort under this general umbrella, coordinated with TAPS, as opposed to something TAPS itself must address.
> 
> Yes, separate, and it also sounds like you're making this too complicated, frankly (I think best effort + happy eyeballing + fall-backs can go a long way). Anyway, separate please.
> 
> 
> >> As for a WG slowing things down, I wasn't aware of that being possible but that's just me not knowing as much about IETF internals as (all of) you.
> >
> > A better way to put this might be that it's better to bring a fully formed idea into a WG to flesh out the ideas and engineering details than it is to do green-field engineering from first principles in a new WG. While TAPS is much closer to the former than the latter at this point, I think there's still enough disgreement/confusion about the scope of all this post-London that I think it would be very useful for a smaller group to sit down and draw boxes around the uncertainties
> >
> > On which: if there's anyone who can't make Saturday afternoon/evening, we can try to reschedule, but we'd strongly prefer to be able to do this before the official festivities start.
> >
> > Cheers,
> >
> > Brian
> 
> 
> 
> Cheers,
> Michael
> 
>

---

From: [email-masked]
Date: Wed, 25 Jun 2014 11:03:40 +0100

Hi,

> design, however frustrating that may seem.)
>
Why do you say this? I don't think this is the case, TSV has been working
on UDP, SCTP, multicast methods, various NAT-related things, etc.It's far
from being TCP, but that may well be a perception held by people outside
the area.

>  There may be some good from
> conversation is general guidelines that can help new transports not have
> to translate everything out of TCP-ese.
>
Could be really useful - because there's also work outside of TSV which
uses UDP, but then tries (in some cases partially) to reinvent existing
transport methods.

> IMO, than we currently let ourselves think about at the IETF.)
Indeed.

> _______________________________________________
> Udp35 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/udp35
>

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 25 Jun 2014 09:02:27 -0700

Ah -- drat. Sorry about missing the call. I'll plan to be there on Saturday.

> One approach to _this_ problem would be a sub-(userspace-)transport
> session layer, which would allow middleboxes to interact with the common
> semantics of these layer-4(.5) transports (mainly here we're talking about
> firewalls applying policy to the first packet of a flow) without breaking
> the transport-specific semantics.
>
> Hence "udp35" :)
>

Minion, as you probably know, was built with exactly that as a goal. I
don't think it solves the entire middlebox issue (I don't think there's any
one solution that can), but it certainly takes us towards it.
I still don't understand "udp35". What is 35?

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 25 Jun 2014 09:12:13 -0700

Gorry,

> from being TCP, but that may well be a perception held by people outside

I said TSV *currently* *largely* focusses on TCP. And I said in terms of
transport protocol work... I may have been playing loose with words.
I agree that there's plenty that TSV does (AQM!) that's important and not
TCP-work.

That said, my point was that TCP dominates our imagination. We've done
SCTP, but there's no SCTPM or DCCPM or RMTM groups equivalent to TCPM. And
traffic on the Internet largely remains TCP. (I don't mean to rathole --
I'm happy to take this offline, or continue this conversation in Toronto.)

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 25 Jun 2014 09:17:17 -0700

Also, FWIW, here's a draft that Bryan Ford and I wrote up a while ago,
for the first MPTCP wg meeting in Stockholm. The intent was to provide
an architectural basis for the work in the wg, and some of it made it
into the MPTCP Architecture RFC. Perhaps there's room for this
conversation here. I was a bit disappointed that more of it didn't
make it into the MPTCP work.

See: http://tools.ietf.org/html/draft-iyengar-ford-tng-00.

Abstract: While there is substantial community interest in
next-generation multipath-capable Internet transports, evolutionary
pressures have gradually eroded the simplicity of the Internet's
original transport architecture to a point where it is no longer
realistically applicable to new tranports.  This document proposes a
new architectural framework for next-generation multipath-capable
transport protocols, focusing immediately on multipath TCP but taking
care to allow for generalization to other multipath-capable
transports.  The architecture places emphasis on enabling new
multipath features in a safe, TCP-friendly, and backward-compatible
fashion, retaining full interoperability with both existing
applications and existing network infrastructure, and enabling reuse
of existing protocols as much as possible while providing incremental
deployment paths to new, more powerful and/or more efficient
protocols.  The architecture re-establishes the long-lost principles
of end-to-end reliability and fate sharing, in the presence of
existing and future network middleboxes, and enables the deployment of
transport-neutral end-to-end protection without interfering with these
policy-enforcing or performance-enhancing middleboxes.  This document
describes architecture goals, a layering model supporting these goals,
abstract properties of the interfaces between the architecture's new
layers, general approaches to multipath congestion control and how
they fit into the architecture, realistic protocol design and
incremental deployment paths, and ways in which this document
complements and relates to ongoing protocol design activities in the
IETF.

- jana

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 25 Jun 2014 18:24:58 +0100

> I'm happy to take this offline, or continue this conversation in Toronto.)
OK, in that case I can also violently agree: TCP-ish behavior is assumed 
(whatever that may mean to people) and is used as a basis for many of 
our recommendations....

I'd of course buy you a drink in Toronto... to continue on how to change 
perceptions and understanding of what needs to be done:-)

Gorry