Subject: Re: [Taps] New Version Notification for draft-trammell-post-sockets-00.txt

From: Michael Welzl <[email-masked]>
Date: Wed, 9 Nov 2016 10:59:43 +0100

Hi,

Thanks for posting a really interesting draft!

Similar to the other (also interesting) draft that was just posted ( draft-mcquistin-taps-low-latency-services-00.txt ), this reads quite a bit like a "wish list".
I'm not saying that's a bad thing, I think that both drafts contain an interesting and very reasonable wish list  (with quite some overlap, 1) with each other, 2) with what current protocols provide, as we can see from draft-gjessing-taps-minset-03).
Both also identify at least one thing that seems missing in currently standardized transports: being able to define dependencies between messages.
Interesting!

However, given that current protocols don't do all these things, what's the context here for TAPS? Should the charter be extended to allow for such "wish lists" for future protocols?
(Personally, I'd say yes)

I mean, draft-mcquistin-taps-low-latency-services-00.txt even explicitly says "The next phase, enabled both by the use of transport services and their separation from transport protocols, is to define novel transport services based on the needs of applications."
I'd say this is clearly *not* what the charter says. When TAPS started, lots of folks wanted this, but TAPS would have never seen the light of day if we had put that in the charter. Now maybe it's time to be more forward-looking?

BTW, something that made me particularly curious is, in the draft announced below, the following statement:
"It must be possible to implement Post over vanilla TCP in the present Internet architecture."
- how? To start with, you require message delineation. What's the idea here?

Cheers,
Michael

> On 27 Oct 2016, at 14:52, Brian Trammell <[email-masked]> wrote:
> 
> Greetings, TAPS,
> 
> We've submitted an initial version of our Post Sockets abstract interface draft, draft-trammell-post-sockets. It's not yet clear that we'd want this to be adopted by the TAPS working group for a future milestone in partial fulfillment of point 3 on the charter -- perhaps the ideas here are merely an illustration of what is possible in the space, that could point toward a more concrete definition of an abstract interface for TAPS.  We'd like to discuss this point in Seoul.
> 
> Thanks, cheers,
> 
> Brian
> 
>> Begin forwarded message:
>> 
>> From: [email-masked]
>> Subject: New Version Notification for draft-trammell-post-sockets-00.txt
>> Date: 27 October 2016 at 11:57:39 GMT+2
>> To: "Mirja Kuehlewind" <[email-masked]>, "Tommy Pauly" <[email-masked]>, "Brian Trammell" <[email-masked]>, "Colin Perkins" <[email-masked]>
>> 
>> 
>> A new version of I-D, draft-trammell-post-sockets-00.txt
>> has been successfully submitted by Brian Trammell and posted to the
>> IETF repository.
>> 
>> Name:		draft-trammell-post-sockets
>> Revision:	00
>> Title:		Post Sockets, An Abstract Programming Interface for the Transport Layer
>> Document date:	2016-10-27
>> Group:		Individual Submission
>> Pages:		17
>> URL:            https://www.ietf.org/internet-drafts/draft-trammell-post-sockets-00.txt
>> Status:         https://datatracker.ietf.org/doc/draft-trammell-post-sockets/
>> Htmlized:       https://tools.ietf.org/html/draft-trammell-post-sockets-00
>> 
>> 
>> Abstract:
>>  This document describes Post Sockets, an asynchronous abstract
>>  programming interface for the atomic transmission of objects in an
>>  explicitly multipath environment.  Post replaces connections with
>>  long-lived associations between endpoints, with the possibility to
>>  cache cryptographic state in order to reduce amortized connection
>>  latency.  We present this abstract interface as an illustration of
>>  what is possible with present developments in transport protocols
>>  when freed from the strictures of the current sockets API.
>> 
>> 
>> 
>> 
>> Please note that it may take a couple of minutes from the time of submission
>> until the htmlized version and diff are available at tools.ietf.org.
>> 
>> The IETF Secretariat
>> 
> 
> _______________________________________________
> Taps mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/taps

---

From: Brian Trammell <[email-masked]>
Date: Wed, 9 Nov 2016 12:30:36 +0100

hi Michael,

> On 09 Nov 2016, at 10:59, Michael Welzl <[email-masked]> wrote:
> 
> Hi,
> 

To some extent, that's the point: if we started over, what would we want the interface to look like.

> Both also identify at least one thing that seems missing in currently standardized transports: being able to define dependencies between messages.

...and shamelessly lifted from Minion.

> (Personally, I'd say yes)

So, there are two ways I see to do this:

(1) Everything here except message delineation *works* without any specific transport protocol support, it just works better if you have transport protocols that support the contracts. Note that receive() doesn't expose any of the object properties that send() takes, those properties are only used for purposes of ordering, scheduling, and path selection at the sender. So if you have a transport protocol that doesn't support partial reliability, for instance, the lifetimes could be silently ignored. The TAPS transport selection layer could give the application access to information about which contracts of the API are honored and which are not, but an application that didn't care could be written once and still work on the worst fallback, if with less optimum performance.

(2) Alternately, a Post implementation more tightly integrated with a TAPS one could raise errors on attempts to use contracts not supported by the underlying transport. This requires more application logic support, though, and negates one of the advantages we see of this approach: write once, use any transport.

> I'd say this is clearly *not* what the charter says. When TAPS started, lots of folks wanted this, but TAPS would have never seen the light of day if we had put that in the charter. Now maybe it's…

Post doesn't say this *explicitly*, but I think you've noticed that it's kind of assumed. Yes, I do think it's time to be forward-looking. Where that looking forward happens is a different question: it could happen in TAPS, or in its own WG or RG. In any case, it definitely needs to happen hand in hand with TAPS, as it's an effort that can provide more diversity in ways to implement services and features.

(reading from my slides which I should send Aaron and Zahed shortly): At least with respect to Post, we're just looking to start that conversation now, to see how many people are interested in this next step, and to get some feedback on what's good about what we're thinking, and what's not so good.

> - how? To start with, you require message delineation. What's the idea here?

Ah, now you've found the older draft buried in the newer draft. :)

That was an original hard requirement of the wishlist, but as we went down the rabbithole of ordering our wishes the object framing (from SEQPACKET) really stood out as a missing feature that makes sense to have, and it's very hard to have a common API for a send() with asynchronous recv() that doesn't just turn into a read/write loop unless the message framing is there.

Again, two approaches I can see to make this happen, neither of which is very good IMO:

(1) Define a framing layer (or choose an existing one; for maximum layer confusion, how about websockets? ;) ) to run over vanilla TCP; in this case, Post is less of an API than a flexible transport layer, since it can't be used to connect to endpoints not running Post.

(2) When running on vanilla TCP, only open_stream() works on an association. Same drawbacks as in (2) above.

Maybe try 1 with a fallback to 2, but then you'd need a negotiation protocol running at both ends, rather than a sensing layer that can work at each endpoint independently as in TAPS.

Thanks, cheers,

Brian

---

From: Michael Welzl <[email-masked]>
Date: Wed, 9 Nov 2016 13:19:32 +0100

> On 09 Nov 2016, at 12:30, Brian Trammell <[email-masked]> wrote:
> 
> hi Michael,
> 

Well - this seems to be a pretty general requirement, coming from HTTP/2 as well...

> (1) Everything here except message delineation *works* without any specific transport protocol support,

well and (as Minion identified) it all begins and ends with message delineation really - you're not able to offer much of what this draft describes unless you have message delineation.

> it just works better if you have transport protocols that support the contracts. Note that receive() doesn't expose any of the object properties that send() takes, those properties are only used for purposes of ordering, scheduling, and path selection at the sender. So if you have a transport protocol that doesn't support partial reliability, for instance, the lifetimes could be silently ignored. The TAPS transport selection layer could give the application access to information about which contracts of the API are honored and which are not, but an application that didn't care could be written once and still work on the worst fallback, if with less optimum performance.

That seems good to me.

> (2) Alternately, a Post implementation more tightly integrated with a TAPS one could raise errors on attempts to use contracts not supported by the underlying transport. This requires more applicat…

The approach we ended up taking in NEAT (to be presented) is to always choose between "required" or "desired".
"Required" means the API fails and gets back to you; "desired" means you get a fall-back as you describe here. This yields the flexibility that TAPS wants to achieve, yet allows applications to be in control of the things they really must control.

> Post doesn't say this *explicitly*, but I think you've noticed that it's kind of assumed. Yes, I do think it's time to be forward-looking. Where that looking forward happens is a different question…

So here's a question. To implement e.g. Post with TCP, one could also provide a layer on top of TCP that offers the suggested API and at least takes care of message delineation (and then that layer could use other protocols instead of TCP too). This would need to be there on both ends of a connection.
I mean, AFAIK, as Minion developed, it was at some point suggested to not insert a delimiter in TCP as part of TCP's functionality, but to simply use delimiters coming from app-layer protocols. I.e., when that protocol has a delimiter, why use another one?

If that's what we need, then all of a sudden, we're not discussing a forward-looking wish-list anymore, but a way to implement a TAPS system. I'd think that would be a better direction than just creating a wish list for new protocols... but both could be done, I guess.

> That was an original hard requirement of the wishlist, but as we went down the rabbithole of ordering our wishes the object framing (from SEQPACKET) really stood out as a missing feature that makes…

Right.

> Maybe try 1 with a fallback to 2, but then you'd need a negotiation protocol running at both ends, rather than a sensing layer that can work at each endpoint independently as in TAPS.

Sorry for inventing the same as you in my answer above  :-)   it seems we completely agree here. I also agree that both (1) and (2) would be possible solutions.
Interesting idea about the fallback - indeed, wouldn't trying (1) with a fallback to (2) be ideal?!

Cheers,
Michael

---

From: Brian Trammell <[email-masked]>
Date: Wed, 9 Nov 2016 16:15:08 +0100

hi Michael,

(snipping some history...)

> "Required" means the API fails and gets back to you; "desired" means you get a fall-back as you describe here. This yields the flexibility that TAPS wants to achieve, yet allows applications to be…

That's a good distinction to make. This, of course, doesn't solve the problem of needing a framing to make this work, as noted below.

> If that's what we need, then all of a sudden, we're not discussing a forward-looking wish-list anymore, but a way to implement a TAPS system. I'd think that would be a better direction than just cr…

So I'm not sure I understand what you're suggesting here... are you saying that there'd be some application-layer callback to determine where the framing is on the receive side? Yeah, that might work. (If that's not what you're saying... well, it might work anyway :) )

> Interesting idea about the fallback - indeed, wouldn't trying (1) with a fallback to (2) be ideal?!

Yes, except that there the application has to know about the fallback, and has to work with streams, which changes its logic. And it'd be nice if we could avoid that. But if there were a way to piggyback on application-layer framing, that might make this less awkward.

I think we're on to something. In any case, it definitely answers the first question we had (do we want to talk about this further in a TAPS context?)

Cheers,

Brian

---

From: Michael Welzl <[email-masked]>
Date: Wed, 9 Nov 2016 21:07:12 +0100

Hi,

Snipping too...

> So I'm not sure I understand what you're suggesting here... are you saying that there'd be some application-layer callback to determine where the framing is on the receive side? Yeah, that might wo…

Yes that’s what I meant

> Yes, except that there the application has to know about the fallback, and has to work with streams, which changes its logic.

Yes, but no way around that unless you know what’s going on on the receiver side.

> And it'd be nice if we could avoid that. But if there were a way to piggyback on application-layer framing, that might make this less awkward.

I would have thought the opposite? I don’t know what an API that uses app-layer framing looks like (I guess Jana worked on that while at Apple?), but I suspect it’s more awkward than just having delimiters available from below?  Somehow the transport, or shim layer above it, on both sides, needs to be able to identify the delimiter, and so the format must be known to it. This means that the application would have to tell that layer the format of its app-layer delimiter - rather than just having a post-like interface that uses messages or objects, and perhaps getting a failure / fallback, which - I think - is annoying but inevitable.

So, considering that delimiters are the key missing piece here, maybe TAPS should define how such a shim over TCP should work, with delimiters and signaling to be able to fall back if the other side doesn’t have it?

> I think we're on to something. In any case, it definitely answers the first question we had (do we want to talk about this further in a TAPS context?)

Yes… I agree, I think we’re on to something here.

Thanks again, actually to all authors of both these drafts, for helping make TAPS more interesting!

Cheers,
Michael