Subject: [Taps] About the draft-ietf-taps-transports-usage rules

From: Michael Welzl <[email-masked]>
Date: Mon, 6 Jun 2016 10:16:50 +0200

Hi all,

Following up on this email, here's an extra thought.

First, a quick summary of the long text below:
- the goal is to enable automation
- to make good use of protocols other than TPC or UDP *some* things must be exposed, or else they can never be used (e.g. unordered message delivery and partial reliability). These are the ones that we call "functional" in draft-gjessing-taps-minset.
- however, eventually a TAPS system should be able to hide as much as possible (just not these "functional" things); therefore, including primitives that are not covered in RFCs may be ok to do but is also maybe not super important, depending on whether the primitives are "functional" or not (and most of the ones that exist in implementations but not in RFCs probably aren't).

Second, the extra thought. With the rules I laid out for draft-ietf-taps-transports-usage, I exclude Experimental RFCs and primitives that MAY be implemented by a protocol.
This removes MPTCP completely, which is probably just plain stupid. I want to be strict for good reasons, but I don't want to be just plain stupid  :)
Also, TFO requires a functional API change, yet it's Experimental...  a TAPS system can never make use of TFO unless its API change is supported.  That also seems just plain stupid to me.

=> I think we should allow these things, but put them in an extra section or an appendix (just like I said below about stuff coming from code but not RFCs).
One note: For the protocols that will be included in it, draft-ietf-taps-transports-usage tries to cover *all* the SHOULD / MUST primitives from standards-track RFCs. I definitely don't want to make a rule that requires covering ALL the Experimental / MAY implement   ... etc. primtives  - but it should be allowed to include them in the draft.

Thoughts?

Cheers,
Michael

> On 27 Apr 2016, at 10:16, Michael Welzl <[email-masked]> wrote:
> 
> Hi all,
> 
> This is mainly about criticism from Toerless. I thought of sending him a private email (after I wanted to have a chat with him in Buenos Aires but ran out of time), but thinking some more, I decided that this is probably a matter of general interest: it's about the bigger picture of TAPS.
> 
> Toerless has repeatedly criticized the use of RFCs as a basis for developing the API in TAPS, when we have newer, better APIs available.
> 
> 
> ***side note***
> 
> Before continuing, let me pause and diverge a bit: much of Toerless' criticism was related to the UDP API, and, as far as I understand, multicast in particular (which isn't the focus of TAPS). Some of his concrete points may be absolutely right and, not knowing enough, I definitely don't take issue with them. Speaking for my own documents (drafts I'm co-authoring), I also do not want to be *religious* about using only the RFCs as a basis when it simply doesn't make sense - however I would like to use them as much as possible and consider diverging from my rules an exception, meaning that text should be marked or even put in an appendix. The reason for being so attached to the RFCs is that OSes have different concrete APIs, for good reason, and I think we want to maintain the freedom developers have in implementing their APIs as they wish.
> 
> 
> 
> ***main point of this email***
> 
> Thinking about Toerless' general point of using more modern APIs made me think of the bigger picture again. For example, one cool recent feature in TCP APIs is the SO_SNDLOWAT socket option, which allows better control of the sender buffer. Not including it in an API document "sucks" and it IS nice to include it in a description of an API.
> 
> It is, however, also pretty irrelevant to the point of TAPS. Anyone is free to implement the SO_SNDLOWAT socket option today, we don't need TAPS for that. Neither do we need to change TCP for that. It's not even necessary to describe it for TCP to be able to use it (won't hurt but also won't gain us much).
> 
> I started the TAPS effort to ENABLE AUTOMATION.
> 
> This actually means *hiding* as much as possible. Today's stacks are highly optimized and a lot of automation happens - but some things are IMPOSSIBLE to automatize unless they are exposed in an API and an application uses them. SCTP contains some obvious examples: unordered delivery of messages and partial reliability. Try doing this to an application that expects a reliable byte stream. So, no system even tries to automatically use SCTP underneath the socket layer because many of its benefits could never be utilized.
> 
> This is why we need to do something. This is why TAPS exists - to specify that an API really should contain these transport service features (as we call them), so that more (richer) automation becomes possible. That's the whole point here. These are the transport service features that, in draft-gjessing-taps-minset, we call "functional".
> 
> At the last meeting, we authors of draft-gjessing-taps-minset got the feedback that we shouldn't put so much effort into trying to minimize the number of exposed services. I agree with this feedback, but here nevertheless I explain WHY we tried to minimize the number of services and why the charter says "the subset of those Transport Services, as identified in item 1, that end systems supporting TAPS will provide" - because, to me (and in line with the charter, I claim), the REAL goal of TAPS is to identify all these functional transport service features and explain how a TAPS system providing them could be implemented. I agree it makes sense to document more but I want to stress that that's not the main point. In other words, I think if we'd end up with a list of ONLY the functional transport service features of all the protocols we consider, and then explain how they can be provided, then TAPS has reached its goal: it would enable benefiting from other-than-TCP-or-UDP protocols.
> 
> It's my impression that discussions in TAPS are perhaps too much fueled by thinking about what modern APIs can do (as opposed to the old RFCs). I think there's no harm in that and it can be useful, but the REAL value comes from something else and we shouldn't lose focus on what really matters here. In other words, I believe that Toerless' criticism may be about things that aren't really very relevant to the goal of TAPS (because implementing them is not related to increasing the ability to automatize usage of mechanisms provided by protocols other than TCP and UDP - just like SO_SNDLOWAT can be implemented today, using TCP, and doesn't change anything about the intended automation). Note that I say may, because perhaps I AM missing the point as I didn't understand enough about the UDP API things he talked about, I do apologize if this is all just me misunderstanding him!
> 
> Thoughts?
> (I am a little worried that nothing that I wrote is new to people, everyone agrees with me, and this is just me not getting Toerless' points   :(   )
> 
> Cheers,
> Michael
> 
> _______________________________________________
> Taps mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/taps

---

From: Michael Welzl <[email-masked]>
Date: Mon, 6 Jun 2016 10:17:30 +0200

> On 27 Apr 2016, at 23:25, Joe Touch <[email-masked]> wrote:
> 
> 
> 
> On 4/27/2016 1:16 AM, Michael Welzl wrote:
>> Thinking about Toerless' general point of using more modern APIs made
>> me think of the bigger picture again. For example, one cool recent
>> feature in TCP APIs is the SO_SNDLOWAT socket option, which allows
>> better control of the sender buffer. Not including it in an API
>> document "sucks" and it IS nice to include it in a description of an
>> API.
> 
> This is an interesting example. It affects how a user-level process
> interacts with the protocol implementation inside the OS but has NOTHING
> to do with TCP.
> 
> I'm not sure where I would put that. It clearly isn't part of the TCP
> API. Nor, e.g., would be the "nice" level of the process that's feeding
> data to TCP -- yet both could affect performance between two applications.
> 
> It seems like there's a gap here you could drive a truck through that
> needs to be addressed. I'm not sure where, though - i.e., whether this
> is part of TAPS or not.

Not part of TAPS I think, because it's not related to automatizing the use of protocols other than TCP or UDP.

Cheers,
Michael

---

From: Joe Touch <[email-masked]>
Date: Mon, 6 Jun 2016 10:15:11 -0700

> Not part of TAPS I think, because it's not related to automatizing the use of protocols other than TCP or UDP.

Please explain why you think that's true.

AFAICT, it's fundamental to how any protocol interacts with the OS.

Joe

---

From: Michael Welzl <[email-masked]>
Date: Mon, 6 Jun 2016 20:20:38 +0200

> On 6. jun. 2016, at 19.15, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Yes it is, but it’s also orthogonal to the choice of protocol - as you said, it really has nothing to do with TCP.
Using my typical example again: you can’t use an SCTP mechanism like e.g. unordered message delivery without knowing that the application wants it, or else you’ll break the semantics of the application data stream.
It's is a mechanism of SCTP that simply cannot be used unless it’s chosen via an API.

Can you say the same about this SO_SNDLOWAT stuff?  As far as I’m concerned such send buffer management is an implementation detail - you could implement this very well or extremely badly for every protocol - but even an implementation that gives you almost no choice of what goes on in the buffer does not prohibit you from using a mechanism that a protocol provides. It’s a different matter, because the semantics stay unaltered: if you set the buffer very small with SO_SNDLOWAT, TCP will just fail on you. It’s not going to lose any packets, it stays reliable. If you could make it lose packets by changing SO_SNDLOWAT, that would be a different matter I think...

Cheers,
Michael

---

From: Joe Touch <[email-masked]>
Date: Mon, 6 Jun 2016 11:34:35 -0700

> Yes it is, but it’s also orthogonal to the choice of protocol - as you said, it really has nothing to do with TCP.
Yes, but configuration of the OS interface isn't orthogonal to that choice.

I.e., constraints of the OS interface may play into the decision of
which transport to use, and which transport is chosen may affect the
configuration of that OS interface.

> Can you say the same about this SO_SNDLOWAT stuff?  As far as I’m concerned such send buffer management is an implementation detail - you could implement this very well or extremely badly for every protocol

The way the OS relays  information to the transport IS a protocol - just
not one that involves communication on the wire. It significantly
affects the way in which the transport operates.
>  - but even an implementation that gives you almost no choice of what goes on in the buffer does not prohibit you from using a mechanism that a protocol provides. 
Sure, but if you're trying to set TCP to very low latency and use a huge
socket buffer, you could easily be creating huge delays and jitter in
the interaction between the process scheduling to fill the buffer and
the transfer of info into TCP.
> It’s a different matter, because the semantics stay unaltered: if you set the buffer very small with SO_SNDLOWAT, TCP will just fail on you. It’s not going to lose any packets, it stays reliable. If you could make it lose packets by changing SO_SNDLOWAT, that would be a different matter I think...
It might not affect TCP, but it could affect a protocol that can choose
between ARQ and FEC and use ARQ only where the latency budget permits.

Joe

---

From: Michael Welzl <[email-masked]>
Date: Mon, 6 Jun 2016 22:14:14 +0200

> On 6. jun. 2016, at 20.34, Joe Touch <[email-masked]> wrote:
> 
> 
> 

You seem to be saying “whether a protocol is implemented efficiently or inefficiently can affect the protocol choice”.
It can - but it doesn’t translate into a necessity to provide something in the API or not.

E.g., you could have neither TCP nor SCTP provide something in the style of SO_SNDLOWAT, or both - either way, which protocol is a better choice isn’t affected by the SO_SNDLOWAT functionality. If only one of them implements it, it becomes less efficient to use it if the application makes demands regarding buffer usage, and makes it a less favorable choice. If the TCP receiver doesn’t implement SACK, this may make TCP a less favorable choice too. There are many such implementation details … but I can’t see what a TAPS RFC should state in order to limit these problems or make things play out better.

But now I’m confused  :-(
I’m thinking of Nagle… you need to offer it in the API or you can’t use it. Why wouldn’t SO_SNDLOWAT be in the same category?

So maybe I’m just wrong.

Cheers,
Michael