Subject: Re: [Taps] Hiding information from applications

From: Michael Welzl <[email-masked]>
Date: Tue, 1 Jul 2014 09:28:51 +0200

On 1. juli 2014, at 02:00, Joe Hildebrand (jhildebr) <[email-masked]> wrote:

> On 6/30/14, 4:00 PM, "Michael Welzl" <[email-masked]> wrote:
> 
>> ... which is why I should never have used the word "hiding". Very sorry
>> for that. Provide information, yes, but relieve application programmers
>> from having to worry about these things and give some control to whatever
>> they use. Not every application programmer wants to care about each and
>> every detail of what goes on along a path, all the time - I'm getting the
>> impression that the view of "application programmers" or "apps" people is
>> biased in the IETF, towards... well, the kinds of app programmers who
>> attend the IETF  :-)
> 
> Perhaps, yes.  In my mind, the target audience of some of these bits are
> library developers.  These folks can expose lots of events to higher-level
> programmers, most of which might be ignored to start.  As an application
> gets more interested in nuanced user experience, the library-using
> developer can pay attention to more detail.

I agree, I also think that developers of libraries and/or middlewares could be the main users of this.
You have a point in that the library-using developer can pay attention to more detail.

>>> - The application is likely going to be the thing implementing the
>>> protocol anyway, since I'm never going to get enough OSes to give me a
>>> new
>>> kernel interface (evidence: SCTP)
>> 
>> What's your "application"?
> 
> In general, the applications I tend to think about have these properties:

I was actually thinking about middleware / library / app on top (layers), which you answered above. Nevertheless, your views below are useful, no doubt:

> - You download it from somewhere (e.g. Internet, app store, or corporate
> repository)
> - You install it yourself in a directory of your (or your admin's, or your
> OS's) choosing
> - It runs with your user privileges, nothing more
>  - No raw sockets

This makes me ask a stupid question, my apologies!  Something I've been wondering about for a while: if I install an application as a user and the *installation* requires root privileges, will that application then be allowed to use raw sockets? e.g. on a Windows machine, or a Mac?  If the answer is yes, then raw sockets aren't maybe that much of a problem after all?

>  - No low-numbered ports
>  - No access to DHCP information
> - It is running on one or more of many operating systems
> - It is written in one or more of many programming languages
> - It speaks only through the socket API to the kernel, or a wrapper around
> that API
> - It has access to multiple active network interfaces, with some limited
> ability to enumerate those devices and even less ability to query the
> properties of those interfaces
> - Its IP addresses change at least multiple times per day, sometimes
> multiple times per minute
> - It is behind a corporate firewall/NAT with an aggressive admin or it's
> behind a crappy consumer firewall/NAT
> - It has a crappy DNS library until the developer finds
> http://getdnsapi.net/
> - It might have one or more IPC mechanisms, but it might not!
> - It might have filesystem access
> - It might not stay connected when it goes into the background
> - It might have threads
> - Latency matters
> - Security matters
> 
> - Power usage matters
> - The app developer cares more about user experience than runtime cost
> - The app developer has some server-side team they have to talk to, who
> thinks the app developer is a (perhaps) necessary evil
> - The server developers care more about cost per user than user experience
> - The app developers and server developers are united in their dislike of
> the corporate firewall admins, and will collude to work around them in
> order to get paid.
> 
> There are likely other properties I didn't mention that are interesting.
> Some of these are likely not interesting to everyone, or might not impact
> the solution space.  However, I figured I'd make a start.

Sounds like a worst case though. There are server-side distributed applications too, and I run applications when I sit in my office, and so do you  :)
I think that we're sometimes too much worst-case-oriented in the IETF. Sure, things have to work in one way or another for the kind of scenario you describe here, no argument there! But when we think of fall-backs and such, designing for the worst case could translate into something like: "always make everything look like TCP port 80, even when you're in an environment where you don't have that limitation."

What I think TAPS should do is to always have such a worst case in mind, and someone in the room should constantly ask "so what happens if ..."?  (hm, actually I'm sure that this will automatically happen anyway  :-)   ).

>> You could have a middleware providing TAPS in user space.
> 
> Maybe.  As long as I didn't need to be root to install it or run it and
> I'm running on an OS with an effective IPC mechanism that didn't open up
> security holes.

Well, I'd argue that having to be root to install isn't a problem for many, many hosts out there. All these private machines.

>>> - The semantics that the application requires in order to function are
>>> mostly known at protocol design time
>> 
>> I agree, but we're discussing whether the application needs to be
>> concerned with *path-specific* information.
> 
> I think this is the crux of the difference in approaches we're discussing.
> If the semantic changes with the path, I think it's hard to write an
> application that continues to work.

The crux of difference: I agree. So, can you describe an example where this is a problem?

Here's why I didn't consider it a problem so far. Say, your application requires ordered delivery, and it uses a path that only delivers packets in-order, so all is good. We can use protocol X which doesn't provide ordering. Then the path changes, and we have the wild Internet with reordering (or maybe we use two paths now). Then the situation breaks app-semantics.

I'm arguing for having a system underneath the application automatically change to Y now, such that the app semantics do not break. With such a system in place, there's no need for all applications to check on the path just to make sure that the semantics are still correct. Sure, an app should be able to ask "what happened?" and get all the information it needs, but it shouldn't be *required* to check what happened to the network just to maintain the semantics.

I think that, as long as we consider a best effort service model, such stuff is always possible (and I'd be curious to see counterexamples). That was actually what I talked about at the IAB ITAT workshop.

>>> - Having to run a responder (e.g. server) that implements all possible
>>> combinations is prohibitively expensive in several dimensions.  In
>>> practice, it's going to be just a few that are (again) known at protocol
>>> design time
>> 
>> Okaaay... so...?
> 
> So having a heavyweight mechanism to try 2000 options when only 2 will
> work doesn't seem like something I would implement.  But it's possible
> there's a way forward I haven't seen yet, and I look forward to hearing
> more.

I didn't understand why you describe a problem. In reality, it's going to be just a few - I agree. But then you're talking about 2000 options.
Will we really have to worry about so many options when happy eyeballing? Consider:
- we can limit it to what's available on both ends
- we don't have to send out all the "happy eyeballs packets" in one go - rather, we can just accept that we can't have it on the first connection, but apply it for later connections only.

This can be further improved by caching what worked before and what didn't.

>>> - The application may want to be able to enter into a discussion with
>>> one
>>> or more of the devices on path in order to negotiate better service
>> 
>> May want to => then it shall!  :-)
> 
> If it has a mechanism to do so that is easy to use, standardized, and
> accessible from user space.

If you're talking about an application knowing which devices there are, finding them, talking to them, selecting the signal and whatnot, then I'd say this is out of scope for TAPS.

However, stuff like AEON (AECON), which builds upon best effort, should be possible to do automatically from within a system that, based on app requirements, "does its best".  So I think that automatizing things below the app and relieving the app from work actually opens up for opportunities that we didn't really have before. But I guess this is already entering a larger discussion here.

Cheers,
Michael

---

From: Brian Trammell <[email-masked]>
Date: Tue, 1 Jul 2014 13:21:25 +0200

hi Michael,

a point on pessimism and raw sockets, below...

> This makes me ask a stupid question, my apologies!  Something I've been wondering about for a while: if I install an application as a user and the *installation* requires root privileges, will that…

That all depends on what the installer is doing with its privileges. Installation privilege doesn't (shouldn't) imply runtime privilege. On everything-but-Windows, you either need to grant a process specific privileges for raw sockets (if the system has fine-grained privileges) or install things setuid root (if it doesn't). 

My _guess_ is that anything that's installing software these days that (1) requires privilege for install and (2) looks like it's doing things that would require raw sockets are actually installing kernel modules / kexts. 

This doesn't change the fact that the privilege problem means that the set of things that can use raw sockets is smaller, brittler, less deployable, and less maintainable than the set of things that don't.

> I think that we're sometimes too much worst-case-oriented in the IETF. Sure, things have to work in one way or another for the kind of scenario you describe here, no argument there! But when we thi…

If you get to use raw sockets, great. If you have a real IP address, great. If you're behind a firewall administered by someone who is following good advice, great.

I'd argue that we'll need a halfway workable plan to provide better transport in the less-than-ideal layer 3 interface that most people find themselves stuck behind. And that means being depressingly pessimistic at the start of the design process. I don't think of this as saying "everything is ruined, so let's go home", rather, "everything is ruined, how can we build on top of that?"

> Well, I'd argue that having to be root to install isn't a problem for many, many hosts out there. All these private machines.

Michael, I presume that you and I, as academics, have access to machines over which we have more or less complete control, both on the client side and the server. The fact that you get to install things for which you are allowed to type a password for an admin account, which you have, would seem to verify this. 

We are not at all in the majority.

I won't even get started on the advancing nightmare presented by the fact that we're moving toward a world in which even getting to *ask* the user for privilege means you need cryptographic proof that you have permission by the platform's owner.

Cheers,

Brian

---

From: Michael Welzl <[email-masked]>
Date: Tue, 1 Jul 2014 13:44:46 +0200

> That all depends on what the installer is doing with its privileges. Installation privilege doesn't (shouldn't) imply runtime privilege. On everything-but-Windows, you either need to grant a proces…

Ok, so first, a lot of machines run Windows, and second, if it depends on what the installer is doing, then the installer could just inform the user that the installed application uses raw sockets, right?

> This doesn't change the fact that the privilege problem means that the set of things that can use raw sockets is smaller, brittler, less deployable, and less maintainable than the set of things tha…

Sure. I was just curious how "impossible" raw sockets *really* are.

> If you get to use raw sockets, great. If you have a real IP address, great. If you're behind a firewall administered by someone who is following good advice, great.

I'm not saying we should assume a "yes" to all these things. I'm saying we shouldn't *always* assume a "no" to *all* these things, for every possible case.

> I'd argue that we'll need a halfway workable plan to provide better transport in the less-than-ideal layer 3 interface that most people find themselves stuck behind. And that means being depressing…

I get that, but not build *only* on top of that is what I'm arguing. We must always have a fall-back solution for the worst case, no doubt.

> We are not at all in the majority.

I said "private" machines. All the clients that people own! Everyone can do whatever they do with all the devices they *own* - I don't know if this is the majority or if business devices are the majority, but without a doubt many, many users access the Internet via their private device!

As for the server side, the story is different there... but if the server is a "server" in the sense of being maintained by someone who's running a service, that someone usually has privileges.

Cheers,
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 1 Jul 2014 13:56:12 +0200

> Ok, so first, a lot of machines run Windows, and second, if it depends on what the installer is doing, then the installer could just inform the user that the installed application uses raw sockets,…
Two things:
* What privileges does the installer need to do its job. It needs some privileges
  to write the files it needs to write to the expected place and sometimes these
  files need specific privileges. Sometimes you can install an application anywhere,
  so nothing special is needed. If your application needs some kernel extension,
  you need higher preferences to install them.
* What privileges do you need to run the program. Here the raw sockets come into the
  game. On some platforms you can grant the application privileges and tell the OS
  to use the privileges of the application, instead of the user, when running the
  application. But this is nothing really good...
> Sure. I was just curious how "impossible" raw sockets *really* are.
The privileges are one thing, the service they provide are another thing.
Taking SCTP as an example. If you open a raw socket for SCTP, you will
receive all SCTP packets sent to the host. If you run two programs doing
this, each of them gets all the packets. This makes Out Of The Blue handling
almost impossible. For the userland stack, we disable it in that case...
So using a UDP socket is much easier.
And don't forget firewalls on Windows... If someone contacts me with a problem
regarding running the userland stack on Windows, one of my first suggestions is
to disable the Firewall completely until he figures out how to configure
it to allow SCTP packets through. This is also much easier for UDP...

Best regards
Michael
> _______________________________________________
> Taps mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/taps
>

---

From: Michael Welzl <[email-masked]>
Date: Tue, 1 Jul 2014 14:15:47 +0200

> it to allow SCTP packets through. This is also much easier for UDP...

UDP is only a 4-byte header. Who cares about this amount of overhead - I guess the pain vs. gain ratio is clearly in favor of UDP.

But even UDP isn't available everywhere. I also vividly remember Lars going to the mic when QUIC was presented and saying that UDP sometimes gets significantly less throughput than TCP, as he saw in measurements (IIRC). So I was more interested in things in the style of Minion for example - with raw sockets, such stuff could be built in userspace.

Cheers,
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 1 Jul 2014 14:30:33 +0200

> But even UDP isn't available everywhere. I also vividly remember Lars going to the mic when QUIC was presented and saying that UDP sometimes gets significantly less throughput than TCP, as he saw i…
Ahh. OK. But to receive TCP packet on a raw socket, (Minion packets are to be considered TCP packets,
since they share the protocol id), you need to disable the TCP kernel stack. At least
on BSD systems. You might be able to use divert sockets (a BSD specific thing), but as
far as I remember Jana statements, Minion contains two parts:
1. The kernel part: deliver TCP data out of order.
2. The application library part: do the decoding / encoding / ordering / whatever part.
You can use 2 on top of an unmodified TCP stack. It will work, just not optimal.

Best regards
Michael

---

From: Michael Welzl <[email-masked]>
Date: Tue, 1 Jul 2014 14:32:59 +0200

> since they share the protocol id), you need to disable the TCP kernel stack. At least

Ahhh right, yes. My bad. Sorry - shame, it's not so easy...

Cheers,
Michael

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 1 Jul 2014 13:31:03 -0500

I'm on vacation with limited email time this week, but TAPS matters, so I'm
working to stay current. On this point ...

On Tuesday, July 1, 2014, Michael Welzl <[email-masked]> wrote:

>
> On 1. juli 2014, at 13:56, Michael Tuexen <[email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>
> On 01 Jul 2014, at 13:44, Michael Welzl <[email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>
> On 1. juli 2014, at 13:21, Brian Trammell <[email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>
> hi Michael,
>
> On 01 Jul 2014, at 09:28, Michael Welzl <[email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>
> This makes me ask a stupid question, my apologies!  Something I've been
> wondering about for a while: if I install an application as a user and the
> *installation* requires root privileges, will that application then be
> allowed to use raw sockets? e.g. on a Windows machine, or a Mac?  If the
> answer is yes, then raw sockets aren't maybe that much of a problem after
> all?
>
>
> That all depends on what the installer is doing with its privileges.
> Installation privilege doesn't (shouldn't) imply runtime privilege. On
> everything-but-Windows, you either need to grant a process specific
> privileges for raw sockets (if the system has fine-grained privileges) or
> install things setuid root (if it doesn't).
>
>
> Ok, so first, a lot of machines run Windows, and second, if it depends on
> what the installer is doing, then the installer could just inform the user
> that the installed application uses raw sockets, right?
>
> Two things:
> My _guess_ is that anything that's installing software these days that (1)
> requires privilege for install and (2) looks like it's doing things that
> would require raw sockets are actually installing kernel modules / kexts.
>
> This doesn't change the fact that the privilege problem means that the set
> of things that can use raw sockets is smaller, brittler, less deployable,
> and less maintainable than the set of things that don't.
>
>
> UDP is only a 4-byte header. Who cares about this amount of overhead - I
> guess the pain vs. gain ratio is clearly in favor of UDP.
>
> But even UDP isn't available everywhere. I also vividly remember Lars
> going to the mic when QUIC was presented and saying that UDP sometimes gets
> significantly less throughput than TCP, as he saw in measurements (IIRC).
> So I was more interested in things in the style of Minion for example -
> with raw sockets, such stuff could be built in userspace.
>

Lars mentioned during our TAPS debrief at the retreat that it's not unusual
for highly restricted access networks to block UDP except for port 53.

I'm not sure whether that qualifies as "significantly less throughput than
TCP", but it reeks ...

Spencer

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 1 Jul 2014 23:35:00 +0000

On 7/1/14, 1:28 AM, "Michael Welzl" <[email-masked]> wrote:

>Sounds like a worst case though. There are server-side distributed
>applications too, and I run applications when I sit in my office, and so
>do you  :)
>I think that we're sometimes too much worst-case-oriented in the IETF.
>Sure, things have to work in one way or another for the kind of scenario
>you describe here, no argument there! But when we think of fall-backs and
>such, designing for the worst case could translate into something like:
>"always make everything look like TCP port 80, even when you're in an
>environment where you don't have that limitation."

I'm open to there being bursts of non-brokenness on private networks, and
that some application protocols may be designed for these cases.

>What I think TAPS should do is to always have such a worst case in mind,
>and someone in the room should constantly ask "so what happens if ..."?
>(hm, actually I'm sure that this will automatically happen anyway  :-)
>).

That would be great.  I think this will work best if we're really
inclusive of each others' use cases, and the limitations and freedoms that
come from those use cases.

>Here's why I didn't consider it a problem so far. Say, your application
>requires ordered delivery, and it uses a path that only delivers packets
>in-order, so all is good. We can use protocol X which doesn't provide
>ordering. Then the path changes, and we have the wild Internet with
>reordering (or maybe we use two paths now). Then the situation breaks
>app-semantics.

This is a good start.  I wonder if we can find a characteristic other than
reordering to use, since I'm having a hard time imagining a path that
provides strict ordering, particularly when you include the kernels on
both sides of the wire as a part of the path between application code in
the endpoints.  In other words, just because one hop on the path provides
a characteristic doesn't imbue the whole path with that characteristic in
the positive sense.  In the negative sense, sure: deliverability is false
if any hop won't deliver, MTU always degrades, etc.

>I'm arguing for having a system underneath the application automatically
>change to Y now, such that the app semantics do not break. With such a
>system in place, there's no need for all applications to check on the
>path just to make sure that the semantics are still correct. Sure, an app
>should be able to ask "what happened?" and get all the information it
>needs, but it shouldn't be *required* to check what happened to the
>network just to maintain the semantics.

I see where you're headed, but I guess I'm not sure that's possible yet.
It probably depends on what the characteristics are, how we detect their
viability on a path, etc.

It may also be that the characteristics I'm thinking of are so radically
different than the one you're thinking of that we may need to do some
bucketization first before we can tell what to do with them.

How about we wait on this topic until we've made a little progress on the
characteristics?

>I didn't understand why you describe a problem. In reality, it's going to
>be just a few - I agree. But then you're talking about 2000 options.
>- we don't have to send out all the "happy eyeballs packets" in one go -
>rather, we can just accept that we can't have it on the first connection,
>but apply it for later connections only.
>

It may be this comes down to how we detect or configure what is supposed
to work.  Again, let's wait a bit before we dig in here.

>If you're talking about an application knowing which devices there are,
>finding them, talking to them, selecting the signal and whatnot, then I'd
>say this is out of scope for TAPS.

Perhaps, but perhaps this is one of the characteristics that you want to
signal or detect about the path.

>However, stuff like AEON (AECON), which builds upon best effort, should
>be possible to do automatically from within a system that, based on app
>requirements, "does its best".  So I think that automatizing things below
>the app and relieving the app from work actually opens up for
>opportunities that we didn't really have before. But I guess this is
>already entering a larger discussion here.

Yes, and an interesting architecture question about how we see the pieces
fitting together.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 1 Jul 2014 23:51:06 +0000

On 7/1/14, 12:31 PM, "Spencer Dawkins at IETF"
<[email-masked]> wrote:

>Lars mentioned during our TAPS debrief at the retreat that it's not
>unusual for highly restricted access networks to block UDP except for
>port 53.
>
>I'm not sure whether that qualifies as "significantly less throughput
>than TCP", but it reeks ...

I was thinking about this as detecting UDP reachability, falling back to
non-HTTP TCP, non-HTTP TCP through an HTTP CONNECT proxy, then WebSockets,
then good-old-HTTPS, e.g.

-- 
Joe Hildebrand

---

From: Michael Welzl <[email-masked]>
Date: Wed, 2 Jul 2014 08:09:21 +0200

> then good-old-HTTPS, e.g.

This makes a lot of sense to me. So the concern with UDP is really about the possible rate limitation because that's harder to detect than blockage.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Wed, 2 Jul 2014 08:40:07 +0200

Hi,

Cutting some bits now:

> On 7/1/14, 1:28 AM, "Michael Welzl" <[email-masked]> wrote:
[snip]

> the endpoints.

I agree, a strictly non-reordering path was perhaps a bit contrived  :)

>  In other words, just because one hop on the path provides

Exactly. This is why I say that the kind of automatizing / fall-back behavior I'm envisioning requires assuming "best effort". When you ask for QoS, that kind of thing won't work because of what you say.

Side note: you call them characteristics, I call them requirements. That's an important difference because I think that an application should only specify what it needs, while "characteristics" reeks of "the app needs to know about the path". Maybe "preference" or "wish" would be even better because they are not strict requirements, it's all assuming best effort.

Ok, so some examples (from my IAB ITAT workshop slides, http://heim.ifi.uio.no/michawe/research/publications/iab_itat2013-best-effort.pdf ):
- drop packets according to intra-flow packet priorities: it's just a wish - assuming best effort, no app semantics break if devices don't do that.
- partially reliable delivery: just because the app says "I don't need a message after this deadline", nothing will break if the transport delivers it anyway, it's just less efficient.
- more capacity from using multiple paths: if that doesn't happen, the app gets less throughput. Fine, with best effort, right?
- lower latency rather than high throughput: could be done with some queuing behavior via a DSCP, could be done with LEDBAT-style-congestion control, ... - if it doesn't work out, the latency gets higher and throughput does, too.

I see a pattern here. As long as we make no strict assumptions regarding latency and throughput (best effort, after all), everything's possible.

> How about we wait on this topic until we've made a little progress on the

Well, I'd really like to be proven wrong with at least one of these radically different examples.

> to work.  Again, let's wait a bit before we dig in here.

I agree.

> signal or detect about the path.

Because you call it "characteristic". How is this an application requirement ("preference", "wish", whatever)?

> Yes, and an interesting architecture question about how we see the pieces

Absolutely!

Cheers,
Michael