Subject: Re: [6lowapp] Where does TCP not work

From: Lars Eggert <[email-masked]>
Date: Mon, 2 Nov 2009 16:04:13 -0800

Hi,

On 2009-10-31, at 10:33, Cullen Jennings wrote:
> I could go dig a 9600 baud modem out of my closet, set the MTU at 100,
> and emulate 10% packet loss on server side and go try some things. I'm
> relatively confidently TCP, HTTP, pop, imap, SSH, and TLS will all
> work just fine.

if by "work" you mean, "complete in some finite time", then yes  
mostly. (Mostly, because there are timeouts that may kick in when a  
connection is stalled for too long.) But my guess is that performance  
would be extremely horrible (as in orders of magnitude worse than with  
more reasonable loss rates <1%.)

Lars

---

From: Kris Pister <[email-masked]>
Date: Mon, 02 Nov 2009 17:11:08 -0800

Lars -
 What's the best way to judge if a protocol stack will support TCP? I 
assume that it's some combination of end-to-end reliability and latency, 
and the statistics that govern them.
In the early days of sensor networks it was not uncommon to see 
end-to-end best-effort loss rates of 10% or more.  I don't think that 
anyone tolerates that kind of performance anymore.  With link-layer ACKs 
and path diversity in the routing layer I think that 99.9% for 
best-effort is now table stakes.  Is that good enough for TCP?  If it's 
not, how many nines do I need?  What kind of constraints are there on 
the latency distribution?

Clearly Adam and his students have gotten TCP to work over multi-hop, 
deeply duty-cycled 15.4 radios.  I think that there are probably several 
others who have as well.  Any feedback from people who have tried and 
failed?

ksjp

Lars Eggert wrote:
> ------------------------------------------------------------------------
>
> _______________________________________________
> 6lowapp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/6lowapp
>

---

From: Lars Eggert <[email-masked]>
Date: Mon, 2 Nov 2009 17:40:24 -0800

Hi,

On 2009-11-2, at 17:11, Kris Pister wrote:
> assume that it's some combination of end-to-end reliability and  
> latency, and the statistics that govern them.

yes.

> In the early days of sensor networks it was not uncommon to see end- 
> to-end best-effort loss rates of 10% or more.  I don't think that  
> anyone tolerates that kind of performance anymore.  With link-layer  
> ACKs and path diversity in the routing layer I think that 99.9% for  
> best-effort is now table stakes.

All that certainly helps, but I haven't seen much discussion on  
whether we all agree that we're only going to target scenarios for  
6LOWAPP for which we can make the assumption that such techniques will  
be in use.

>  Is that good enough for TCP?  If it's not, how many nines do I  
> need?  What kind of constraints are there on the latency distribution?

It's really difficult to predict this in general. Basically, the more  
your link/path looks like a wire, the better TCP will do. That means  
you want a relatively stable RTT, relatively stable available capacity  
and not too much (<1%?) loss.

The PILC WG published a bunch of very relevant documents, for example,

Advice for Internet Subnetwork Designers
http://tools.ietf.org/html/rfc3819

End-to-end Performance Implications of Links with Errors
http://tools.ietf.org/html/rfc3155

Advice to link designers on link Automatic Repeat reQuest (ARQ)
http://tools.ietf.org/html/rfc3366

End-to-end Performance Implications of Slow Links
http://tools.ietf.org/html/rfc3150

> Clearly Adam and his students have gotten TCP to work over multi- 
> hop, deeply duty-cycled 15.4 radios.  I think that there are  
> probably several others who have as well.  Any feedback from people  
> who have tried and failed?

I'm not saying that it can't be done in some scenarios - far from it.  
But there clearly are many scenarios where TCP won't work well.

Lars

---

From: Jonathan Hui <[email-masked]>
Date: Mon, 2 Nov 2009 18:24:00 -0800

On Nov 2, 2009, at 5:40 PM, Lars Eggert wrote:

> 6LOWAPP for which we can make the assumption that such techniques  
> will be in use.

There seems to be an implicit assumption that the target link  
technologies will be similar, if not identical, to those targeted by  
ROLL.  For those networks, Kris is right that providing >99.9% end-to- 
end success rates with <1% duty cycle for both hosts *and* routers is  
yesterday's problem.  But, yes, we need to be more explicit about what  
link technologies we are targeting.

> I'm not saying that it can't be done in some scenarios - far from  
> it. But there clearly are many scenarios where TCP won't work well.

... and to decide whether or not TCP will work, we need to flip this  
thread to a top-down approach and first agree on the application  
requirements/use cases.  I have no doubt we can get TCP to work on a  
mesh of low-power, lossy links.  But are TCP's services/constraints  
appropriate for the application?  I can think of a variety of LLN  
applications that work just fine with TCP and others that could stand  
to use something else to improve latency, message efficiency, non p2p  
flows, multihoming, mobility,  etc.

--
Jonathan Hui

---

From: Lars Eggert <[email-masked]>
Date: Mon, 2 Nov 2009 18:38:44 -0800

Hi,

> link technologies we are targeting.

I also should have mentioned that packet reordering is an issue. TCP  
basically treats a reordered packet as a loss, so you want that ration  
to be at most in the same ballpark as your loss ratio.

> requirements/use cases.

I'm not sure this can work. Sure, we have a set of applications  
identified now that we can guesstimate the requirements of. But the  
thing with a network is that once you build it, new apps will come.  
But there certainly is no harm in thinking about currently identified  
apps.

>  I have no doubt we can get TCP to work on a
> mesh of low-power, lossy links.

I wouldn't agree in to this generic statement. But I do believe that  
if you make some strong assumptions on radio technologies, routing,  
etc. that it can be made to work *under those conditions*.

>  But are TCP's services/constraints

That's another question, yes. But I'd see building new transport layer  
functions for 6LOWAPPs as an activity that would need some very strong  
ties to the TSV area, if it wouldn't be hosted there in the first place.

Lars

---

From: Jonathan Hui <[email-masked]>
Date: Mon, 2 Nov 2009 19:15:08 -0800

On Nov 2, 2009, at 6:38 PM, Lars Eggert wrote:

> basically treats a reordered packet as a loss, so you want that  
> ration to be at most in the same ballpark as your loss ratio.

Of course.  There are ways to manage reordering but, as you said,  
requires making certain assumptions.

> But there certainly is no harm in thinking about currently  
> identified apps.

In the general sense, I agree.  I was suggesting this as an approach  
to answering this thread's subject line.  Indeed, the question of  
"will TCP work?" is a harder question.

> etc. that it can be made to work *under those conditions*.

I agree with you from a strict layering perspective with minimal  
assumptions about the lower layers.  I guess I come from a world where  
many of the applicable links (802.15.4, PLC, etc.) are still being  
specified in IEEE (15.4e, 15.4g) and IETF (6lowpan).  Members of those  
groups want to minimize complexity in their respective layers, of  
course.  But some layer has to deal with some of the hard problems.   
If pushing them into the lower layers is the right long term solution,  
then I say we guide the current efforts that way.  But if we're going  
to deal with all the hard problems at higher layers, then there's  
little incentive to do so at the lower layers.

> That's another question, yes. But I'd see building new transport  
> layer functions for 6LOWAPPs as an activity that would need some  
> very strong ties to the TSV area, if it wouldn't be hosted there in  
> the first place.

Agreed.  Though the line seems to be a bit blurred because some  
suggest building transport-like mechanisms in whatever "app-level"  
protocol we're working on.

--
Jonathan Hui

---

From: Lars Eggert <[email-masked]>
Date: Mon, 2 Nov 2009 19:46:36 -0800

Hi,

one quick follow-up.

> course.  But some layer has to deal with some of the hard problems.

Of course. And currently, because there is more or less a single TCP  
standard for all deployment cases, a network needs to have certain  
properties in order for TCP to work well, so that the apps on top of  
it work well. This so far has meant that lower layers need to  
implement some functionality that results in a path that approximates  
the characteristics that TCP expects. (See the PILC documents.)

> little incentive to do so at the lower layers.

It's definitely debatable whether the approach that we've so far  
followed (making links and lower-layer network technologies implement  
stuff so that TCP/IP runs OK) is the One True Answer.

As we expand IP into environments that are very different from the  
"normal" Internet, it's not fully clear (to me at least) whether this  
approach will continue to be efficient. At the bar BOF in Stockholm, I  
made a strawman proposal that for 6LOWAPPs, we might instead start  
thinking about giving up on TCP and implementing a set of end-to-end  
transport functions that apps can pick & choose from, depending on  
their needs (reliability, uni/multicast, error protection, loss  
protection/FEC, flow control, etc.) Those functions may not need to be  
provided by a separate protocol with a header and a layer in the  
stack; we might make them available as a common library that apps link  
against, for example.

Lars

---

From: Jonathan Hui <[email-masked]>
Date: Mon, 2 Nov 2009 20:21:53 -0800

On Nov 2, 2009, at 7:46 PM, Lars Eggert wrote:

> followed (making links and lower-layer network technologies  
> implement stuff so that TCP/IP runs OK) is the One True Answer.

Yes.

> "normal" Internet, it's not fully clear (to me at least) whether  
> this approach will continue to be efficient. At the bar BOF in  
> Stockholm, I made a strawman proposal that for 6LOWAPPs, we might  
> instead start thinking about giving up on TCP and implementing a set  
> of end-to-end transport functions that apps can pick & choose from,  
> depending on their needs (reliability, uni/multicast, error  
> protection, loss protection/FEC, flow control, etc.) Those functions  
> may not need to be provided by a separate protocol with a header and  
> a layer in the stack; we might make them available as a common  
> library that apps link against, for example.

Certainly sounds like an interesting idea and it would be a breath of  
fresh air not to be encumbered by TCP.  So can we afford to sacrifice  
interoperability with TCP-based tools/infrastructure?  There are  
clearly people on both sides and it's not clear to me how to come to  
closure except to recognize both sides for the time being.

--
Jonathan Hui

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 2 Nov 2009 23:06:51 -0700

Makes sense, but I suspect that solving this type of problem is a bit  
outside of something we would do in the Apps area - seems like a an  
issues where you would want the type of expertise of a a WG in the  
transport area. Would it make sense to say something like today we  
need to run over TCP (and/or UDP) and in the future needs to be able  
to run over new transport protocols?

On Oct 31, 2009, at 6:02 PM, Don Sturek wrote:

> Hi Cullen,
>
> Sorry, I meant to provide some references on this issue.  Basically,  
> here is
> the problem:
> 1)  TCP was written back in the day when Ethernet was the primary  
> transport.
> One key assumption made in TCP was that packet timeout was related to
> congestion.  TCP reacts to packet timeouts by adjusting the transmit  
> timing
> and using back off.
> 2)  Wireless networks (especially mesh topologies) experience packet  
> loss as
> a result of failure of the RF link.  Note that IEEE 802.11 does not
> typically exhibit this type of loss.  The reason is that for IEEE  
> 802.11
> (actually WiFi) the last hop wireless link from the AP can be  
> guaranteed via
> MAC level acknowledges and retries.  The trouble with mesh links is  
> that
> link errors are not propagated and, even if they are, result in route
> re-establishment which violates the TCP timeout on the far end and  
> starts
> the "congestion management" procedure in TCP (which actually makes the
> problem worse as some of the links below indicate).
>
> I am not saying categorically that TCP does not work for wireless  
> links.  I
> am saying that for many wireless links, TCP does not work well (and  
> in some
> extreme cases, at all).  Here are some examples from the IETF  
> archives and
> other industry trials......
>
> http://www.rfc-editor.org/rfc/rfc4653.txt
> http://www.sics.se/~adam/ewsn2004.pdf
> http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=686057
>
> I used to have a more exhaustive reference list but I think if you  
> do a
> search you will see this is a long standing problem....
>
> Don
>
>
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On  
> Behalf
> Of Cullen Jennings
> Sent: Saturday, October 31, 2009 10:33 AM
> To: [email-masked]
> Subject: [6lowapp] Where does TCP not work
>
>
> Multiple people have told me that TCP won't work on some of the types
> of networks we want to run on. Anyway I'd like to understand a bit
> more on why this is.
>
> So, what are the network conditions that we think are going to cause a
> problem for TCP? and, what might one do to make something that worked
> better than TCP in these cases.
>
> Thanks, Cullen
>

---

From: Zach Shelby <[email-masked]>
Date: Tue, 3 Nov 2009 08:06:33 +0200

Hi,

(New thread title)

So to me the answer seems to be that TCP will work for some  
configurations, but obviously not for all networks, and not for all  
requirements (e.g. multicast).

> protocol we're working on.

In the Stockholm BarBof, Lars had a very good explanation on this one.  
We very well may start TSV area work on transport improvements - but  
that is a long-term effort and won't happen in time for CoAP. I do  
think we should aim at starting that work eventually, but let's get  
CoAP started first.

Obviously UDP is something we will need to support as a transport for  
CoAP, and I argue that we also need to allow the use of CoAP over TCP  
as this is useful for some applications. When using UDP, some simple  
mechanism for reliability (e.g. stop-and-wait) would of course be  
needed along with a transaction ID (see e.g. 6lowapp-frank-chopan).  
When doing this we should of course cooperate with TSV people.

So this brings us to the next question, what should we say in the CoAP  
charter about transports? Right now it says

"The protocol will operate over UDP...".

I would suggest it says something more like:

"The protocol will operate over UDP by default, and should define an  
alternative binding to TCP or another suitable reliable transport  
layer."

What do you think?

Zach

-- 
http://www.sensinode.com
http://zachshelby.org - My blog “On the Internet of Things”
Mobile: +358 40 7796297

Zach Shelby
Head of Research
Sensinode Ltd.
Kidekuja 2
88610 Vuokatti, FINLAND

This e-mail and all attached material are confidential and may contain  
legally privileged information. If you are not the intended recipient,  
please contact the sender and delete the e-mail from your system  
without producing, distributing or retaining copies thereof.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 3 Nov 2009 11:03:37 +0100

Hi Jonathan:

We went around that same question of transport at ISA100.11a.

We ended up using a form of DTLS (over UDP) that 6LoWPAN actually
compresses efficiently.

Why did we end up with this?

- every app seemed to have very different requirement on transport. The
common ground was reduced to a secured datagram service.

- we resolved to use the CCM provided by the chipset crypto engines
(same as used at L2) to save power.

- ISA100 (client-server) apps that need ack usually have app layer ack
that carries some data back. Transport layer ack is there a costly and
unwanted operation.

- Most of the data (buffered, eg readings repeated over and over) do not
require retries anyway 

- packet loss during TCP start can delay the connection for too long
(that's what started reliable UDP to pass dial numbers that later became
SCTP)

- we considered other transports like DCCP but for a number of reasons
that include those middle boxes that know only TCP and UDP, we resolved
to stay with UDP with ECN passed up to the app layer.

Cheers,

Pascal

>From: [email-masked] [mailto:[email-masked]] On
Behalf Of Jonathan Hui
>Sent: mardi 3 novembre 2009 05:22
>To: Lars Eggert
>Cc: [email-masked]
>Subject: Re: [6lowapp] Where does TCP not work
>
>

---

From: Kris Pister <[email-masked]>
Date: Tue, 03 Nov 2009 02:32:25 -0800

Pascal -
 I think that these choices in isa100 were good ones, for the reasons 
that you listed.
I'd summarize as "the legacy protocols in industrial automation aren't 
built around TCP".

Do you believe that this will be true in 6lowapp?
My opinion is that almost by definition (the "6" in the name), that TCP 
will be an important part of that future.  I'm 100% sure that we'll find 
other ways of doing things as well, but it makes me nervous when it 
looks like we're trying to reinvent every protocol from bottom to top, 
and still call it IP.

ksjp

---

From: Kris Pister <[email-masked]>
Date: Tue, 03 Nov 2009 03:19:16 -0800

Don - I like the WiFi example.  Clearly it's possible to make TCP work 
over wireless links.
If there had been a separate industry consortium based on 802.11 pushing 
DumbFi, a standard that didn't use L2 ARQ, then there would have been a 
group in the IETF trying to figure out a new transport protocol to use 
for these lossy links.

Do you believe that there is a fundamental reason why WiFi can work and 
15.4 radios can't?  I don't see any evidence that this is true.  Some of 
the early protocols in sensor networks were not well thought out, and 
the result is that there's a perception that 15.4 can't be as reliable 
as WiFi.  The reality is that 15.4 can be quite a bit *more* reliable 
than WiFi.

 >> The trouble with mesh links is that link errors are not propagated 
and, even if they are,
 >> result in route re-establishment which violates the TCP timeout on 
the far end
This remains the case with naive routing protocols, but it is *not* a 
general truth about mesh links.

ksjp

Don Sturek wrote:
> Hi Cullen,
>
> Sorry, I meant to provide some references on this issue.  Basically, here is
> the problem:
> 1)  TCP was written back in the day when Ethernet was the primary transport.
> congestion.  TCP reacts to packet timeouts by adjusting the transmit timing
> and using back off.
> 2)  Wireless networks (especially mesh topologies) experience packet loss as
> typically exhibit this type of loss.  The reason is that for IEEE 802.11
> (actually WiFi) the last hop wireless link from the AP can be guaranteed via
> MAC level acknowledges and retries.  The trouble with mesh links is that
> re-establishment which violates the TCP timeout on the far end and starts
> I am not saying categorically that TCP does not work for wireless links.  I
> am saying that for many wireless links, TCP does not work well (and in some
> extreme cases, at all).  Here are some examples from the IETF archives and
> I used to have a more exhaustive reference list but I think if you do a
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Cullen Jennings

---

From: "Don Sturek" <[email-masked]>
Date: Tue, 3 Nov 2009 05:59:59 -0800

Hi Kris,

Yes, there is a fundamental reason WiFi works well with TCP and other
wireless links don't:   the architecture of WiFi has been to have an access
point with a set of single wireless hops where data link acknowledges and
retries are employed AND where these operations can occur, where needed,
within the congestion time window TCP uses.

Have a look at an old RFC called TCP-SAT for an example where that time
window is violated.

Also, I think many of the references Lars supplied are good as well.

As you can tell, I agree with Lars on this (having worked with TCP over many
data link medium dating from the 1980's to present):  If you run TCP over
small networks or in very short time windows where the wireless link
exhibits behavior consistent with TCPs operating assumptions, you won't see
any issues.  If you operate TCP over a wireless link where the timeout
windows in TCP are invoked, you will see a slow degradation of operation
over time to where TCP is nearly unusable (all due to assumptions in TCP
around retry, in general.....)

Don

-----Original Message-----
From: Kris Pister [mailto:[email-masked]] 
Sent: Tuesday, November 03, 2009 3:19 AM
To: [email-masked]
Cc: 'Cullen Jennings'; [email-masked]
Subject: Re: [6lowapp] Where does TCP not work

and, even if they are,
the far end

ksjp

transport.
timing

---

From: "Don Sturek" <[email-masked]>
Date: Tue, 3 Nov 2009 06:08:52 -0800

I would like to see the issue of transport added to the charter.

For smart energy, we plan to start with TCP, hoping that our relatively
small networks and ROLL don't violate the TCP congestion control timeout
threshold and that the code size of TCP are not too much for our small
devices.

This said, we would like a back-up plan.  We need to have UDP as that plan
but not using TCP removes many of the interesting applications that rely on
TCP.

I do think long term having transport in the charter for 6LowAPP/CoAp is a
necessary thing.

Don

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Zach Shelby
Sent: Monday, November 02, 2009 10:07 PM
To: Jonathan Hui
Cc: [email-masked]
Subject: Re: [6lowapp] Charter and transports

Hi,

(New thread title)

What do you think?

Zach

-- 
http://www.sensinode.com
http://zachshelby.org - My blog "On the Internet of Things"
Mobile: +358 40 7796297

Zach Shelby
Head of Research
Sensinode Ltd.
Kidekuja 2
88610 Vuokatti, FINLAND

---

From: "Don Sturek" <[email-masked]>
Date: Tue, 3 Nov 2009 06:19:59 -0800

Hi Cullen,

This makes sense.

I think there is a Smart Grid bar BOF in Hiroshima.  This is probably in
response to the NIST Smart Grid roadmap and "Priority Action Plan" activity
(IETF is involved in priority action plan #1, IP for Smart Grid).  This
might be the place to suggest work on this topic in the Transport area.

Don

---

From: Richard Kelsey <[email-masked]>
Date: Tue, 03 Nov 2009 10:27:29 -0500

Date: Tue, 03 Nov 2009 02:32:25 -0800
   From: Kris Pister <[email-masked]>

Kris,

I hope we don't end up reinventing anything.  If we get the
layering right we can add a new protocols as needed and
still have everyone agree that the result is IP.  In
particular if 6LoWPAN and ROLL get their parts right, then
TCP will work out to the nodes and will be used when it is
appropriate.

I think that this thread is somewhat missing the point.  The
question isn't just whether TCP will fail to get data
delivered in this or that situation, but also, in the larger
picture, whether using TCP out to the nodes is a
cost-effective way to get the job done.  If someone has 10k
water meters sending one short packet a day, they may not
want to invest in the infrastructure needed to keep 10k TCP
connections open.  At some point in the process their data
will be available via TCP, but that doesn't require an
end-to-end TCP connnection down to each meter.  The great
thing about IP is that you get enormous flexibility in how
to structure networks and applications.

We cannot create a whole new protocol stack with a gateway
to convert between IP and our new stack.  What we can do is
provide alternatives where appropriate and take advantage of
the layering to allow people to choose which protocols they
use and, most importantly, when and where they want to do
any conversions.
                                -Richard Kelsey

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 3 Nov 2009 16:44:39 +0100

Hi Don:

Another aspect of this is that there little room in the LoWPAN for
queuing packets so congestion should be avoided proactively when
possible; one way of doing that is to model the sensor data the way
voice calls are in larger network and perform some call admission
control / reservation. 

This is fine for regular sensor output but falls short for file
transfers, unless the transport is able to control its output rate.
There has been interesting protocols that used a rate-based flow
control, for instance IBM's RTP (HPR). Rate-based outperforms
window-based when dealing with very large TTL / very large network
countenance, such as deep space... and sensor networks.

But like Cullen said, we might need a 6Lowtrans or something to work
that out :)

Pascal

>From: [email-masked] [mailto:[email-masked]] On
Behalf Of Don Sturek
>Sent: mardi 3 novembre 2009 15:00
>To: 'Kris Pister'
>Cc: [email-masked]
here
>is
loss
>as
802.11
>> (actually WiFi) the last hop wireless link from the AP can be
guaranteed
>via
that
starts
>> I am not saying categorically that TCP does not work for wireless
links.
>I
in
>some
archives and
Behalf

---

From: Lisa Dusseault <[email-masked]>
Date: Tue, 3 Nov 2009 08:39:59 -0800

I hope you could live with transport issues in another WG's charter.  As the
Areas in the IETF are currently organized, I assure you Lars as Transport
Area director knows much more about TCP and other transports than I do;
conversely I hope I am a pretty fair HTTP expert.

Anyway, that's probably the default way that IETF oldtimers would think of
splitting up the work, but we could always try to do something unusual.

Lisa

---

From: "Don Sturek" <[email-masked]>
Date: Tue, 3 Nov 2009 08:52:50 -0800

Hi Lisa,

Yes, I was expecting to work with multiple WGs to get all of this work done.
I didn't know Lars was Transport Area director but I am happy to know that
given he obviously knows the challenges of TCP in non-ethernet type
deployments.

I am pretty familiar with IETF (mostly from before the mid-1990s) so I was
expecting to see a division of work.  I do like to explain the overall
reasons for some of these requests and appreciate that the explanation
probably crosses WG boundaries.

Don

From: Lisa Dusseault [mailto:[email-masked]] 
Sent: Tuesday, November 03, 2009 8:40 AM
To: [email-masked]
Cc: Zach Shelby; Jonathan Hui; [email-masked]
Subject: Re: [6lowapp] Charter and transports

Lisa

Don

-----Original Message-----

Hi,

(New thread title)

What do you think?

Zach

Zach Shelby
Head of Research
Sensinode Ltd.
Kidekuja 2
88610 Vuokatti, FINLAND

---

From: Lloyd Wood <[email-masked]>
Date: Tue, 3 Nov 2009 16:57:13 +0000

> When doing this we should of course cooperate with TSV people.

http://tools.ietf.org/html/draft-wood-tsvwg-saratoga

is such a protocol, providing reliability and IDs.

L.

DTN work: http://info.ee.surrey.ac.uk/Personal/L.Wood/saratoga/

<http://info.ee.surrey.ac.uk/Personal/L.Wood/><[email-masked]>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 3 Nov 2009 18:02:50 +0100

Hi Lars and Kris:

>All that certainly helps, but I haven't seen much discussion on whether
we all agree that we're only going to
>target scenarios for 6LOWAPP for which we can make the assumption that
such techniques will be in use.

And path diversity when applied to TCP flows can have very adverse
effects, if the packets are actually balanced over paths of very
different latencies. A packet that is not lost but delayed in an
alternate path will trigger fast recovery and cause the congestion
window to drop. 

Add a very long TTL to the mixture and you get a recipe for a very low
throughput with packet duplication.

MPTCP might offer some benefits if the subflows were routed on a given
path each, but that's not what we are doing here so I'm not too hopeful
though I'd like to dig more.

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 3 Nov 2009 18:09:59 +0100

Hi Lloyd,

Glad you mention it.

Is there a way to see Saratoga without the 'DTN' piece? 
Could you elaborate a bit on how flow control is done in Saratoga?

Cheers,

Pascal

>From: [email-masked] [mailto:[email-masked]] On
Behalf Of Lloyd Wood
>Sent: mardi 3 novembre 2009 17:57
>To: [email-masked]

---

From: Lloyd Wood <[email-masked]>
Date: Tue, 3 Nov 2009 17:26:42 +0000

> Is there a way to see Saratoga without the 'DTN' piece?

The DTNRG bundle support is specified in a separate DTNRG draft, which  
can
be ignored entirely, along with the single bit saying 'carrying  
bundles'.
That's unimportant.

http://tools.ietf.org/html/draft-wood-tsvwg-saratoga
specifies only Saratoga and how it does file delivery and streaming.

(As an aside, I'm not convinced by the utility of DTN bundling;
doing HTTP over a variety of transports including TCP to make
it useful for unusual environments seems promising, though.)

> Could you elaborate a bit on how flow control is done in Saratoga?

If you mean congestion sensitivity, it's entirely open.
There's sufficient information flowing between sender
and receiver for a variety of schemes to be implemented, as
http://tools.ietf.org/html/draft-wood-tsvwg-saratoga
mentions - LEDBAT, TCP friendliness, etc. Pick a scheme
that suits; in many private dedicated links where link
utilization is a concern, and losses are due to channel
errors and not congestion, one might choose to run at link rate
for fast transfers.

Otherwise - HOLESTOFILL packets are sent by the receiver
or requested by the sender, and include necessary selective
negative acknowledgement (SNACK) information for reliable
file transfer or reliable streaming.

L.

---

From: Zach Shelby <[email-masked]>
Date: Tue, 3 Nov 2009 22:35:25 +0200

> alternative binding to TCP or another suitable reliable transport

Is that suitable for you? It allows for binding to future transports  
as well.

> I do think long term having transport in the charter for 6LowAPP/ 

We can't do real transport work (TCP improvements e.g.) in CoAP or the  
APP area as Lisa pointed out. The goal is however that this group (the  
6lowapp interest group in general) can spin off a new working group  
(or general activity) in the transport area for this purpose in the  
future. I think that makes sense. It would be best to first get some  
experience with CoAP and see where the real requirements and problems  
with TCP are.

Zach

Zach Shelby
Head of Research
Sensinode Ltd.
Kidekuja 2
88610 Vuokatti, FINLAND

---

From: "Don Sturek" <[email-masked]>
Date: Tue, 3 Nov 2009 12:41:55 -0800

Hi Zach,

Yes, what you wrote below is all good.   The part on transport is exactly
what I was expecting.  We need to leave it in the charter but expect it to
be spun off into transport if real work is needed.

Don

-----Original Message-----
From: Zach Shelby [mailto:[email-masked]] 
Sent: Tuesday, November 03, 2009 12:35 PM
To: [email-masked]
Cc: 'Jonathan Hui'; [email-masked]
Subject: Re: [6lowapp] Charter and transports

Zach

Zach Shelby
Head of Research
Sensinode Ltd.
Kidekuja 2
88610 Vuokatti, FINLAND

---

From: Lars Eggert <[email-masked]>
Date: Tue, 3 Nov 2009 13:39:09 -0800

Hi,

> might be the place to suggest work on this topic in the Transport  

that's a reasonable idea, but I think you've all been hanging around  
the IETF long enough to realize that any such effort will need to be  
driven by folks who want a solution. In other words, saying that "the  
TSV area should solve this" and then hoping that someone else will  
produce a solution isn't going to happen. We need the folks who need a  
solution put their cycles into designing one (in TSV, if that's where  
it'll fit, or wherever).

Lars

---

From: "Don Sturek" <[email-masked]>
Date: Tue, 3 Nov 2009 13:44:28 -0800

Hi Lars,

Yes, I would expect that someone who needs this feature will help sponsor it
via I-Ds and participation.

Could be ZigBee Smart Energy depending on how our interop is going and what
issues come up.

Don

-----Original Message-----
From: Lars Eggert [mailto:[email-masked]] 
Sent: Tuesday, November 03, 2009 1:39 PM
To: [email-masked]
Cc: 'Cullen Jennings'; [email-masked]
Subject: Re: [6lowapp] Where does TCP not work

Hi,

Lars

---

From: Kris Pister <[email-masked]>
Date: Tue, 03 Nov 2009 22:17:07 -0800

Pascal - I assume that we would only have path diversity inside the 
WPAN, and that the purpose of diversity would be to reduce latency, 
rather than increase it (e.g. RoLL, siblings when parents are having 
problems, etc.).  If you can't get the packet through to your preferred 
parent in a timely way, you try someone else in the hopes that you can 
still hit your latency target.  Yes?

Anyway, the goal of my original question was to understand what kinds of 
latency distributions would work.  I agree that it is easy to find 
distributions that will bring TCP to it's knees.  I'm hopeful that we 
can find some ways to make our meshes TCP-friendly under those 
circumstances when it is advantageous.  Under other circumstances (e.g. 
the ISA100 use cases that you presented) it's neither necessary nor 
efficient.

ksjp

---

From: Kris Pister <[email-masked]>
Date: Tue, 03 Nov 2009 22:26:09 -0800

Completely agree that TCP doesn't make any sense if you're sending one 
packet per mote per day.
Maybe I misread the intent, or I'm thinking about different apps.
I'm not saying that we should use TCP for everything - quite the 
contrary, I agree with Pascal's reasons why we didn't use it in isa100.
Having said that, I am always surprised by the strong negative reactions 
that arise when TCP is mentioned in a wireless context.  I realize that 
there have been many good and painful lessons learned in the past, but 
it also seems quite clear that TCP can run successfully on low power 
multi-hop 15.4 meshes as well.  Many/most applications won't need this, 
but for at least a few it will be perfect.  Yes?

ksjp

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 4 Nov 2009 11:26:52 +0100

Great :)

I'm certainly willing to help out here.
 
For the problem of routing in that environment (at ROLL) we produced no
less than 4 requirements drafts, and that certainly helped a lot
figuring what was left to be done. This is because wireless (even Low
Power) is so diverse in its usages and constraints. Following those
tracks we could:

* go back to the requirements we have on transports for the flows we
know of

* see which ones can be accommodated with existing transports and flow
control techniques - produce BCP there - 

* and then see if additional work is needed to accommodate real world
constraints like very femtomatic buffer capacity in the intermediate
routers, and randomized path diversity.

Maybe we could organize a BAR BOF in Anaheim?

Pascal

>From: [email-masked] [mailto:[email-masked]] On
Behalf Of Lars Eggert
>Sent: mardi 3 novembre 2009 22:39
>To: [email-masked]
>Cc: [email-masked]
>> I think there is a Smart Grid bar BOF in Hiroshima.  This is probably
>> in response to the NIST Smart Grid roadmap and "Priority Action Plan"
>> activity
the IETF long enough to realize that any
>such effort will need to be driven by folks who want a solution. In
other words, saying that "the TSV area
>should solve this" and then hoping that someone else will produce a
solution isn't going to happen. We need
>the folks who need a solution put their cycles into designing one (in
TSV, if that's where it'll fit, or
>wherever).

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 4 Nov 2009 11:51:20 +0100

Hi Kris:

The long and correct answer is in RFC 5681 section 3.2. 

The short and partial one is that packets arriving out of order cause
additional churn (acks). 3 duplicate acks (still waiting for a same
delayed segment) will trigger a fast retransmission of that delayed
segment and temporarily reduce the number of outstanding segments
(congestion window).

TCP can live with long latencies but what it will not like too much is
if we spread packets over paths of different latencies for the reasons
above. I'm not so anxious about the effect on the congestion window as I
am about the energy and bandwidth cost from the additional acks and
retries being generated for nothing.

At the core, the problem goes beyond this.

I think that our buffers cannot properly accommodate a window-based
mechanism. The large TAT that we experience in those networks only makes
things worse. All this calls for rate-based, something like a SARATOGA,
with a Committed Information Rate mechanism that would use IP ECN bit
the way FR uses FECNs and BECNs, and some reservation/CAC mechanism in
the radio fringe to proactively obtain the required resources.

Cheers,

Pascal

>Sent: mercredi 4 novembre 2009 07:17
>To: Pascal Thubert (pthubert)
>Cc: Lars Eggert; [email-masked]
end-
>>> All that certainly helps, but I haven't seen much discussion on
whether