Subject: Re: [Roll] draft-ietf-roll-protocols-survey-00

From: Emmanuel Baccelli <[email-masked]>
Date: Mon, 08 Sep 2008 13:48:50 +0200

Hi Phil,

>>
>> - OLSR is said to "fail" the "loss response" criteria. However in the
>> quick justification in section 4, nothing is said about link hysteresis
>> strategy (see OLSR RFC 3626 section 14.3.) which arguably makes OLSR
>> "pass" this criteria.
>
> I disagree. All link hysteresis does is smooth a noisy signal. This 
does not mean that links do not come and go. It can reduce the maximum
frequency of advertised link state changes. It is important to note that
this technique does have a cost, in terms of routing correctness and
efficiency. I.e., a link can go to 0% suddenly yet hysteresis will
prevent this change from propagating quickly (a basic property of
smoothing). Nevertheless, it will propagate.

This level of detail should be documented in the document. For each
mechanism of each protocol, and for each criteria that is being
evaluated. Then people will have the elements to make up their mind. So
we agree: the draft is incomplete.

>>
>>
>> - OLSR is said to "fail" the "control cost" criteria. However, simple
>> fisheye TTL strategy on OLSR control messages makes OLSR "pass" this
>> criteria. See this paper "Fish Eye OLSR Scaling Properties" in Journal
>> of Communication and Networks, 2004. Actually this paper even proves
>> that OLSR can thus reach the Gupta and Kumar bounds in terms of control
>> overhead scalability.
>
> I think it's important to have a clear line defining what 
sufficiently demonstrates properties of a protocol. I think the paper
has some interesting ideas, but it isn't a protocol specification, nor
does it demonstrate a working protocol with real wireless behavior
(e.g., link dynamics). Do you have running code?

What is disturbing is that claims to look at "RFCs" and "running code"
contrast with the restricted and purely theoretical evaluation of the
protocols present in the draft. We've got to be coherent.

So if we are relying on theoretical evaluation, the comments I made are
valid. If we are instead talking about real experience with real devices
and running code, then the draft should be altered quite a bit.

regards,
Emmanuel

_______________________________________________
Roll mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/roll

---

From: JP Vasseur <[email-masked]>
Date: Mon, 08 Sep 2008 14:04:53 +0200

Hi Emmanuel,

I'd like to make two comments.
1) Writing such a document is a really difficult exercise. We have to limit
the scope in term of number of protocols studied, number of criteria and
then evaluation of each criteria. And for sure, there will be disagreement.
In the end, we will have to make a compromise and we will hopefully have a
rough consensus. We need to bear in mind in that industry needs an IP-based
solution and the routing piece is essential. In the mean-time, non-IP and
proprietary solutions do make "progress", thus the need for a solution
fairly quickly (which does not mean "rushing out", we do have to make sure
that we make a careful evaluation)
2) We need to make sure not to require a thorough evaluation of all the
characteristics of dozen of protocols and we have to draw the line
somewhere. This aim of this document is not to become a text book but rather
to look at routing protocol characteristics that are relevant to the context
of ROLL (in light with the application-specific routing requirements
documents).

Of course, if you feel that items are missing, thanks to continue the
discussion and suggest improvement. This document is undoubtedly not
finalized.

> protocols present in the draft. We've got to be coherent.

There are aspects that are indeed theoretical. That said, several of the
criteria have been derived from the application-specific documents, which
does not appear very clearly, I agree. Phil and I had a discussion about
this, and it should be more clearly explained.

Phil, are you planning to post a new revision taking into account the
various comments received on the ML from Adrian, Emmanuel, ...?

Thanks.

JP.

---

From: Emmanuel Baccelli <[email-masked]>
Date: Mon, 08 Sep 2008 16:09:11 +0200

Hi JP,
see inline

> that we make a careful evaluation)

As far as I am concerned, the protocols addressed in the draft are the 
ones that ROLL should be looking at. The criteria addressed in the draft 
are the ones that ROLL should be concerned with. But it makes no sense 
whatsoever to have an incomplete evaluation for these criteria.

> discussion and suggest improvement. This document is undoubtedly not

Yes. I mentionned some items that are missing for the evaluation of some 
criteria. I believe that they would make the evaluation of the criteria 
more complete. This level of completeness should be met for each 
criteria, for each protocol that is mentionned.

We agree that for now the document is incomplete. But another problem is 
that the draft is not transparent. And, maybe because of this lack of 
transparency, the draft also seems to be incorrect, as pointed out 
earlier in the discussion.

For now it is not clear how the evaluation is conducted: purely 
theoretically? Or more practically? Both? I guess that this survey 
serves as some kind of "problem statement" for the ROLL working group. 
So in this respect, its aim should be to explain what is currently 
missing within the IETF in order to achieve the goal(s) of ROLL. The 
document should talk about existing RFCs, existing drafts, and existing 
WG activities in detail, in order to evaluate how these could (or could 
not) be used to achieve the goal(s) of ROLL.

regards
Emmanuel

---

From: "Stephen Dawson-Haggerty" <[email-masked]>
Date: Mon, 8 Sep 2008 14:50:45 -0700

Other then the comments about inaccuracies and a lack of detail in Annex A,
it seems like we're still debating the scope of the document, as Phil said.
We don't want to consider arbitrary protocol modifications and extensions
because they aren't generally full protocol specifications and so couldn't
usually be adopted without further work.

The analysis presented in the draft isn't based particularly on studies of
actual protocol performance, although we did look at those where they did
exist (I remember at least AODV and TBRPF have them).  I suppose this makes
it more theoretical then practical; the issue is that it's going to be
difficult if not impossible to compare the different studies in the
literature due to differing methodology, incomplete implementations, etc.
This could be made more clear, and we can provide references to these
external studies.

One thing I would suggest is to make the guidance and lessons about figure
design more explicit and concrete, and to do it by citing mechanism examples
from the protocols we examined.  This might help address some of the
concerns raised while keeping the document within scope.

Thanks for all the good comments!
Steve

---

From: Philip Levis <[email-masked]>
Date: Tue, 9 Sep 2008 08:46:26 -0700

On Sep 8, 2008, at 2:50 PM, Stephen Dawson-Haggerty wrote:

>
> Other then the comments about inaccuracies and a lack of detail in  
> Annex A, it seems like we're still debating the scope of the  
> document, as Phil said.  We don't want to consider arbitrary  
> protocol modifications and extensions because they aren't generally  
> full protocol specifications and so couldn't usually be adopted  
> without further work.
>
> The analysis presented in the draft isn't based particularly on  
> studies of actual protocol performance, although we did look at  
> those where they did exist (I remember at least AODV and TBRPF have  
> them).  I suppose this makes it more theoretical then practical; the  
> issue is that it's going to be difficult if not impossible to  
> compare the different studies in the literature due to differing  
> methodology, incomplete implementations, etc.  This could be made  
> more clear, and we can provide references to these external studies.

Thanks for articulating this much better than I did, Stephen. It's not  
that running code is necessary for an understanding of a protocol;  
rather, a precise specification is. RFCs and other standardization  
documents can meet this requirement, as can running code. But I'm  
uncomfortable with simulations based on simplified wireless models  
being sufficient; practice has shown us time and time again that the  
real complexities (and costs) come from all of the tricky edge cases  
which rarely emerge in simplified environments. I can read RFC 3561  
and think through what will happen when a link is bursty; I'm not sure  
I can with most academic papers, as they (due to space concerns) are  
often forced to elide the details which would let a reader do the same.

> One thing I would suggest is to make the guidance and lessons about  
> figure design more explicit and concrete, and to do it by citing  
> mechanism examples from the protocols we examined.  This might help  
> address some of the concerns raised while keeping the document  
> within scope.

This is a great idea. It's clear from this discussion that the  
document needs to be clearer on its methodology and why that  
methodology is used. Our initial intention had been to make it very  
concise and factual rather than explanatory.

Phil

---

From: "Arsalan Tavakoli" <[email-masked]>
Date: Tue, 9 Sep 2008 09:42:22 -0700

I would just like to follow-up on what some others have been saying.
In terms of scope, I think the purpose of this document was to provide
a baseline benchmark of the suitability of the discussed protocols for
L2N applications.  The criteria and quantitative benchmarks were
culled from the application requirement documents, but admittedly can
be more clearly stated in the draft rather than just simple
references.  As far as the set of extensions/twists/etc., I would
argue the following two points: first, the set of all these is
immense, at least to the point that it is not practical to include
them all in a single survey.  Second, it's not quite clear what's an
extension, and what's a reworking.  If I take one of the protocols,
and completely change its concept of link estimation and path
calculation, that is in essence twisting it into a new protocol.  As
Phil has pointed out, the RFCs are a much more precise specification
than an academic paper or code, which allows us to go through the
process of theoretically verifying bounds and performances.  The
unique requirements of L2N networks (which prompted such a document in
the first place), make it very difficult to hypothesize how existing
performance studies would be affected by L2N specific conditions.
Granted that the justifications can be lengthier and any inaccuracies
should be cleaned up, but the purpose of the table is to provide a
strict evaluation of the protocol RFCs, which serve to provide a
starting point for discussions such as the one on this list, and allow
subsequent drafts to detail specific extensions or modifications that
can address shortcomings of the RFC (according to the criteria in this
survey) in a much more focused manner.  This draft absolutely does not
take a position on whether a new protocol is needed or whether
extensions can address all the requirements.  However, if all
extensions and code and tweaks were included in the evaluation, due to
the sheer quantity and imprecision/uncertainty of those
specifications, we would end up with a table filled with all "?'s" or
passes, which I argue would make it essentially useless.

Thanks for all the comments; they have been quite helpful in
highlighting things in the draft that need to be clarified.

Arsalan

---

From: JP Vasseur <[email-masked]>
Date: Wed, 10 Sep 2008 10:05:08 +0200

Hi Emanuel,

> whatsoever to have an incomplete evaluation for these criteria.

We're in full sync. Could you then propose additional text/sections/... ?

> criteria, for each protocol that is mentionned.

Yes, and this is a -00 version. That said, we need to find the right
compromise between conciseness and accuracy, not so easy in this case.

> earlier in the discussion.

Not sure what you mean by transparent. It is probably just because of
missing explanations, which I am sure, the authors will be happy to add. DO
not hesitate to propose text.

> theoretically? Or more practically? Both?

It should certainly be both.

I guess that this survey
> serves as some kind of "problem statement" for the ROLL working group.

We fortunately pass that stage. The aim of this document is to determine the
next steps for the WG.

> missing within the IETF in order to achieve the goal(s) of ROLL.

This is the aim of the table.

The 
> not) be used to achieve the goal(s) of ROLL.

Not quite. Remember, we cannot afford to write a text book or to document
the activity of other WGs. Still, the document has to be clear, detailed
enough and concise. Thanks for your help.

Thanks.

JP.

---

From: Emmanuel Baccelli <[email-masked]>
Date: Wed, 10 Sep 2008 10:27:17 +0200

Hi Arsalan,

OK. I think we basically agree on these three points:

1 - the methodology of the draft should be more transparent.

2 - overall, it should be clearer that the draft does not take a 
position on whether a new protocol is needed or whether an existing 
protocol can be extended.

3 - the evaluation of mentionned RFCs, drafts and related on-going 
activities in other IETF WGs should be more precise.

Cheers,
Emmanuel

---

From: Emmanuel Baccelli <[email-masked]>
Date: Wed, 10 Sep 2008 10:37:28 +0200

Hi JP,

Do you also agree with the points mentionned in my previous email?

Moreover, what do you mean exactly by your comment below?

> We fortunately pass that stage. The aim of this document is to determine the

Problem statements are there to do exaclty this: indentify next steps 
for a WG. So what do you mean by "we pass(ed) this stage"?

cheers
Emmanuel

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Wed, 10 Sep 2008 10:05:30 +0100

(Apologies if this doesn't thread properly, it's from a posting
before I subscribed.)

francois2.jan at orange-ftgroup.com wrote:
> Theoretically link hysteresis solution solves the issue of bad links.
In practice, link hysteresis is not very efficient > for at least 2
reasons. The link hysteresis solution is based on the reception of hello
packets

It doesn't have to be. OLSR (RFC 3626) only discusses use
of HELLO packets as an option. OLSRv2 is similar, but leaves
some options more open.

> which are generally sent at lower throughput than data packets because
hello packets are sent in broadcast. So hello
> packets have more chance to be received than data packets (see "grey
zone" issue),

First note that the grey zone issue isn't universal
- it arises for 802.11 because broadcast packets are
sent at a lower data rate than unicast packets. It
will however be an issue for any lower layers standard
that does the same.

And that points to where you really want information
to base your hysteresis solutions on - from lower layers.
In a radio-based system not having/using such information
is never going to do as well as you should be able to.
Use of signal to noise ratio to base the link quality in
OLSRv1/v2 works well in 802.11 systems, provided your
driver software gives you access to it.

> all the more so OLSR tends to minimize the number of hops and so tends
to choice the worst links.

That's a reason why, in 802.11 (and similar) systems at
least you need something more. The link quality mechanism
in RFC 3626, and as in current OLSRv2, is one approach that
does make things work. An alternative solution is full-blown
link metrics. This is actually where what others have called
the "infamous table" is actually generous to OLSRv2 - while
I don't agree with all of the Fails (putting aside whether I
agree with the metholodology), the Passes for metrics are
generous, I would have said ? with an agreement at the last IETF
to move to Pass, because we should be adding link metrics to
OLSRv2.

> Furthermore, the packet loss is mainly due to the SINR variation which
can be very huge depending on the material and
> the environment and link quality determined by link hysteresis can be
in phase difference with the SNR variation. 

I think this is misusing the term hysteresis, but I think that
derives from RFC 3626. I hope the OLSRv2 draft is clearer. There
are two things to be separated out

- Link quality. That can be assessed by HELLO messages (with
  the limitations you note), or signal to noise ratio, or
  other possibilities.

- Hysteresis. That you can't just flap the link on/off according
  to the latest link quality, you need to constrain that. The
  two threshold approach in OLSRv1/v2 is its implementation of
  hysteresis.

"Link quality determined by link hysteresis" is thus confusing.
But wordingh aside, yes, hysteresis can put a delay (I think
that's a more useful view than phase, as cyclic behaviour is
not the general case) on whether use or nor of a link matches
actual SNR. That indicates engineering is needed.

-- 
Christopher Dearlove
Technology Leader, Communications Group
Networks, Security and Information Systems Department
BAE Systems Advanced Technology Centre
West Hanningfield Road, Great Baddow, Chelmsford, CM2 8HN, UK
Tel: +44 1245 242194  Fax: +44 1245 242124

BAE Systems (Operations) Limited
Registered Office: Warwick House, PO Box 87,
Farnborough Aerospace Centre, Farnborough, Hants, GU14 6YU, UK
Registered in England & Wales No: 1996687

********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************

---

From: JP Vasseur <[email-masked]>
Date: Wed, 10 Sep 2008 12:11:39 +0200

Hi,

> Do you also agree with the points mentionned in my previous email?

As stated in my previous email, yes I agree with some of them.

> for a WG. 

JP> Problem statements are usually used to scope the WG.

So what do you mean by "we pass(ed) this stage"?

JP> ROLL is very much well-scoped ;-)

Thanks.

JP.

---

From: JP Vasseur <[email-masked]>
Date: Wed, 10 Sep 2008 12:13:07 +0200

Hi,

> protocol can be extended.

The aim of the document IS to help determine whether a new protocol is
needed. The outcome of the WG decision will be documented in a separate ID.

> activities in other IETF WGs should be more precise.

Do not hesitate to propose text.

Thanks.

JP.

---

From: Emmanuel Baccelli <[email-masked]>
Date: Wed, 10 Sep 2008 12:33:01 +0200

Hi JP,
please see inline.

> As stated in my previous email, yes I agree with some of them.

Well, since we now boiled down to 3 simple items, it would be better if 
you stated again which item(s) you disagree with, and why. I had the 
impression we basically agreed on all of them.

> JP> ROLL is very much well-scoped ;-)

Problem statements usually say something like:

"we have a target scenario, and the tools curently available or in 
development are not enough because... So what do we do?".

Isn't it exactly what draft-ietf-roll-protocols-survey is about?

The target scenario may already be clear to some, but people in general 
do not see why the protocols curently available or in development are 
not up to the task. And this is why a document such as 
draft-ietf-roll-protocols-survey is needed. Isn't it so? Or am I missing 
something here?

Regards,
Emmanuel

---

From: JP Vasseur <[email-masked]>
Date: Wed, 10 Sep 2008 12:41:44 +0200

Hi Emmanuel,

First of all, there are no strong disagreements here, just (important)
clarifications.

JP> I would not say "transparent" but sure it is worth detailing the
methodology. Please do not hesitate to propose text.

JP> As I said before, the aim of this draft is to draw a conclusion and
hopefully have a consensus. Please note that the evaluation is made in light
of the requirements spelled out in the application specific requirements
IDs. This is particularly important ... We're not "evaluating" protocols in
general but in light of specific requirements.

JP> Not quite correct. Evaluation of RFCs/drafts, yes but we are not
chartered to give an overview of other WG activities.

> The target scenario

For the target scenario, see the requirements IDs.

may already be clear to some, but people in general
> draft-ietf-roll-protocols-survey is needed. Isn't it so? Or am I missing

I do not think that we need to argue on what a problem statement is. The
point I was trying to make is that we passed the stage to understand why we
have a WG. The aim of this document is to provide an evaluation of the
existing routing protocol *in light* of the requirements ID and draw a
consensus.

Thanks.

JP.

---

From: Emmanuel Baccelli <[email-masked]>
Date: Wed, 10 Sep 2008 13:37:13 +0200

Hi JP,

> methodology. Please do not hesitate to propose text.

OK I guess we agree about this point then.

> general but in light of specific requirements.

This point is too fuzzy. In a previous post, you said that the 
conclusion drawing was supposed to happen in a different draft. Now you 
say it is supposed to happen in this draft. So it's either of the two, 
but not both:

(i) conclusion drawing is supposed to happen in a different draft. It 
should thus be clear that draft-ietf-roll-protocols-survey does not take 
a position on whether a new protocol is needed, or whether an existing 
protocol can be extended.

(ii) conclusion drawing is supposed to happen in 
draft-ietf-roll-protocols-survey, and then it should be clear in the 
draft that the different alternatives are being discussed, and in 
particular the discussion whether a new protocol is needed, or whether 
an existing protocol can be extended.

What is it then? It should be cristal clear whether we place ourselves 
in (i) XOR (ii) in order to avoid wasting time.

> chartered to give an overview of other WG activities.

I guess WG activities are supposed to be documented in drafts ;) So we 
agree here too.

Cheers
Emmanuel

---

From: JP Vasseur <[email-masked]>
Date: Wed, 10 Sep 2008 14:12:39 +0200

Hi,

On 9/10/08 1:37 PM, "Emmanuel Baccelli" <[email-masked]> wrote:

> say it is supposed to happen in this draft.

Looks like we're quibbling on words ;-) The objective is this ID is to
determine whether or not we can find a routing protocol that meets the
specific requirements of ROLL. The final WG decision will be documented in a
separate ID.

So it's either of the two,
> in (i) XOR (ii) in order to avoid wasting time.

I hopefully clarified.

Cheers,

JP.

---

From: Philip Levis <[email-masked]>
Date: Wed, 10 Sep 2008 11:34:48 -0700

> specific requirements of ROLL. The final WG decision will be  

I agree. I'd like to repeat a paragraph from my September 2 mail:

"This seems to be an argument about the scope of the document.
Currently, the draft simply examines *existing* protocols to determine
if any of them meet the requirements of ROLL. If the answer to this
question is "yes," then ROLL does not need to consider how to come up
with a protocol that meets its needs. If the answer to this question
is "no," then ROLL needs to discuss how best to proceed. Doing so
could be either extending existing protocols or defining new ones; the
draft takes no position on this."

In particular, work item 2 of the ROLL charter is:

"Survey the applicability of existing protocols to LLNs. The aim of
this document will be to analyze the scaling and characteristics of
existing protocols and identify whether or not they meet the routing
requirements of the applications identified above. "

Making a conclusion based on the factual findings of the draft is  
something that will guide the decision of whether to recharter or  
close the WG.

Phil

---

From: JP Vasseur <[email-masked]>
Date: Wed, 10 Sep 2008 20:47:57 +0200

Phil, 

Could you please post a new revision of the draft taking into account the
various comments received so far fairly quickly ? I¹ll so make a detailed
pass. It would be nice to have a more mature version of the ID for the ROLL
interim WG meeting in a month.

Thanks.

Cheers,

JP.

On 9/10/08 8:34 PM, "Philip Levis" <[email-masked]> wrote:

> Making a conclusion based on the factual findings of the draft is something
> that will guide the decision of whether to recharter or close the WG.
> 
> Phil
> 
>

---

From: Philip Levis <[email-masked]>
Date: Wed, 10 Sep 2008 12:05:45 -0700

On Sep 10, 2008, at 11:47 AM, JP Vasseur wrote:

> Phil,
>
> Could you please post a new revision of the draft taking into  
> account the various comments received so far fairly quickly ? I’ll  
> so make a detailed pass. It would be nice to have a more mature  
> version of the ID for the ROLL interim WG meeting in a month.

Absolutely. Stephen, Arsalan and I are working on it now.

Phil