Subject: [manet] Ben Campbell's No Objection on draft-ietf-manet-olsrv2-dat-metric-10: (with COMMENT)

From: "Ben Campbell" <[email-masked]>
Date: Wed, 02 Dec 2015 14:17:12 -0800

Ben Campbell has entered the following ballot position for
draft-ietf-manet-olsrv2-dat-metric-10: No Objection

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-manet-olsrv2-dat-metric/

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

- section 1, last paragraph:
The first part of this paragraph sounds like justification for making
this standards track. It would be useful to discuss why this is
experimental.  Is there an expectation some version of this may be
republished in the standards track in the future? Is there a need for
deployment experience prior to standardization?

Editorial Comments:

- Abstract:
Please expand OLSRv2 on first mention in abstract. (In addition to the
existing expansion in the body.)

- section 4, 1st paragraph:
s/metric/metrics

---

From: Henning Rogge <[email-masked]>
Date: Thu, 3 Dec 2015 10:34:50 +0100

> draft-ietf-manet-olsrv2-dat-metric-10: No Objection

While I think the metric should work well, I think it should get more
testing by other parties in different scenarios and mesh topologies.
This is the first metric published in the Manet WG, I don't feel
comfortable to push it to standard track without more data.

There is also the possibility that someone will come up with a better
idea to calculate the packet loss over time than the "queue of
counters" in the DAT metric. Something that detects a "relevant"
metric change would be great and would allow the metric to change its
value faster without producing more noise.

> _______________________________________________
> manet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/manet

---

From: Emmanuel Baccelli <[email-masked]>
Date: Thu, 3 Dec 2015 11:35:49 +0100

> comfortable to push it to standard track without more data.
I agree with Henning. More experiments are needed before going standards
track with this.

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Thu, 3 Dec 2015 10:48:47 +0000

Another “I agree with Henning”. That’s based in particular on my understanding that although this is related to a metric which there is experience with in the Freifunk network, it’s not the same.

From: manet [mailto:[email-masked]] On Behalf Of Emmanuel Baccelli
Sent: 03 December 2015 10:36
To: Ben Campbell
Cc: [email-masked]; MANET IETF; The IESG; [email-masked]
Subject: Re: [manet] Ben Campbell's No Objection on draft-ietf-manet-olsrv2-dat-metric-10: (with COMMENT)

On Thu, Dec 3, 2015 at 10:34 AM, Henning Rogge <[email-masked]<mailto:[email-masked]>> wrote:

I agree with Henning. More experiments are needed before going standards track with this.

> [email-masked]<mailto:[email-masked]>

********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 03 Dec 2015 12:00:13 +0100

> Something that detects a "relevant" metric change would be great and
> would allow the metric to change its value faster without producing more
> noise.

Could you please explain?

---

From: Henning Rogge <[email-masked]>
Date: Thu, 3 Dec 2015 12:13:42 +0100

> Could you please explain?

DAT metric uses the average of the packet loss event over a time
period. While this helps to reduce the variance of the input, it also
makes the metric slow to react to changes.

In a mobile environment links might change very quickly, which means
using the average over 60 seconds (as an example) will give you a bad
result.

I am still working on a more "statistical based" approach, but its not
complete at the moment.

Henning Rogge

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 03 Dec 2015 12:43:30 +0100

> makes the metric slow to react to changes.

Right.

> I am still working on a more "statistical based" approach, but its not

Have you looked at Section III.E of

  http://arxiv.org/pdf/1403.3488.pdf

-- Juliusz

---

From: [email-masked]
Date: Thu, 3 Dec 2015 12:50:30 +0100

Henning,

If I understood you correctly last we spoke on this matter:

	1)	DAT derived from a metric, with which you have great deployment experience
		when combined with OLSR(v2) —  but it is a derivative of, and not exactly 
		that which you have deployed.

	2)	“Great deployment experience” refers to one (large and long lasting) 
		deployment, but (as far as we know) DAT has not been tested in multiple
		diverse deployments, certainly not exhaustively.

While I am confident that DAT is awesome (having played with it a little, but only in the lab), I do applaud your recommendation to go through an experimental phase before aiming for std.track.

I would be surprised, though, if in 1-2 years time we would not feel happy move it (possibly with some nits fixed) to std. track.

Best,

Thomas

> On Dec 3, 2015, at 10:34, Henning Rogge <[email-masked] <mailto:[email-masked]>> wrote:
> 
> On Wed, Dec 2, 2015 at 11:17 PM, Ben Campbell <[email-masked] <mailto:[email-masked]>> wrote:
>> [email-masked] <mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/manet <https://www.ietf.org/mailman/listinfo/manet>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/manet <https://www.ietf.org/mailman/listinfo/manet>

---

From: Henning Rogge <[email-masked]>
Date: Thu, 3 Dec 2015 13:02:26 +0100

>   http://arxiv.org/pdf/1403.3488.pdf

We have experimented with exponential average for an ETX metric with
the OLSR.org olsrd implementation in the past and the results were...
not good.

The problem with using EWMA for frame/packet loss (as far as I
understand it) is that the input is purely binary... which means that
the EWMA is much more unstable than for "continuous input" like a
delay metric.

That is why we moved from EWMA to the current "fixed time average".

Henning

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 03 Dec 2015 14:38:51 +0100

> the EWMA is much more unstable

Yeah, makes sense -- the exponential average gives too much weight to the
latest sample.

-- Juliusz

---

From: Henning Rogge <[email-masked]>
Date: Thu, 3 Dec 2015 14:51:17 +0100

> Yeah, makes sense -- the exponential average gives too much weight to the

Or it converges too slowly... or both.

Henning Rogge

---

From: "Ben Campbell" <[email-masked]>
Date: Thu, 03 Dec 2015 15:46:24 -0600

> value faster without producing more noise.

Those are perfectly good reasons for experimental. It might be worth 
adding a sentence to the draft to that effect.

Thanks!

Ben.