Subject: [Pearg] I-D Action: draft-irtf-pearg-safe-internet-measurement-00.txt

From: [email-masked]
Date: Sun, 07 Jul 2019 17:03:20 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Privacy Enhancements and Assessments Research Group RG of the IRTF.

        Title           : Guidelines for Performing Safe Measurement on the Internet
        Author          : Iain R. Learmonth
	Filename        : draft-irtf-pearg-safe-internet-measurement-00.txt
	Pages           : 8
	Date            : 2019-07-07

Abstract:
   Researchers from industry and academia often use Internet
   measurements as part of their work.  While these measurements can
   give insight into the functioning and usage of the Internet, they can
   come at the cost of user privacy.  This document describes guidelines
   for ensuring that such measurements can be carried out safely.

Note

   Comments are solicited and should be addressed to the research
   group's mailing list at [email-masked] and/or the author(s).

   The sources for this draft are at:

   https://github.com/irl/draft-safe-internet-measurement

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-irtf-pearg-safe-internet-measurement/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-irtf-pearg-safe-internet-measurement-00
https://datatracker.ietf.org/doc/html/draft-irtf-pearg-safe-internet-measurement-00

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Iain Learmonth <[email-masked]>
Date: Mon, 8 Jul 2019 14:08:34 +0100

Hi All,

I wasn't sure that I'd have the time today to work on this, and I wanted
to make sure the chairs had time to approve the rename before the
cut-off, so I've uploaded two versions in quick succession. Sorry if
you've already read the new -00 only to find that a -01 has already
appeared.

If you'd like to diff the new -01 against the previous individual draft,
see:

https://tools.ietf.org/rfcdiff?url1=draft-learmonth-pearg-safe-internet-measurement-02.txt&url2=draft-irtf-pearg-safe-internet-measurement-01.txt

Big changes are the inclusion of "case studies" in to help describe
situations in which differing levels of consent might be considered
acceptable for measurements to be performed, and also expansion of the
data minimalization section.

Instead of redefining anonymity, psuedonymity and the thresholds needed
to say that a system can achieve these things, it may be useful to
reference draft-irtf-hrpc-anonymity depending on how that draft develops.

The guidelines on where consent is required do belong in this draft, but
the definition of informed consent, proxy consent and implied consent
may instead be in an HRPC draft. I've not looked extensively through
those drafts yet.

As always, comments are welcome either directly to me or via the PEARG list.

I won't be at this IETF meeting but am happy to chat by mail or set up a
call with anyone on this topic.

Thanks,
Iain.

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 8 Jul 2019 06:30:09 -0700

After a quick look, I see....

Document: draft-irtf-pearg-safe-internet-measurement-01.txt

   The reduced impact should not be used as an excuse for pushing higher
   risk updates, only updates that could be considered appropriate to
   push to all users should be A/B tested.

This may just be wordsmithing, but as written, this text is entirely
unrealistic. One of the major reasons that one does A/B testing is
that you are concerned about risk in the Treatment group (e.g., that
there will be a higher risk of failures, crashes, etc.) and the
reason you are doing an A/B test is to mitigate that risk.

I'm not sure how to phrase this (internally, we usually talk about
the code having been appropriately reviewed and landed according
to policy), but the current text doesn't work.

-Ekr

On Mon, Jul 8, 2019 at 6:10 AM Iain Learmonth <[email-masked]> wrote:

> --
> Pearg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/pearg
>

---

From: Iain Learmonth <[email-masked]>
Date: Mon, 8 Jul 2019 14:47:41 +0100

Hi Eric,

> reason you are doing an A/B test is to mitigate that risk.

The point that I'd like to put across here is that it is not an excuse
to be reckless or careless. A/B testing can mitigate risk to reputation
perhaps, and sure it can reduce the risk that any individual user is
affected by a bad update, but it doesn't mitigate the impact for the
users that are affected.

> to policy), but the current text doesn't work.
How about:

  The reduced impact should not be used as an excuse for pushing updates
without the same careful consideration and review that would be applied
to any software update pushed to all users.

and perhaps also adding:

  In cases which may have high impact for any individual user,
simulations or emulations should be carried out in a testing environment
before pushing the updates to users via automated channels.

Thanks,
Iain.

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 8 Jul 2019 07:15:39 -0700

> affected by a bad update, but it doesn't mitigate the impact for the

This seems like the kind of product question that is well out of scope for
PEARG. Software vendors have a wide variety of processes for determining
whether a given piece of code is suitable for shipping to their users,
ranging (at least) from "some developer thought it was good" to "multiple
detailed code reviews". I think we can all agree that defining that is out
of scope, but without that, defining what you have to do before you ship to
a fraction of your users is largely irrelevant.

-Ekr

---

From: Niels ten Oever <[email-masked]>
Date: Mon, 8 Jul 2019 16:41:19 +0200

On 7/8/19 4:15 PM, Eric Rescorla wrote:
> 
> 
> On Mon, Jul 8, 2019 at 6:47 AM Iain Learmonth <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     Hi Eric,
> 
> This seems like the kind of product question that is well out of scope for PEARG. Software vendors have a wide variety of processes for determining whether a given piece of code is suitable for shipping to their users, ranging (at least) from "some developer thought it was good" to "multiple detailed code reviews". I think we can all agree that defining that is out of scope, but without that, 

Why could we not document best practices for this? Seems quite useful to me. 

> defining what you have to do before you ship to a fraction of your users is largely irrelevant.
> 

Why ?

Niels

-- 
Niels ten Oever
Researcher and PhD Candidate
Datactive Research Group
University of Amsterdam

PGP fingerprint	   2458 0B70 5C4A FD8A 9488  
                   643A 0ED8 3F3A 468A C8B3

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 8 Jul 2019 07:47:38 -0700

On Mon, Jul 8, 2019 at 7:42 AM Niels ten Oever <[email-masked]>
wrote:

> > This seems like the kind of product question that is well out of scope
> for PEARG. Software vendors have a wide variety of processes for
> determining whether a given piece of code is suitable for shipping to their
> users, ranging (at least) from "some developer thought it was good" to
> "multiple detailed code reviews". I think we can all agree that defining
> that is out of scope, but without that,
>

The question isn't useful, but rather what this group is chartered to do.
Do you have some charter text that would support such an activity?

-Ekr

---

From: Iain Learmonth <[email-masked]>
Date: Mon, 8 Jul 2019 16:00:45 +0100

Hi Eric,

On 08/07/2019 15:15, Eric Rescorla wrote:
> determining whether a given piece of code is suitable for shipping to
> their users, ranging (at least) from "some developer thought it was
> good" to "multiple detailed code reviews".

The point is not so much to define what make code suitable for shipping,
but to say that you should not use A/B testing as an excuse to ship code
that you would not have otherwise shipped.

Your criteria for shipping to some users should be the same as the
criteria for shipping to all users. The code quality should be the same,
it should have undergone the same pre-shipping testing, etc. The only
difference should be the feature that is enabled/disabled.

This is also needed to make the experiment into good science. If you've
got other variables like relaxed review processes then you've introduced
other factors that don't make it a fair comparison between the A/B groups.

This all comes back to implied consent, and when it is appropriate to
presume that users will be expecting something. If you typically have
stringent review processes, users will expect this from future updates.
You violate the user's expectations, and also their trust, when you push
a shoddy update to them. This is true even if you've only pushed it to a
tiny fraction of your users.

Thanks,
Iain.

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 8 Jul 2019 08:25:02 -0700

> that you would not have otherwise shipped.

But again, that's precisely the point of much A/B testing: to verify that
the
code is good enough to ship. It's *part* of the shipping process.

> tiny fraction of your users.

As I said to Niels, the question of precisely what code to ship to users
seems well outside the charter of this group, which is focused strictly
on privacy. Can you please point to some charter text which would
authorize the group to work on this?

-Ekr

---

From: "Christopher Wood" <[email-masked]>
Date: Wed, 10 Jul 2019 09:55:17 -0700

Hi Ian,

As Eric points out, criteria for shipping software is well outside the scope of this RG. Thus, it should not be used as a basis for specifying measurement practices. More generally, as a document describing safe Internet measurements in a privacy-oriented RG, issues such as code quality are out of scope. While it’s true that software bugs can lead to privacy issues, criteria for preventing bugs is not a topic this group is chartered to work on. That said, being mindful of privacy when conducting A/B testing is good (though not very actionable) advice.

Also, for what it’s worth, I think it’s prudent to consider the advice and recommendations from folks who have done these experiments at scale. 

Best,
Chris, on behalf of the chairs 

On Mon, Jul 8, 2019, at 8:27 AM, Eric Rescorla wrote:
> But again, that's precisely the point of much A/B testing: to verify that the