Subject: Re: [p2pi] After-BoF charter

From: "Woundy, Richard" <[email-masked]>
Date: Mon, 11 Aug 2008 12:17:13 -0400

>I believe you have just stated what is optimal for two providers, but I'm not convinced that the entire universe of providers agrees with you.

I have to agree with Bruce. I still maintain that for Comcast, access costs are much more of interest than transit / cross-ISP traffic costs. But I have heard that the relative magnitude of costs could be different for other ISPs.

Some examples I am aware of:
- ISPs in geographies outside of North America / Europe / East Asia, due to relative scarcity of intercontinental/international fiber trunks.
- Smaller ISPs, due to their smaller geographic footprint and lower traffic commitments, leading to higher transit costs.
- Universities and other 'special case' ISPs, as Henning pointed out in an earlier email on costs.

-- Rich

----- Original Message -----
From: [email-masked] <[email-masked]>
To: Bob Briscoe <[email-masked]>
Cc: [email-masked] <[email-masked]>; Lisa Dusseault <[email-masked]>
Sent: Mon Aug 11 09:51:48 2008
Subject: Re: [p2pi] After-BoF charter

Good points Bob, most of which I agree with. A little clarification  
below:

On Aug 8, 2008, at 8:40 PM, Bob Briscoe wrote:

> Enrico,
>
> Charter generally looks good.
> Generally agree with Lars, Dave's & Laird's responses so far.
> My additional 3 penny's worth:
>
[snip]

>
> 3/ What does optimal mean, and what does better mean?
>
> The charter has hidden assumptions about what optimal is, and what  
> better is. From whose perspective? It needs to either say what  
> these assumptions are, or say that the w-g will define what optimal  
> means in a w-g document.

I would suggest that there will be different views of optimal for  
different providers, and the WG should work on capturing that fact  
and developing solutions that can deal with it.

>
> As I have to keep saying, and as Richard Woundy from ComCast has  
> agreed, excessive cross domain traffic is much less bad than  
> excessive local traffic.

I believe you have just stated what is optimal for two providers, but  
I'm not convinced that the entire universe of providers agrees with  
you. I think the WG needs to deal with the fact that some providers  
will care about reducing backhaul traffic and others will care about  
reducing inter-domain traffic.

Rather than trying to get this all nailed down in the charter, I  
think it would be good if the charter mentioned the problem of  
defining optimality from a variety of perspectives as one of its work  
items.

I also note that the charter is ultimately intended only to specify  
the interface between clients and "peer selection services"  - so we  
need to make sure that the information carried across that interface  
is sufficiently rich to allow for a range of different definitions of  
optimality. We don't need to get everyone to buy into a single  
definition of optimality.

The rest of your email below finds me in complete agreement.

Bruce

>
> I agree that the /current/ traffic arrangement resulting from p2p  
> has excessive cross-domain traffic. But this is the not-seeing- 
> beyond-the-first-step problem I tried to explain at the mic in the  
> ALTO BoF.
>
> Once traffic is sitting in an optimum arrangement (possibly  
> bootstrapped by ALTO then optimised by e2e congestion control), the  
> best place to add the /next/ connection will likely be cross-domain  
> not local. Because the optimium is a /balance/ of light congestion  
> across both. This is why the goal MUST NOT be unconditional  
> localisation. That could eventually drive the network /beyond/ the  
> balance, persistently trying to drive up congestion in the access  
> and backhaul. That will be much worse than excessive cross-domain  
> traffic.
>
> You might say it would be nice to have that problem.
>
> But if this service only does localisation, it will only be safe / 
> if/ it is used by a minority. If the popularity of an ALTO service  
> goes beyond a certain point, it will start to kill the ISP's local  
> network. Of course, then less people will use it and an equilibrium  
> might be reached. But that's what I said. Optimisation is a  
> process, not just a first step. ALTO has to be able to move traffic  
> out, not just in.
>
> \
>  \
>   \      /\
>    \    /  \ ^----- <<<---goal
>     \  /    V
>      \/
>
> NOT
> \
>  \
>   \
>    \          <<<---goal
>     \
>      \
>       \
>        \
>         \
>          \
>
>
> Bob
>
> At 20:06 07/08/2008, Enrico Marocco wrote:
>> We have tried to collect all comments received during the meeting, in
>> mailing list and hallway discussions. The new version of the charter,
>> available as always at http://alto.tilab.com/docs/charter.txt, should
>> reflect at least some of them. Please take a look and comment.  
>> Feel free
>> to say if you don't like it -- entirely or partially -- and why;  
>> or if
>> you like it. And why.
>>
>> Major changes include:
>> + terminology: dropped terms like "the oracle" and "best peer";
>> + scope: removed cache/TURN/whatever service discovery;
>> + path selection: stressed the importance of routing preferences and
>>   peering policies;
>> + forbidden third-party queries;
>> + mentioned IRTF for analysis on more sophisticated metrics.
>>
>> The current version of the charter is also more explicit about what
>> pieces of information will be provided by an ALTO service; I'll  
>> start a
>> separate thread specifically for this discussion.
>>
>> --
>> Ciao,
>> Enrico
>>
>>
>> _______________________________________________
>> p2pi mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/p2pi
>
> ______________________________________________________________________ 
> ______
> Bob Briscoe, <[email-masked]>      Networks Research Centre, BT  
> Research
> B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44  
> 1473 645196
> _______________________________________________
> p2pi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/p2pi

_______________________________________________
p2pi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2pi
_______________________________________________
p2pi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2pi

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 12 Aug 2008 10:01:58 +0100

Richard, Bruce, Laird & all,

Yes, of course, I agree with you all. I shouldn't have said the w-g 
should define what optimality means. I didn't mean that, sorry. 
Indeed, I originally gave counter-examples of NZ & Oz where 
localisation would be more important than reducing cross-domain.

But I /did/ mean that the w-g should document what different parties 
might reasonably want optimality to mean.

I want the charter to acknowledge the same old point I started with 
at the mic and on the list:
- that unconditional localisation will not be sufficient for /any/ 
ISP, so the parameters that the ALTO service will have to gather are 
more than just the static ones: topology and policies
- To avoid this being inconsistent with the other rule about not 
using fast changing metrics like congestion, we have to say 
congestion will have to be taken into account, but only long term 
averages in time, and probably across broad groups of paths.

The first thing you teach a learner driver is how to stop. Then you 
teach them how to start. Similarly, before you start moving traffic 
in one direction, you also have to know /when/ to stop and /how/ to 
move it back in the other direction.

Bob

At 17:17 11/08/2008, Woundy, Richard wrote:
> >I believe you have just stated what is optimal for two providers, 
> but I'm not convinced that the entire universe of providers agrees with you.
>
>I have to agree with Bruce. I still maintain that for Comcast, 
>access costs are much more of interest than transit / cross-ISP 
>traffic costs. But I have heard that the relative magnitude of costs 
>could be different for other ISPs.
>
>- ISPs in geographies outside of North America / Europe / East Asia, 
>due to relative scarcity of intercontinental/international fiber trunks.
>- Smaller ISPs, due to their smaller geographic footprint and lower 
>traffic commitments, leading to higher transit costs.

____________________________________________________________________________
Bob Briscoe, <[email-masked]>      Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Laird Popkin <[email-masked]>
Date: Tue, 12 Aug 2008 06:16:26 -0400 (EDT)

I'm in violent agreement. :-)

- Laird Popkin, CTO, Pando Networks
  mobile: 646/465-0570

----- Original Message -----
From: "Bob Briscoe" <[email-masked]>
To: "Richard Woundy" <[email-masked]>
Cc: [email-masked], [email-masked]
Sent: Tuesday, August 12, 2008 5:01:58 AM (GMT-0500) America/New_York
Subject: Re: [p2pi] After-BoF charter

Bob

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Thu, 14 Aug 2008 14:28:57 -0500

Bob Briscoe wrote:
> I want the charter to acknowledge the same old point I started with at 
> the mic and on the list:
> - that unconditional localisation will not be sufficient for /any/ ISP, 

Bob: It is my belief that the post-BoF charter acknowledges this
(please see http://alto.tilab.com/docs/charter.txt).  That said ...

> so the parameters that the ALTO service will have to gather are more 
> than just the static ones: topology and policies

I think that which parameters to gather will turn out to be an
interesting topic to nail down.  Enrico had sent an email out
last week to try and gather this information; there was
some discussion and the issue got sidetracked.  So, I'd like to
refocus on that issue.

At present, we have the following discrete pieces of information
mentioned in the post-BoF charter:

   (1) Routing preferences
   (2) Priority values
   (3) AS numbers
   (4) Geographic location

Add to this the two more:

   (5) Policies
   (6) Topology

1, 2, and 5 are something that the network operator is in possession
of.  3 and 4 can be gleaned in some manner (probably not using
any standardized APIs but well known services, nonetheless.)  6
can be provided by the network operator or approximated using
synthetic coordinate systems and the like.

Information that has been deemed out of scope based on list
discussion is:

   (1) Last hop bandwidth (reason: too much uncertainty -- are we
   talking about upload or download bandwidth?  Provisioned band-
   width or current snapshot?  User privacy concerns, ISP marketing
   concerns, etc.  In short, while this is a good research problem,
   it may not be a good piece of information to be provided by
   ALTO.)

What other sort of information should be a focus of ALTO?

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
2701 Lucent Lane, Rm. 9F-546, Lisle, Illinois 60532 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
WWW:   http://www.alcatel-lucent.com/bell-labs
_______________________________________________
p2pi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2pi

---

From: John Leslie <[email-masked]>
Date: Thu, 14 Aug 2008 16:42:49 -0400

Vijay K. Gurbani <[email-masked]> wrote:
> interesting topic to nail down.

   Interesting, certainly; but IMHO unwise to nail down too tightly.

> mentioned in the post-BoF charter:

   These are a bit amorphous...

>   (3) AS numbers

   Sometimes very useful; sometimes not...

>   (4) Geographic location

   Often wildly inaccurate, and even more often quite unrelated to
network topology. Would be _very_ useful in an alternate universe where
routing found next-hops _physically_ closer to the destination...

> Add to this the two more:

   Clearly potentially useful

>   (6) Topology

   Useful, but hard to use without latency measurments (and IMHO
also indicators of congestion).

> of.

   Each BGP router will have these, very likely reflecting configuration
choices of the ISP. But I'm not convinced they're useful unless you
know you'll pass through that router. Even then, what you actually
experience is the aggregate of all routers you pass through...

> 3 and 4 can be gleaned in some manner (probably not using
> any standardized APIs but well known services, nonetheless.)

   AS numbers are reasonably accurately gleaned; geographic location
is not nearly as accurate.

> 6 can be provided by the network operator or approximated using

   Actual topology is likely to change without warning; so you should
be prepared for it to prove wildly inaccurate. More often than not,
it turns out to be only roughly approximated or hopelessly optimistic.

>   (1) Last hop bandwidth

   The most useful number yet listed, _if_ folks will give it to us
and we interpret it correctly...

>   (reason: too much uncertainty -- are we talking about upload or
>   download bandwidth?

   Both, of course (though too often the upstream dominates).

>  Provisioned band-width or current snapshot?

   Provisioned, of course (though it would be good to know penalty-box
bandwidth too)

>   User privacy concerns, ISP marketing concerns, etc.

   These are by no means limited to last-hop bandwidth. _Any_ question
we might ask might generate a "none of your business" reply -- we
need to allow for that.

   Besides, last-hop bandwidth (provisioned) can't really be hidden.
If the ISP doesn't want to provide it, a third party can...

> In short, while this is a good research problem, it may not be a
> good piece of information to be provided by ALTO.)

   We'd be foolish to design a protocol which _isn't_ also good as
a research tool. _Every_ content supplier will want research done.

   IMHO, ruling out anything except content-IDs (if insufficiently
obfuscated to be of no interest to DCMA lawyers) is unwise. OTOH,
expecting _any_ particular query to always receive a useful reply
is naive.

> What other sort of information should be a focus of ALTO?

   It would be rather useful to have "characteristics that indicate
congestion" on a path. The applications are going to have to judge
instantaneous congestion anyway, and basing that judgment on
known characteristics rather than guesswork should speed progress
towards an optimal mix.

   Jitter in the latency will be of interest to certain applications.

--
John Leslie <[email-masked]>
_______________________________________________
p2pi mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/p2pi

---

From: Laird Popkin <[email-masked]>
Date: Thu, 14 Aug 2008 17:10:04 -0400 (EDT)

This is a very good discussion.

Responding to the comment that topology is "Useful, but hard to use without latency measurments (and IMHO
also indicators of congestion)." I would agree that a realtime, detailed topology would be very hard to provide. But even a broad, static topology has a lot of value, and that's easier to provide. That is, if an ISP can provide a list of IP prefixes that identify users in a metro area, and adjacent metro areas, we can connect people based on that and improve performance and efficiency, without knowing anything about the physical network or individual link status.

Also keep in mind that ALTO only guides initial peer selection (i.e. the endpoints of the streams). The transport layer (BGP, etc.) guides the actual routing of the data, and the applications adapt based on observed throughput. So ALTO doesn't have to solve everything (congestion, real-time conditions) to be useful - it just has to be better than not knowing anything.

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Tue, 19 Aug 2008 12:48:17 -0500

>    Interesting, certainly; but IMHO unwise to nail down too tightly.

John: Agreed, and my intention was not to nail them down but
continue discussion on which pieces of information are useful to
guide work that will be done in the future.  After all is said and
done, one of the deliverables will be a schema document -- as
extensible as it may be -- to communicate information between
the pertinent entities.

>    These are a bit amorphous...

Fair enough.  Folks have suggested that preferences and priorities
along the lines of:

   <local 1> _is_better_than_
    <external ASN 2> _is_better_than_
     <local 2>

How do you see that we can make these a bit more concrete?

>    Sometimes very useful; sometimes not...

But arguably a low-hanging fruit that does indeed provide some
information that is helpful in the absence of any other guidance.

> routing found next-hops _physically_ closer to the destination...

Probably needs a bit more discussion on whether or not this is
valuable before discarding it.

> it turns out to be only roughly approximated or hopelessly optimistic.

Sure; synthetic coordinate systems have been designed with
approximations and opportunistic optimism in mind.  While the
active guidance of an ISP in imparting topological information
is best, we should not preclude third-party services from
approximating the same to the best of their abilities.

> and we interpret it correctly...

Hmmm ... so I suspect that it is premature to say that last-hop
bandwidth is out of scope?  What do others think?

>    Jitter in the latency will be of interest to certain applications.

I believe that these will fall squarely in TANA, if not some other
TSV WG.

Thank you.

- vijay

---

From: Yu-Shun Wang <[email-masked]>
Date: Tue, 19 Aug 2008 11:04:41 -0700

Short comment on this bit:

> > network topology. Would be _very_ useful in an alternate universe
> > where routing found next-hops _physically_ closer to the
> > destination...
>

This may have policy (distribution rights, etc.) implications.
That said, not sure if we want to go there. Maybe worth retaining
it, but leave the interpretation/usage up to the apps.

yushun

---

From: John Leslie <[email-masked]>
Date: Tue, 19 Aug 2008 17:28:16 -0400

> How do you see that we can make these a bit more concrete?

   I get nervous when I see sorting by priority -- all it takes is one
item changing and your whole sorting is invalidated. I'd like to gather
information with very little tendency to change and report that.

   When we talk "local", we risk confusing different categories of
"local". Are we talking a neighborhood, a geographic area where
concentration is done, a routing domain, an Autonomous System, an
island, a continent, or what? Also, if "local is better", better for
what?

   I claim that what we really care about is congestion choke points.
And obviously, we care the most about choke points which will actually
be congested during our p2p transfer.

   So I'd prefer information such as

- the uplink from this CIDR block is congested part of most days;
  when its latency exceeds 10 milliseconds, assume it's congested.

- the downlink to this CIDR block is congested part of some days;
  when its latency exceeds 20 milliseconds, assume it's congested.

- the up & down link from this CIDR block to our nearest peering
  point is seldom congested; it's minimum latency is 11 milliseconds.

- our peering for CIDR blocks (a, b, c...) to AS #### is seldom
  congested.

- our peering for these CIDR block to AS #### is occasionally
  congested; packets start dropping at a latency of 1.4 milliseconds.

- peering for AS #### usually flows through AS ####

   These are all things that a NOC is aware of -- though admittedly
management _may_ believe that admitting to these would somehow
benefit their competiton.

   What the application wants to know, of course, is the above for
its local ISP plus the corresponding info for the destination ISP's
CIDR block. Being a simple-minded fellow, I thing it would be nice
to be able to query both ISPs. (The query to the remote ISP, of
course, could be done more efficiently by the tracker.)

> information that is helpful in the absence of any other guidance.

   AS numbers are very useful if we're trying to model the behavior
of routers; but this is not always the case: rather often we're
overwhelmed by local congestion.

> valuable before discarding it.

   I don't mean to discard it; but I hesitate to put much weight on
it.

> approximating the same to the best of their abilities.

   Actually, most of the stuff I listed at the beginning of this
email can be reasonably well approximated by trackers cooperating
with peers. IMHO, they'd be better advised to gather those data
than chase synthetic coordinates. (YMMV)

> bandwidth is out of scope?  What do others think?

   For myself, I think we'd be silly to rule it out-of-scope because
we don't believe ISPs would agree to provide it. _Any_ question we
ask an ISP may well receive a "none-of-your-business" reply.

> I believe that these will fall squarely in TANA, if not some other

   I don't mean to infringe on any other WG's charter: I just mean
that jitter information would sometimes be helpful in choosing a
peer. I venture no opinion on who should provide it.

---

From: "Woundy, Richard" <[email-masked]>
Date: Wed, 27 Aug 2008 16:48:41 -0400

(Catching up after coming back from vacation)

>For myself, I think we'd be silly to rule it out-of-scope because we
don't believe ISPs would agree to provide it. _Any_ question we ask an
ISP may well receive a "none-of-your-business" reply.

I might be OK with leaving this in scope. I stated two potential ISP
issues in a previous email: the marketing concern (the ISP may be
revealing its mix of regular versus premium customers to competitors)
and traffic concentration concern (traffic may be drawn to fewer ISP
links with premium customers). But I would agree with John that
different ISPs may make different choices with regards to those
concerns.

My bigger concern, however, is the user privacy concern, about the ISP
revealing to the general public about its premium/regular/economy
customers. I agree that in many cases that this information may be
actively measured, but that would seem to require at least some
complicity by the end user (e.g. the end user's host needs to respond to
the measurement traffic).

So at least we should discuss whether a user opt-in or opt-out mechanism
should be required for user-specific data like this.

-- Rich

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
John Leslie
Sent: Tuesday, August 19, 2008 5:28 PM
To: Vijay K. Gurbani
Cc: [email-masked]
Subject: Re: [p2pi] Information in an ALTO protocol

>> network topology. Would be _very_ useful in an alternate universe
where

---

From: John Leslie <[email-masked]>
Date: Thu, 28 Aug 2008 20:12:58 -0400

> revealing its mix of regular versus premium customers to competitors)

   This is certainly possible (in which case they won't answer, or will
answer with "noisy" data); but I don't think it would actually reveal
anything terribly useful.

> links with premium customers).

   I would hope that congestion would be noticed before this becomes
a problem.

> the measurement traffic).

   Customers could legitimately object to this information being revealed.

> should be required for user-specific data like this.

   IMHO, that is best left for ISPs and customers to work out themselves.