Subject: Re: [alto] Review of draft-ietf-alto-deployments-08#appendix-A

From: Qin Wu <[email-masked]>
Date: Tue, 11 Mar 2014 10:27:40 +0000

Hi, Michael:
Sorry for late reply.
As I raised earlier in the ALTO session, it is hard to believe 
We have to use the single monitoring infrastructure or measurement tool
to measure not only the impact of ALTO but also system performance.
They could be two different monitoring infrastructures, one is used to evaluate
How alto server affects inter domain traffic and influence the application performance while
The other one is used to measure how powerful the alto server is.
Sometimes even though the ALTO server is very powerful, however we misuse the alto sever, the alto server
May not help improve traffic optimization and user experience.
Sometimes measuring the alto sever performance may be equivalent to evaluate how alto server influence 
inter-domain traffic if no misusing happen. 

I would suggest to change the title of section 3.4.4
as follows:
s/ Monitoring Infrastructure/ Monitoring Infrastructures

Also I am not sure "the number of retransmitted TCP segments" can be 
used to measure application performance.

In the 2nd bullet of section 3.4.2, it says:
"
For instance, if ALTO guidance achieves traffic
localization, one would expect that applications achieve a higher
throughput and/or smaller delays to retrieve data.  Application-
specific performance characteristics (e.g., video or audio
quality) can be useful as well.
"
It seems two sentences are disconnected. You'd better fix this as well.

Regards!
-Qin
-----Original Message-----
From: Scharf, Michael (Michael) [mailto:[email-masked]] 
Sent: Wednesday, February 26, 2014 4:53 AM
To: Qin Wu
Cc: IETF ALTO
Subject: RE: Review of draft-ietf-alto-deployments-08#appendix-A

Hi,

In order to sort out this discussion regarding terminology, I have now ...

1. Reverted the main headline of the section to what it was in -08 and before ("Monitoring ALTO"), 

2. Aligned the wording with draft-ietf-alto-protocol-26 Section 16.1.4 and Section 16.2.5, under the assumption that the content and wording therein is WG consensus

3. Added the desired sentence on third parties and trust issues to Section 3.4.1

4. Separated the discussion of "impact" and "performance" into two sections, as suggested in this thread

Below is a copy of the updated, pre-10 text.

Please let me know if I missed something.

Thanks

Michael

*** draft-ietf-alto-deployments-010pre ***

3.4.  Monitoring ALTO

3.4.1.  Impact and Observation on Network Operation

   ALTO presents a new opportunity for managing network traffic by
   providing additional information to clients.  In particular, the
   deployment of an ALTO Server may shift network traffic patterns, and
   the potential impact to network operation can be large.  An ISP
   providing ALTO may want to assess the benefits of ALTO as part of the
   management and operations (cf. [I-D.ietf-alto-protocol]).  For
   instance, the ISP might be interested in understanding whether the
   provided ALTO maps are effective, and in order to decide whether an
   adjustment of the ALTO configuration would be useful.  Such insight
   can be obtained from a monitoring infrastructure.  An NSP offering
   ALTO could consider the impact on (or integration with) traffic
   engineering and the deployment of a monitoring service to observe the
   effects of ALTO operations.  The measurement of impacts can be
   challenging because ALTO-enabled applications may not provide related
   information back to the ALTO Service Provider.

   To construct an effective monitoring infrastructure, the ALTO Service
   Provider should decide how to monitor the performance of ALTO and
   identify and deploy data sources to collect data to compute the
   performance metrics.  In certain trusted deployment environments, it
   may be possible to collect information directly from ALTO clients.
   It may also be possible to vary or selectively disable ALTO guidance
   for a portion of ALTO clients either by time, geographical region, or
   some other criteria to compare the network traffic characteristics
   with and without ALTO.  Monitoring an ALTO service could also be
   realized by third parties.  In this case, insight into ALTO data may
   require a trust relationship between the monitoring system operator
   and the network service provider offering an ALTO service.

   The required monitoring depends on the network infrastructure and the
   use of ALTO, and an exhaustive description is outside the scope of
   this document.

3.4.2.  Measurement of the Impact

   ALTO realizes an interface between the network and applications.
   This implies that an effective monitoring infrastructure may have to
   deal with both network and application performance metrics.  This
   document does not comprehensively list all performance metrics that
   could be relevant, nor does it formally specify metrics.

   The impact of ALTO can be classified regarding a number of different
   criteria:

   o  Total amount and distribution of traffic: ALTO enables ISPs to
      influence and localize traffic of applications that use the ALTO
      service.  An ISP may therefore be interested in analyzing the
      impact on the traffic, i.e., whether network traffic patterns are
      shifted.  For instance, if ALTO shall be used to reduce the inter-
      domain P2P traffic, it makes sense to evaluate the total amount of
      inter-domain traffic of an ISP.  Then, one possibility is to study
      how the introduction of ALTO reduces the total inter-domain
      traffic (inbound and/our outbound).  If the ISPs intention is to
      localize the traffic inside his network, the network-internal
      traffic distribution will be of interest.  Effectiveness of
      localization can be quantified in different ways, e.g., by the
      load on core routers and backbone links, or by considering more
      advanced effects, such as the average number of hops that traffic
      traverses inside a domain.

   o  Application performance: The objective of ALTO is improve
      application performance.  ALTO can be used by very different types
      applications, with different communication characteristics and
      requirements.  For instance, if ALTO guidance achieves traffic
      localization, one would expect that applications achieve a higher
      throughput and/or smaller delays to retrieve data.  Application-
      specific performance characteristics (e.g., video or audio
      quality) can be useful as well.  In addition, selected statistics
      from the TCP/IP stack in hosts can be useful, e.g., the number of
      retransmitted TCP segments.

   Of potential interest can also be the share of applications or
   customers that actually use an offered ALTO service, i.e., the
   adoption of the service.

   Monitoring statistics can be aggregated, averaged, and normalized in
   different ways.  This document does not mandate specific ways how to
   calculate metrics.

3.4.3.  System and Service Performance

   A number of interesting parameters can be measured at the ALTO
   server.  [I-D.ietf-alto-protocol] suggests certain ALTO-specific
   metrics to be monitored:

   o  Requests and responses for each service listed in a Information
      Directory (total counts and size in bytes).

   o  CPU and memory utilization

   o  ALTO map updates

   o  Number of PIDs

   o  ALTO map sizes (in-memory size, encoded size, number of entries)

   This data characterizes the workload, the system performance as well
   as the map data.  Obviously, such data will depend on the
   implementation and the actual deployment of the ALTO service.

3.4.4.  Monitoring Infrastructure

   Understanding the impact of ALTO may require interaction between
   different systems, operating at different layers.  Some information
   discussed in the preceding sections is only visible to an ISP, while
   application-level performance can hardly be measured inside the
   network.  It is possible that not all information of potential
   interest can directly be measured, either because no corresponding
   monitoring infrastructure or measurement method exists, or because it
   is not easily accessible.

   One way to quantify the benefit of deploying ALTO is to measure
   before and after enabling the ALTO service.  In addition to passive
   monitoring, some data could also be obtained by active measurements,
   but due to the resulting overhead, the latter should be used with
   care.  Yet, in all monitoring activities an ALTO service provider has
   to take into account that ALTO Clients are not bound to ALTO Server
   guidance as ALTO is only one source of information, and any
   measurement result may thus be biased.

   Potential sources for monitoring the use of ALTO include:

   o  Network Operations, Administration, and Maintenance (OAM) systems:
      Many ISPs deploy OAM systems to monitor the network traffic, which
      may have insight into traffic volumes, network topology, and
      bandwidth information inside the management area.  Data can be
      obtained by SNMP, Netflow, IP Flow Information Export (IPFIX),
      syslog, etc.

   o  Applications/clients: Relevant data could be obtained by
      instrumentation of applications.

   o  ALTO server: If available, log files or other statistics data
      could be analyzed.

   o  Other application entities: In several use cases, there are other
      application entities that could provide data as well.  For
      instance, there may be centralized log servers that collect data.

   In many ALTO use cases some data sources are located within an ISP
   while some other data is gathered at application level.  Correlation
   of data would require a collaboration agreement between the ISP and
   an application owner, including agreements of data interchange
   formats, methods of delivery, etc.  In practice, such a collaboration
   may not be possible in all use cases of ALTO, because the monitoring
   data can be sensitive, and because the interacting entities may have
   different priorities.  Details of how to build an over-arching
   monitoring system for evaluating the benefits of ALTO are outside the
   scope of this memo.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Wed, 12 Mar 2014 11:26:26 +0000

Hi Qin,

Sure, we can change the headline of Section 3.4.4 to plural, and remove the example of TCP retransmissions.

Would the text below address your comment? ([...] is unchanged to my last e-mail).

Thanks

Michael

*****************************************

[...]

[...]

   o  Application performance: The objective of ALTO is improve
      application performance.  ALTO can be used by very different types
      applications, with different communication characteristics and
      requirements.  For instance, if ALTO guidance achieves traffic
      localization, one would expect that applications achieve a higher
      throughput and/or smaller delays to retrieve data.  If
      application-specific performance characteristics (e.g., video or
      audio quality) can be monitored, such metrics related to user
      experience could also help to analyze the benefit of an ALTO
      deployment.  If available, selected statistics from the TCP/IP
      stack in hosts could be leveraged, too.

[...]

3.4.4.  Monitoring Infrastructures

[...]

> From: Qin Wu [mailto:[email-masked]]
> Sent: Tuesday, March 11, 2014 11:28 AM
> To: Scharf, Michael (Michael)
> Cc: IETF ALTO
> 2. Aligned the wording with draft-ietf-alto-protocol-26 Section 16.1.4
> and Section 16.2.5, under the assumption that the content and wording
> therein is WG consensus
>