Subject: Re: [CDNi] Comments on draft-bertrand-cdni-logging-00

From: <[email-masked]>
Date: Wed, 4 Apr 2012 18:04:13 +0200

Hi Roy,

Thanks for these comments.

1)      I agree that minimizing the number of reformatting steps is important to provide Logging data ASAP and to limit the Logging processing burden. I think the format of the logs must be negotiable between the uCDN and the dCDN. So, both options (1) dCDN provides logs in his native format and (2) dCDN provides logs in a standard format are considered. A standard format brings value, for instance as a fallback option if the CDNs do not support the same formats (e.g., the uCDN is not able to handle the dCDN's native format).

2)      Agreed

3)      Agreed

Best regards,

Gilles

De : [email-masked] [mailto:[email-masked]] De la part de Roy Peterkofsky
Envoyé : vendredi 30 mars 2012 02:52
À : [email-masked]
Objet : [CDNi] Comments on draft-bertrand-cdni-logging-00

First off, by way of introduction, I head product management for Skytide.  We are a leading supplier of CDN reporting and analytics software and thus a key consumer of CDN logs.  I think this gives us a good perspective on requirements for CDNi logging.

Secondly, I am very new to the IETF working group so I apologize if my comments are not in the customary or correct form for this.  I did want to get these to the group before you meet in Paris to discuss this subject area so I focused my attention on making the comments timely.

Anway:

1) I think there could stand to be some discussion around the impact of even having standardized log formats for interconnected CDNs.  I start with the assumption that each individual CDN, as an independent entity, has one (or possibly several) log file formats that it works with internally for its own reporting, billing, etc. (call these the "native" log file formats).  What format(s) the CDN uses may well be determined by which platform technology or technologies the CDN uses.  If this CDN is the dCDN in a particular scenario, it could either (a) ship the relevant logs to the uCDN in their "native" format(s) or (b) first translate them into a standardized format.  Assuming that the uCDN has its own "native" format used internally, in either case, the uCDN would then have to translate the received logs into this native format (from either the dCDN's native format or the standardized format).  The use of a common standardized format for all possible dCDNs, of course, will minimize the number of translators/connectors that the uCDN must have available (and ensure that the required data elements are always present).  But it also requires two translations to take place (dCDN native format to standardized format to uCDN native format) while simply sending the logs in the dCDN's native format will only require one translation (dCDN native format to uCDN native format).  In a high-volume (e.g. ABR/HAS) scenario where reporting/analytics/monitoring are required in real-time, the extra translation step will consume time that is not necessarily available!  Based on my work with an analytics system that can easily ingest and parse/translate multiple log file formats, I don't see the need to do so at the uCDN end to be a major issue.

2) I think the specification should consider use cases for both push and pull (or "on demand") log delivery.  In the former scenario, the dCDN would dispatch files of appropriate log records to the uCDN on some metered basis (either every X time periods or when Y log records have accumulated, for example).  In the latter, the uCDN would request log records meeting some set of criteria (time range at the least, possibly other filter elements as well) from the dCDN.  I was happy to see the subject of aggregated log delivery covered, as the "on demand" scenario for log delivery can easily be seen to have one "flavor" which returns every individual log record and another which returns only aggregated data.   It would have to be determined which measures (e.g., request, bytes, total duration) are aggregated and at what levels (anywhere from grand totals at one end of the spectrum to, at the other end, aggregation only when every other field of the log record matches).  This becomes, actually, more of a "data query" than a "log request" and can be looked at as much as a "reporting interface" as a "logging interface."

3) The value of the logging standards will be greatly enhanced if they allow for (at least optional) inclusion of certain extended/custom fields.  User agent and referrer are key extensions to the standard W3C logging formats that are of great value in CDN reporting (these are mentioned in the document).  Beyond these, the standard should describe how additional fields can be added so that interconnected CDNs, by mutual agreement, can pass additional data elements without having these extended logs "break" the integration for other CDNs that are only looking for the standard fields. 

-- 
Roy Peterkofsky
Vice President, Product Management
Skytide -- the leader in Digital Media Performance Management
www.skytide.com
(510) 250-4284

Read our new white paper: The 4 Keys to Telco CDN Success <http://www.slideshare.net/skytide/the-4-keys-to-telco-cdn-success>