Subject: [CDNi] 答复: RE: Re: Fwd: draft-jin-cdni-content-deduplication-optimization-02.txt

From: [email-masked]
Date: Thu, 2 Aug 2012 09:43:20 +0800

Dear Theodore,

Thank you for your quick response and detailed explanation. Please see my 
feedback below:

"Theodore Zahariadis" <[email-masked]> 写于 2012-08-01 16:00:20:

> Dear li,
> Maybe my reaction was fast and I did not explain it very well.
> 
> The proposal is to have a unique ID of the content (CID) calculated 
> from the content itself.
> Like a fingerprint of the file, (which in the worst case could be 
> recalculated, but in the general  case should not).  
> E.g.
> A  256 hash (or another low level descriptor) on the complete 
> content of the file.
> Or a combination of the file content and the file size… (as the hash
> may be only 99,999% unique)
> In this way you have an identity of the content file itself.
>  [Li Mian]: so which entity/party would do this HASH calculation and 
generate the UID (CM-CID-filename.xyz), the content provider or the CDN?
   [Li Mian]: also if CID is calculated on the complete content of the 
file, is it calculated once and deliverd through out the CDNs within CDNi 
scope without re-calculation? Or is it calculated every time the content 
enters into one CDN, e.g. from uCDN to dCDN?

> The second step is to add that CID to the content name, so it is 
> carried in the name.
> So you have a CURL:
> http://www.server1.com/.../CM-CID-filename.xyz
>  [Li Mian]: Does the embedding of CID into CURL have some security issue 
or some limitation to the URL extension? Do you have some more 
consideration on this?

> When a file content changes CDN, the name may remain the same or not.
> If the dCDN follows the rule, it may change to something like:
> http://www.server2.com/.../CM-CID-filename2.xyz
> As you see the CID is still embedded in the name and can be the 
> identity of the content even if the remaining URL has changed. 
> 
> Now lets’ assume that for a reason the content is moved to a dCDN 
> that does not follow the rule, it changes name and then to another 
> dCDN that follows the rule. The new dCDN may calculate and 
> regenerate the CID and use it again as the content key index.
>  [Li Mian]: a. if you want to keep the CID unchanged, i.e. the same as 
the original one, you need to guarantee the consistency of HASH arithmetic 
throughout the CDNi scope 'coz different CDNs may have different HASH 
calculating results. and this, to me, may not be an easy work; b. the CID 
is generated based on the complete download of the whole content. So you 
need to finish downloading of the content before you generate the CID and 
check whether this content is already stored in this CDN. This, to some 
extent, does not improve the efficiency of content de-duplication. How do 
you think?
[Li Mian]: in our content de-duplication draft, we always exchange and 
check the existence of one content object by using the content 
identification model before the actual content item is download, which may 
have better efficiency.

> Now lets’ move to the clients. 
> If the MDP file (using DASH streaming) or the http page contains 
> this CURL, the CDN can directly extract the CID (as it follows the 
> CM “magic word”), look for it using this CID and ignore the 
> remaining URL. If for a reason the file is not in the CDN and can’t 
> be found at the other CDNs, it is very simple for the CDN to remove 
> the CM-CID, regenerate the original URL and request it from the 
> original web site.
>  [Li Mian]: so how could the CDN check the content is not found at the 
other CDNs? In current implementation in framework draft, the dCDN only 
acquires content from uCDN that redirects the user's request to it.

> This is some experimentation that we have done in the project COAST (
> www.coast-fp7.eu) and the naming scheme is also explained in the Th.
> Zahariadis, E. Quacchio, “Fast content-aware delivery in overlay 
> networks,” IEEE COMSOC MMTC E-Letter, Vol.6, No. 7, July 2011, pp. 
44-47
> (http://committees.comsoc.org/mmc/e-news/E-Letter-July11.pdf  page 
44-47)  
>[Li Mian]: thank you, I will read them in detail. 

> Best Regards,
> Theodore
> 
> 
> 
> 
> From: [email-masked] [mailto:[email-masked]] 
> Sent: Wednesday, August 01, 2012 9:52 AM
> To: [email-masked]; [email-masked]; [email-masked]; 
> 'Bhumip Khasnabish'; [email-masked]
> Subject: 答复: Re: [CDNi] Fwd: draft-jin-cdni-content-deduplication-
> optimization-02.txt
> 
> 
> Dear Theodore, 
> 
> Thank you for your comments and sharing with us your proposals. 
> Could you please share the documents you mentioned below with me? 
> Based on what you explained in previous email, I have some questions
> for clarification: 
> 1. We both consider it necessary to have a unique content identifier
> for a content object. Our opinion in this draft is that this content
> ID is CSP-unique. Could you please explain more about the uniqueness
> of the UID in your proposal? 
> 2. By applying the mechanism you proposed below, we can guarantee 
> the content is uniquely stored in one CDN. And according to my 
> analysis on the basis of your explanation, the content replication 
> is checked by comparing the CURL which is similar to the current 
> mechanism in CDNi work, correct me if my understanding is wrong. If 
> so, when redirection occurs between uCDNs and dCDN, the URL would be
> changed and thus be different with the original one - CURL. And if a
> dCDN is connected with two different uCDNs and two end users request
> the same content from these two uCDNs respectively, the two uCDNs 
> may generate different redirection URLs to the dCDN. In this case we
> think the content duplication issue still exists. How do you think? 
> If my understanding is wrong, could you please explain more on how 
> to consider this issue by using your proposal? 
> 
> Thank you again~ 
> 
> [email-masked] 写于 2012-08-01 02:00:23:
> 
> > Dear Bhumip, 
> > 
> > Some ideas and considerations on the draft. 
> > 
> > We have done some work in the area of de-duplication of content 
> > (Please see Th. Zahariadis, E. Quacchio, “Fast content-aware 
> > delivery in overlay networks,” IEEE COMSOC MMTC E-Letter, Vol.6, No.
> > 7, July 2011, pp. 44-47)) 
> > 
> > Actually, we believe that it is needed to define Unique IDs (UID) 
> > per content object/chunk in order to: 
> > a) Avoid extended data replications at the CDN level and minimize 
> > the load to find the correct object. 
> > b) Detect and retrieve very fast the content. This should be fast 
> > enough to allow even seamless real-time video streaming by 
> > retrieving video chunks 
> > c) Be backwards compatible with todays’ URLs (in case the 
> > content/chunk is not found, you should be able to go to the original 
site).
> > 
> > In order to meet the (a), UID should be always associated with the 
> > content object itself (e.g. encapsulated in the object) or be based 
> > on unique characteristics of the content object (e.g. a set of low 
> > level descriptors). However, (b), poses that this could be 
> > calculated once (or sometimes), but should not be calculated or 
> > generated each time the object is requested; instead it should be 
> > “carried” and “extracted” in most cases.  On the other hand, due 
to 
> > backwards compatibility needs (c), we should not change the standard
> > file format (e.g. we could not encapsulate UID or low level 
> > descriptor in the content object). 
> > 
> > One solution could be to create a wrapper that would encapsulate the
> > UID whenever the content object enters the CDN and extracts that at 
> > the time that the content object leaves the CDN, but this would 
> > increase the complexity and processing time. 
> > 
> > Instead, we propose to use as UID a formal file name format. UID 
> > will be a string concatenation in the format: 
> > 
> > CM-CID-filename.ext 
> > 
> > where: 
> > • CM is a “Content Marker” 
> > • CID is a content signature, which could be a self-certifying 
> > identifier e.g. MD5 or SHA-1 based-hash function on the file's 
> > content or even a combination of searchable low level descriptors. ,
> > which guarantees an easy and fast detection that this name is a UID 
> > • the original filename, which is used for making UID easily 
> > recognized by humans and avoid complex self-certifying names. 
> > 
> > Whenever new content is published or stored at the CDN, the object 
> > filename could be renamed to a UID and the URL to a Content URL 
> > (CURL) with the 
> > following format: 
> > 
> > http://www. website.com/…/CM-CID-filename.ext 
> > 
> > The CURL is a URI, which is designed to enable caching mechanisms 
> > and trigger CDN-related functionality (e.g. accessing the CDN 
> > overlay network and querying for locally or “nearby” cached 
> > content). It should also be emphasized that from the CURL, the 
> > original URL and the UID may be easily extracted, while on the other
> > hand it is fully backwards compatible with existing browsers and no 
> > modifications are needed. In this way, we may seamlessly support any
> > kind of data available in the Internet (text, images, various types 
> > of audio or video), while in case the content object is not cached 
> > in the CDN, we can always go back to the original source URL. 
> > 
> > Best regards, 
> > Theodore 
> > 
> > From: [email-masked] [mailto:[email-masked]] On Behalf 
Of 
> > Bhumip Khasnabish
> > Sent: Tuesday, July 31, 2012 8:44 PM
> > To: [email-masked]
> > Subject: [CDNi] Fwd: draft-jin-cdni-content-deduplication-
> optimization-02.txt 
> > 
> > 
> > To: i-d-announce at ietf.org 
> > Subject: I-D Action: draft-jin-cdni-content-deduplication-
> optimization-02.txt 
> > From: internet-drafts at ietf.org 
> > Date: Sun, 29 Jul 2012 19:29:14 -0700 
> > Delivered-to: i-d-announce at ietfa.amsl.com 
> > List-archive: <http://www.ietf.org/mail-archive/web/i-d-announce> 
> > List-help: <mailto:[email-masked]?subject=help> 
> > List-id: Internet Draft Announcements only <i-d-announce.ietf.org> 
> > List-post: <mailto:[email-masked]> 
> > List-subscribe: <https://www.ietf.org/mailman/listinfo/i-d-announce>, 
<
> > mailto:[email-masked]?subject=subscribe> 
> > List-unsubscribe: <https://www.ietf.org/mailman/options/i-d-announce>, 
<
> > mailto:[email-masked]?subject=unsubscribe> 
> > Reply-to: internet-drafts at ietf.org 
> > 
> > A New Internet-Draft is available from the on-line Internet-Drafts 
> > directories. 
> > 
> > 
> >         Title           : Content De-duplication for CDNi Optimization 

> >         Author(s)       : WeiYi Jin 
> >                           Mian Li 
> >                           Bhumip Khasnabish 
> >         Filename        : draft-jin-cdni-content-deduplication-
> > optimization-02.txt 
> >         Pages           : 17 
> >         Date            : 2012-07-29 
> > 
> > Abstract: 
> >    Recent explosive growth of content delivery/distribution networks 
> >    (CDNs) and their interconnection are causing unintended repetition 
of 
> >    content storage in the same dCDN.  This can be avoided by using a 
> >    suitable de-duplication mechanism.  This document explores the 
> >    scenarios which create the problems, and then discusses the 
> >    approaches to eliminate the duplicated transmission of the same 
> >    content from uCDN(s) to dCDN in CDNi networks.  To implement the 
> >    optimization, some enhancements to the CDNi metadata model and 
> >    interface are required. 
> > 
> > 
> > The IETF datatracker status page for this draft is: 
> > https://datatracker.ietf.org/doc/draft-jin-cdni-content-
> > deduplication-optimization 
> > 
> > There's also a htmlized version available at: 
> > http://tools.ietf.org/html/draft-jin-cdni-content-deduplication-
> > optimization-02 
> > 
> > A diff from previous version is available at: 
> > http://tools.ietf.org/rfcdiff?url2=draft-jin-cdni-content-
> > deduplication-optimization-02 
> > 
> > 
> > Internet-Drafts are also available by anonymous FTP at: 
> > ftp://ftp.ietf.org/internet-drafts/ 
> >  _______________________________________________
> > CDNi mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/cdni