Subject: [CDNi] Fwd: draft-jin-cdni-content-deduplication-optimization-02.txt

From: Bhumip Khasnabish <[email-masked]>
Date: Tue, 31 Jul 2012 13:43:36 -0400

- *To*: i-d-announce at ietf.org <[email-masked]>
   - *Subject*: I-D Action:
   draft-jin-cdni-content-deduplication-optimization-02.txt
   - *From*: internet-drafts at ietf.org <[email-masked]>
   - *Date*: Sun, 29 Jul 2012 19:29:14 -0700
   - *Delivered-to*: i-d-announce at ietfa.amsl.com<[email-masked]>
   - *List-archive*: <http://www.ietf.org/mail-archive/web/i-d-announce>
   - *List-help*:
<mailto:[email-masked]?subject=help<[email-masked]?subject=help>>

   - *List-id*: Internet Draft Announcements only <i-d-announce.ietf.org>
   - *List-post*: <mailto:[email-masked] <[email-masked]>>
   - *List-subscribe*: <https://www.ietf.org/mailman/listinfo/i-d-announce>,
   <mailto:[email-masked]?subject=subscribe<[email-masked]?subject=subscribe>>

   - *List-unsubscribe*: <https://www.ietf.org/mailman/options/i-d-announce>,
   <mailto:[email-masked]?subject=unsubscribe<[email-masked]?subject=unsubscribe>>

   - *Reply-to*: internet-drafts at ietf.org <[email-masked]>

------------------------------

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title           : Content De-duplication for CDNi Optimization
	Author(s)       : WeiYi Jin
                          Mian Li
                          Bhumip Khasnabish
	Filename        : draft-jin-cdni-content-deduplication-optimization-02.txt
	Pages           : 17
	Date            : 2012-07-29

Abstract:
   Recent explosive growth of content delivery/distribution networks
   (CDNs) and their interconnection are causing unintended repetition of
   content storage in the same dCDN.  This can be avoided by using a
   suitable de-duplication mechanism.  This document explores the
   scenarios which create the problems, and then discusses the
   approaches to eliminate the duplicated transmission of the same
   content from uCDN(s) to dCDN in CDNi networks.  To implement the
   optimization, some enhancements to the CDNi metadata model and
   interface are required.

The IETF datatracker status page for this draft
is:https://datatracker.ietf.org/doc/draft-jin-cdni-content-deduplication-optimization

There's also a htmlized version available
at:http://tools.ietf.org/html/draft-jin-cdni-content-deduplication-optimization-02

A diff from previous version is available
at:http://tools.ietf.org/rfcdiff?url2=draft-jin-cdni-content-deduplication-optimization-02

Internet-Drafts are also available by anonymous FTP
at:ftp://ftp.ietf.org/internet-drafts/

---

From: "Theodore Zahariadis" <[email-masked]>
Date: Tue, 31 Jul 2012 21:00:23 +0300

Dear Bhumip, 

Some ideas and considerations on the draft.

We have done some work in the area of de-duplication of content (Please see
Th. Zahariadis, E. Quacchio, "Fast content-aware delivery in overlay
networks," IEEE COMSOC MMTC E-Letter, Vol.6, No.7, July 2011, pp. 44-47))

Actually, we believe that it is needed to define Unique IDs (UID) per
content object/chunk in order to: 

a) Avoid extended data replications at the CDN level and minimize the load
to find the correct object.

b) Detect and retrieve very fast the content. This should be fast enough to
allow even seamless real-time video streaming by retrieving video chunks 

c) Be backwards compatible with todays' URLs (in case the content/chunk is
not found, you should be able to go to the original site).

In order to meet the (a), UID should be always associated with the content
object itself (e.g. encapsulated in the object) or be based on unique
characteristics of the content object (e.g. a set of low level descriptors).
However, (b), poses that this could be calculated once (or sometimes), but
should not be calculated or generated each time the object is requested;
instead it should be "carried" and "extracted" in most cases.  On the other
hand, due to backwards compatibility needs (c), we should not change the
standard file format (e.g. we could not encapsulate UID or low level
descriptor in the content object). 

One solution could be to create a wrapper that would encapsulate the UID
whenever the content object enters the CDN and extracts that at the time
that the content object leaves the CDN, but this would increase the
complexity and processing time.

Instead, we propose to use as UID a formal file name format. UID will be a
string concatenation in the format: 

CM-CID-filename.ext 

where:

. CM is a "Content Marker" 

. CID is a content signature, which could be a self-certifying identifier
e.g. MD5 or SHA-1 based-hash function on the file's content or even a
combination of searchable low level descriptors. , which guarantees an easy
and fast detection that this name is a UID

. the original filename, which is used for making UID easily recognized by
humans and avoid complex self-certifying names.

Whenever new content is published or stored at the CDN, the object filename
could be renamed to a UID and the URL to a Content URL (CURL) with the 

following format: 

http://www. website.com/./CM-CID-filename.ext

The CURL is a URI, which is designed to enable caching mechanisms and
trigger CDN-related functionality (e.g. accessing the CDN overlay network
and querying for locally or "nearby" cached content). It should also be
emphasized that from the CURL, the original URL and the UID may be easily
extracted, while on the other hand it is fully backwards compatible with
existing browsers and no modifications are needed. In this way, we may
seamlessly support any kind of data available in the Internet (text, images,
various types of audio or video), while in case the content object is not
cached in the CDN, we can always go back to the original source URL.

Best regards,

Theodore

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Bhumip Khasnabish
Sent: Tuesday, July 31, 2012 8:44 PM
To: [email-masked]
Subject: [CDNi] Fwd:
draft-jin-cdni-content-deduplication-optimization-02.txt

*	To: i-d-announce at ietf.org <mailto:[email-masked]>  
*	Subject: I-D Action:
draft-jin-cdni-content-deduplication-optimization-02.txt 
*	From: internet-drafts at ietf.org
<mailto:[email-masked]>  
*	Date: Sun, 29 Jul 2012 19:29:14 -0700 
*	Delivered-to: i-d-announce at ietfa.amsl.com
<mailto:[email-masked]>  
*	List-archive: <http://www.ietf.org/mail-archive/web/i-d-announce> 
*	List-help: <mailto:[email-masked]?subject=help> 
*	List-id: Internet Draft Announcements only <i-d-announce.ietf.org> 
*	List-post: <mailto:[email-masked]> 
*	List-subscribe:
<https://www.ietf.org/mailman/listinfo/i-d-announce>,
<mailto:[email-masked]?subject=subscribe> 
*	List-unsubscribe:
<https://www.ietf.org/mailman/options/i-d-announce>,
<mailto:[email-masked]?subject=unsubscribe> 
*	Reply-to: internet-drafts at ietf.org
<mailto:[email-masked]>  

  _____  

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

        Filename        :
 
Abstract:

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-jin-cdni-content-deduplication-optimi
zation
 
There's also a htmlized version available at:
http://tools.ietf.org/html/draft-jin-cdni-content-deduplication-optimization
-02
 
A diff from previous version is available at:
http://tools.ietf.org/rfcdiff?url2=draft-jin-cdni-content-deduplication-opti
mization-02

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: [email-masked]
Date: Wed, 1 Aug 2012 14:52:19 +0800

Dear Theodore,

Thank you for your comments and sharing with us your proposals. Could you 
please share the documents you mentioned below with me? Based on what you 
explained in previous email, I have some questions for clarification:
1. We both consider it necessary to have a unique content identifier for a 
content object. Our opinion in this draft is that this content ID is 
CSP-unique. Could you please explain more about the uniqueness of the UID 
in your proposal?
2. By applying the mechanism you proposed below, we can guarantee the 
content is uniquely stored in one CDN. And according to my analysis on the 
basis of your explanation, the content replication is checked by comparing 
the CURL which is similar to the current mechanism in CDNi work, correct 
me if my understanding is wrong. If so, when redirection occurs between 
uCDNs and dCDN, the URL would be changed and thus be different with the 
original one - CURL. And if a dCDN is connected with two different uCDNs 
and two end users request the same content from these two uCDNs 
respectively, the two uCDNs may generate different redirection URLs to the 
dCDN. In this case we think the content duplication issue still exists. 
How do you think? If my understanding is wrong, could you please explain 
more on how to consider this issue by using your proposal?

Thank you again~

[email-masked] 写于 2012-08-01 02:00:23:

> We have done some work in the area of de-duplication of content 
> (Please see Th. Zahariadis, E. Quacchio, “Fast content-aware 
> delivery in overlay networks,” IEEE COMSOC MMTC E-Letter, Vol.6, No.
> 7, July 2011, pp. 44-47))
> 
> Actually, we believe that it is needed to define Unique IDs (UID) 
> per content object/chunk in order to: 
> a) Avoid extended data replications at the CDN level and minimize 
> the load to find the correct object.
> b) Detect and retrieve very fast the content. This should be fast 
> enough to allow even seamless real-time video streaming by 
> retrieving video chunks 
> c) Be backwards compatible with todays’ URLs (in case the 
> content/chunk is not found, you should be able to go to the original 
site).
> 
> In order to meet the (a), UID should be always associated with the 
> content object itself (e.g. encapsulated in the object) or be based 
> on unique characteristics of the content object (e.g. a set of low 
> level descriptors). However, (b), poses that this could be 
> calculated once (or sometimes), but should not be calculated or 
> generated each time the object is requested; instead it should be 
> “carried” and “extracted” in most cases.  On the other hand, due to 
> backwards compatibility needs (c), we should not change the standard
> file format (e.g. we could not encapsulate UID or low level 
> One solution could be to create a wrapper that would encapsulate the
> UID whenever the content object enters the CDN and extracts that at 
> the time that the content object leaves the CDN, but this would 
> increase the complexity and processing time.
> 
> Instead, we propose to use as UID a formal file name format. UID 
> will be a string concatenation in the format: 
> 
> CM-CID-filename.ext 
> 
> where:
> • CM is a “Content Marker” 
> • CID is a content signature, which could be a self-certifying 
> identifier e.g. MD5 or SHA-1 based-hash function on the file's 
> content or even a combination of searchable low level descriptors. ,
> which guarantees an easy and fast detection that this name is a UID
> • the original filename, which is used for making UID easily 
> recognized by humans and avoid complex self-certifying names.
> 
> Whenever new content is published or stored at the CDN, the object 
> filename could be renamed to a UID and the URL to a Content URL 
> (CURL) with the 
> following format: 
> 
> http://www. website.com/…/CM-CID-filename.ext
> 
> The CURL is a URI, which is designed to enable caching mechanisms 
> and trigger CDN-related functionality (e.g. accessing the CDN 
> overlay network and querying for locally or “nearby” cached 
> content). It should also be emphasized that from the CURL, the 
> original URL and the UID may be easily extracted, while on the other
> hand it is fully backwards compatible with existing browsers and no 
> modifications are needed. In this way, we may seamlessly support any
> kind of data available in the Internet (text, images, various types 
> of audio or video), while in case the content object is not cached 
> in the CDN, we can always go back to the original source URL.
> 
> Best regards,
> Theodore
> 
> 
> 
> To: i-d-announce at ietf.org 
> Subject: I-D Action: 
> From: internet-drafts at ietf.org 
> Date: Sun, 29 Jul 2012 19:29:14 -0700 
> Delivered-to: i-d-announce at ietfa.amsl.com 
> List-archive: <http://www.ietf.org/mail-archive/web/i-d-announce> 
> List-help: <mailto:[email-masked]?subject=help> 
> List-id: Internet Draft Announcements only <i-d-announce.ietf.org> 
> List-post: <mailto:[email-masked]> 
> List-subscribe: <https://www.ietf.org/mailman/listinfo/i-d-announce>, <
> mailto:[email-masked]?subject=subscribe> 
> List-unsubscribe: <https://www.ietf.org/mailman/options/i-d-announce>, <
> mailto:[email-masked]?subject=unsubscribe> 
> Reply-to: internet-drafts at ietf.org 
> 
> https://datatracker.ietf.org/doc/draft-jin-cdni-content-
> deduplication-optimization
> 
> http://tools.ietf.org/html/draft-jin-cdni-content-deduplication-
> optimization-02
> 
> http://tools.ietf.org/rfcdiff?url2=draft-jin-cdni-content-
> deduplication-optimization-02
> 
> 
>  _______________________________________________
> CDNi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cdni