Subject: RE: NFSv4 Replication and Migration: design team conference call/ Draft

From: "Yoder, Alan" <[email-masked]>
Date: Wed, 5 Jun 2002 17:22:00 -0700

If backing up desktop filesystems is envisioned
as a use of this protocol, then there is a huge
amount of replicated data out there.  I give you
Microsoft Office.

Alan

===============================================================
Alan G. Yoder                                    [email-masked]
Network Appliance, Inc.                           
Sunnyvale, CA                                      408-822-6919
=============================================================== 

> -----Original Message-----
> From: Juan Gomez [mailto:[email-masked]]
> Sent: Wednesday, June 05, 2002 5:07 PM
> To: Brent Callaghan
> Cc: [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: NFSv4 Replication and Migration: design team conference
> call/Draft
> 
> 
> Brent,
> 
> This is in fact in the paper, I think the issue we need to 
> weight is how
> often we use this in real life filesystems. The paper is 
> focused on some
> very spciela cases that favor the algorithm.
> Perhaps we may run a replication search algorithm on 
> filesystemns and find
> that similar blocks are only 1% and then this is a hopeless 
> alternative. It
> would be nice if we could have estimates of replicated blocks in a
> filesystem we need to migrate and use our block-matching 
> algorithm on cases
> where it is promissing to do so this may be another option to 
> explore. I
> think the chunking/hashing algorithms proposed on the paper may cheap
> enough to predetermine block replication at a resonable CPU 
> cost....all
> this we may want to explore.
> 
> 
> Juan
> 
> 
> 
> |---------+---------------------------->
> |         |           Brent Callaghan  |
> |         |           <[email-masked]|
> |         |           m>               |
> |         |           Sent by:         |
> |         |           Brent.Callaghan@e|
> |         |           ng.sun.com       |
> |         |                            |
> |         |                            |
> |         |           06/05/02 02:12 PM|
> |         |                            |
> |---------+---------------------------->
>   
> >-------------------------------------------------------------
> -----------------------------------------------------------|
>   |                                                           
>                                                              |
>   |       To:       Robert Thurlow 
> <[email-masked]>                                  
>                           |
>   |       cc:       Juan Gomez/Almaden/IBM@IBMUS, 
> [email-masked]                                  
>            |
>   |       Subject:  Re: NFSv4 Replication and Migration: 
> design team conference call/Draft                                 |
>   |                                                           
>                                                              |
>   |                                                           
>                                                              |
>   
> >-------------------------------------------------------------
> -----------------------------------------------------------|
> 
> 
> 
> 
> > I think compression could be worthwhile to explore.  It's not clear
> > that we can take advantage of similar blocks except for blocks of
> > zeros, which we should certainly optimize.  What amount of block
> > replication would you expect, and under what circumstances?
> 
> I'm not sure if I read it in the paper Juan referenced, but
> you could in theory avoid some data transfers if you kept
> track of each block of transferred data together with a
> checksum.
> 
> Then when you need to transfer a block of new data in
> a file, you checksum the block and see if you've transferred
> it previously as part of another file.  If you get a hit,
> then instead of transferring the block, you just tell the
> other end where it is, and where it needs to go.
> 
> I've no idea whether this happens often enough to be
> worthwhile - maybe I should go back and read Maziere's paper
> again ...
> 
>              Brent
> 
> 
> 
>

---

From: Brent Callaghan <[email-masked]>
Date: Wed, 05 Jun 2002 17:37:48 -0700

Although "push" technology is officially out of fashion amongst
the Internet cognoscienti, I remember being quite impressed by
Marimba's Castanet protocol for file distribution and replication.

It's built on a model of a central server and a large number
of replicas that need to be maintained by having file differences
transferred.  Their protocol also uses some sophisticated block
hashing to avoid transferring identical data blocks.

The protocol eventually found its way to the W3C, where
it morphed into the "HTTP Distribution and Replication
Protocol".  There's a description of the protocol on the
W3C website:

	http://www.w3.org/TR/NOTE-drp

Their model, based on file indexes, isn't quite what we're
looking at for this protocol.  But the protocol's use of
checksums to identify previously downloaded data might be
interesting.

	Brent