Subject: Re: NFSv4 Replication and Migration: design team conference call/Draft

From: "Juan Gomez" <[email-masked]>
Date: Sun, 2 Jun 2002 11:14:50 -0700

Rob,

I think I was looking at the paper more from the point of view of
compression: recognizing similar blocks in a filesystem may help us a lot
in compressing what we have to send during a migration, I do not have
numbers but I think we may expect reasonable block replication on a given
filesytem, enough to be worth exploring.

Also in replication, I think we should have some protocol capable of
shipping only new blocks (i.e. updates) and not the old ones already
present in the other servers. I am not sure if we want to consider the
complications associated with this as it requires persistent storage of
replication-related metadata.

Juan

                      Robert Thurlow                                                                       
                      <Robert.Thurlow@e        To:       Juan Gomez/Almaden/IBM@IBMUS                      
                      ng.sun.com>              cc:       [email-masked]                      
                                               Subject:  Re: NFSv4 Replication and Migration: design team  
                      06/03/02 08:47 AM         conference call/Draft                                      
                      Please respond to                                                                    
                      Robert Thurlow                                                                       

> 1.-Do we want to provide for failure recovery in the
migration/replication
> protocol? I know that in most cases we should be sending differential
> changes to the filesystems but I keep thinking of the case were we have a
> full (intial) migration/replication of a file system and one side
involved
> in the process fails; do we want to have some type of migration
> checkpointing?

Yes; I believe that restartability in this case is a requirement.

> 2.-I think striving for a efficient bandwith utilization is a great idea
so
> I wanted to make the group aware of recent work in this area that may
> relevant to the design of a a bandwith efficient migration replication
> protocol:
>
> http://www.cs.ucsd.edu/sosp01/papers/mazieres.pdf
>
> I think the bandwith-saving ideas presented in this paper may work great
in
> migration/replication. What do you guys think?

This is an interesting paper I had not known about, but I believe
it is really only applicable with conventional filesystems traffic.
With replication/migration, you have a largely unidirectional flow
of data, and you can't benefit from caching and block recognition
as the paper describes.  The main thing would seem to be to keep
the pipe full; a streaming protocol which minimizes the risk of a
stall waiting for back-traffic from destination to source would
seem to cover us here.  Compression of data could be a benefit;
are there other ideas in the paper you believe would apply?

Rob T

---

From: Robert Thurlow <[email-masked]>
Date: Wed, 5 Jun 2002 14:34:13 -0600 (MDT)

> filesytem, enough to be worth exploring.

I think compression could be worthwhile to explore.  It's not clear
that we can take advantage of similar blocks except for blocks of
zeros, which we should certainly optimize.  What amount of block
replication would you expect, and under what circumstances?

> replication-related metadata.

The protocol certainly needs to be able to send only files which have
changed, and also to send only changed regions of individual files.
Otherwise, huge files cannot be dealt with.  The draft covers this.

Rob T

---

From: Brent Callaghan <[email-masked]>
Date: Wed, 05 Jun 2002 14:12:02 -0700

> replication would you expect, and under what circumstances?

I'm not sure if I read it in the paper Juan referenced, but
you could in theory avoid some data transfers if you kept
track of each block of transferred data together with a
checksum.

Then when you need to transfer a block of new data in
a file, you checksum the block and see if you've transferred
it previously as part of another file.  If you get a hit,
then instead of transferring the block, you just tell the
other end where it is, and where it needs to go.

I've no idea whether this happens often enough to be
worthwhile - maybe I should go back and read Maziere's paper
again ...

	Brent

---

From: David Robinson <[email-masked]>
Date: Wed, 05 Jun 2002 17:02:19 -0500

> other end where it is, and where it needs to go.

Be careful here, checksums are not generally designed to
uniquely identify a block, but to detect errors. As an
example, a 2TB storage device is all that is in theory needed
to return any 512 byte block based on a 32-bit checksum.
Clearly all the world's information is not represented
by 2TB of data. By the time you get to a high enough probability
of no error, the size of the checksum is large enough that
it is probably better to just use a lossless compression
algorithm.

	-David

---

From: Brent Callaghan <[email-masked]>
Date: Wed, 05 Jun 2002 15:31:27 -0700

> it is probably better to just use a lossless compression

Beg to differ.

I can think of two examples of checksums being used to
identify data blocks.

The rsync protocol (and program) use a 32-bit rolling
checksum to identify blocks - and a 128 bit MD4 to
verify them.

	http://www.rsync.org

Lucent's Venti filesystem uses a 160 bit SHA1 hash
to identify blocks.  You can read the paper here:

http://www.usenix.org/publications/library/proceedings/fast02/quinlan.html

Where they justify its use.

In both cases, the saving in data volume far exceeds
anything you can do with lossless compression.

	Brent