Subject: Re: [decade] Updated DECADE Charter

From: Wesley Leggette <[email-masked]>
Date: Mon, 1 Feb 2010 11:35:00 -0600

On 1/31/10 23:52, "Richard Alimi" <[email-masked]> wrote:

> 
> I do realize that you are asking these questions in the context of a more
> general distributed storage protocol, and that my answers are focused on the
> use cases for DECADE.  I'm basically trying to illustrate how much simpler the
> protocol design can be with our original goal.

I agree that a simple WORM system for P2P application-specific object
storage is much simpler to design. I'm glad to see that the scope and use
cases are going to be constrained as, yes, my point was that to not limit
the scope would would vastly complicate the requirements.

What I think is interesting is that this is really meant as an in-network
storage for application-specific use. This is to say, while it is meant for
generic storage, it is not meant for generic cross-application *access*.
After realizing this, looking back over the use cases, they were a lot more
clear to me.

I've come up with the following analogy which has helped me to understand
the scope of DECADE. While I don't have any more specific comments, I share
it here in the hopes of helping anyone else who is similarly confused:

I now view the scope of DECADE as a sort of "one-way" approach that is
counter-intuitive when you think of it in terms of network storage protocols
such as CIFS, WebDAV, and NFS, which all have a fixed data model (the "file
system paradigm") and are meant for various clients to be able to understand
what the data is to some degree (as, at least, files, if not the format of
the files themselves).

When you think of this in terms of more general "cloud storage" such as
Amazon S3, the picture becomes a bit clearer. In that sort of system, the
point isn't to define a data model for client interoperability, just to
provide a shared mechanism for storing data. It is up to the application
written on top to interpret how the data fits together when it is accessed.

This is probably all very obvious to most of you, of course, but as P2P and
"cloud storage" is still fairly new, it's probably worth mentioning at some
level.

>> While I'm not sure that DECADE wants to deal with any of these issues, it
>> could very well be worth creating a second working group to deal with
>> general purpose distributed storage system.
> 
> Personally, I would actually be very interested to follow such a working group
> :)

I'll make sure to keep this list informed.

I should note that we fully embrace the sort of "object-based",
"no-paradigm" storage DECADE is all about, but my goal with another working
group would be to build a slightly more complicated base storage protocol
which would allow vendors to build (or simulate) more traditional access
patterns like "file system" and "block device" on top of it. Not the scope
of DECADE, but that is fine.

Wesley Leggette
Cleversafe, Inc.

---

From: "David A. Bryan" <[email-masked]>
Date: Mon, 1 Feb 2010 13:11:41 -0500

Hi Wesley,

A little bit off-topic, but sort of wanted to close loop on this topic.

I think I understand what might have been a bit confusing about DECADE
to you (and likely others) after reading your post.

Unfortunately, now I'm a bit confused on what you are thinking about
for this other group/proposal you mentioned. You mentioned a set of
primitives in a storage protocol that would allow you to build a
distributed file system. I guess my question is how this differs from
NFS? Would the idea be that once you built this new set of primitives,
NFS (and other things) could be built on top? If so, what applications
(other than NFS, and maybe an implementation of whatever DECADE ends
up being, I suppose) are you imagining such a protocol being used to
implement?

Thanks,

David

On Mon, Feb 1, 2010 at 12:35 PM, Wesley Leggette
<[email-masked]> wrote:
> _______________________________________________
> decade mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/decade
>

---

From: Wesley Leggette <[email-masked]>
Date: Mon, 1 Feb 2010 13:51:55 -0600

> up being, I suppose) are you imagining such a protocol being used to

Right, exactly. NFS is meant to be a "file system" protocol, and as such has
specific semantics implicit in that. For example, when you write a file, the
linking of the file to the directory happens for you. File renames and
deletes are atomic.

For this stuff to occur, you need to perform locking or transactions (either
one or a combination of both will generally work). Thus, CIFS and NFS (and
WebDAV) work fine when the "access point" (the NFS server) is a single
system, because it can perform the locks internally.

Now, let's say that you want to have multiple access points. You want to
allow file creates and deletes (which require directory metadata changes)
simultaneously on the different systems. They still need to perform locking
or transactions (so that creating, renaming, or deleting a file works
correctly), so they need to cooperate to do this.

pNFS nominally does this, but it does not seem to (unless I am missing
something) define the locking mechanism (presumably some server), so
interoperability (between servers, not between clients) is affected.

Further, pNFS does handle in a form a "distributed file system" protocol
(and the interoperability stuff is easily addressed), but what if you wanted
something else? What about block storage? What about distributed queue and
message services? What about a distributed database? What about a object
storage systems where the server assigns the object name for either content
addressable storage or to make lookups faster? And what about handling
replication, multiple access points, and what if you want to federate the
storage to increase scalability (maybe a protocol that allows indirection
between the object name and the remote address where the object is
accessible from -- much like P2P systems do today)?

In any distributed system, you're not going to get atomic operations, of
course. But if the goal is to write some sort of application that will mimic
other storage paradigms (block, file system, queues) there are typically a
fairly small set of primitive guarantees that you need. If there is a
general storage protocol that allows you to store data and provides some
basic distributed computing primitives, that gives you a lot of flexibility
and provides a platform on which higher-level protocols could be written.

Wesley Leggette