Subject: [nfsv4] Question on file layout!

From: "Suchit Kaura" <[email-masked]>
Date: Thu, 18 Oct 2007 15:50:45 -0700

I have been reading the spec and on page 272:

http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-13#section-14

The following are very confusing if not inconsistent, I would really
appreciate your taking a few moments to look through and explain the
same: 

    The number of elements in nfla_stripe_indices is always equal to the
stripe count.

   4.  nfl_fh_list.  An array of data server filehandles for each list
       of data servers in each element of the nflda_multipath_ds_list
       array.  The number of elements in nfl_fh_list MUST be of three
       values:

...

       *  The stripe count, i.e. the number of elements as
          nflda_multipath_ds_list.  When issuing an I/O to any data
          server in nfla_multipath_ds_list[X], the filehandle in
          nfl_fh_list[X] MUST be used.

In the following example:

      nflda_multipath_ds_list<> = { A, B, C, D }, { E }, { F, G }
stipe count = 3 ?

      nflda_stripe_indices<> = { 2, 0, 1, 0 }
stripe count = 4? The example seems to take this route so the file
handle list should have 4 elements but it has 

      nfl_fh_list = { 0x36, 0x87, 0x67 }.
3 elements?

Regards,

Suchit

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Robert Gordon <[email-masked]>
Date: Fri, 19 Oct 2007 02:44:10 -0500

>       nfl_fh_list = { 0x36, 0x87, 0x67 }.

Suchit; It is a know draft issue and (not speaking for) the editors
know about it.

the example, of course, should read something like this.

       nflda_stripe_indices<> = { 2, 0, 1}

I put together a couple of pretty pictures over here in an effort to
help illustrate.

http://blogs.sun.com/macrbg/

Robert.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 19 Oct 2007 07:30:56 -0700 (PDT)

> appreciate your taking a few moments to look through and explain the

Suchit,

You are absolutely correct: it is inconsistent and confusing.

The problem is that stripe_count is being defined two ways:

- the length of index array
- the length data server array

The former is the correct definition.

Thus the valid values for the length of nfl_fh_list
are zero, one, or the length of nflda_multipath_ds_list.

I will fix this now.

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Fri, 19 Oct 2007 13:07:09 -0400

> The former is the correct definition.

isn't it requried that the length of the index array = length of the
dataserver array = length of nfl_fh_list (if nfl_fh_list is not 0 or 1)?

-->Andy

---

From: Robert Gordon <[email-masked]>
Date: Fri, 19 Oct 2007 12:20:46 -0500

> dataserver array = length of nfl_fh_list (if nfl_fh_list is not 0  

That's certainly what i understood.

Robert.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 19 Oct 2007 10:37:44 -0700 (PDT)

> dataserver array = length of nfl_fh_list (if nfl_fh_list is not 0 or

No. If it were, there would be no point in having a separate index
array
and data server array. Actually there would be no point in 
having an index array at all. Keep in mind the index and ds
arrays are in the same data structure: the device address.

It is certainly RECOMMENDED that the index array be at least as long
as the data server array. Otherwise the data address has ds addresses
that will never be accessed, and this would be wasteful.

http://www1.ietf.org/mail-archive/web/nfsv4/current/msg04360.html

In the link you see an example that shows the index and ds array's
of different length. The same example that is in the I-D now, which
Suchit reproduced.

There is even a spreadsheet in that email that encodes the algorithm.
If Excel can do it, pNFS servers can do it.

---

From: "dean hildebrand" <[email-masked]>
Date: Fri, 19 Oct 2007 10:45:49 -0700

> >> The former is the correct definition.
The *correct* yet abstract definition is on page 276:
   Stripe Count.  A stripe count is the number of stripe units in a pattern.

To me, this means that the stripe count is the number of data servers
in a pattern where each data server can appear multiple times.

The i.e. part of the following line is wrong and should be either
removed or changed to indicate indices list.
       *  The stripe count, i.e. the number of elements as
nflda_multipath_ds_list.

> > dataserver array = length of nfl_fh_list (if nfl_fh_list is not 0

So I would say no,  the index array creates the repeatable pattern
which may have each data server listed multiple times, e.g., {2, 0, 1,
0}.  But this raises an interesting question of whether the size(index
array) must be >= size(data server array)  I would say no, I don't see
why it should have to use all of the devices.

Dean

---

From: Lisa Week <[email-masked]>
Date: Fri, 19 Oct 2007 19:27:09 -0600

> If Excel can do it, pNFS servers can do it.

Hi,

(I apologize in advance for being so long winded...)

One thing that has been confusing to me (and I have just noticed this  
now after studying it harder) is that the example in "Section 14.4 -  
Interpreting the File Layout" of Draft 14 (http://www.nfsv4- 
editor.org/draft-14/draft-ietf-nfsv4- 
minorversion1-14.html#file_layout_interpret) just implicitly assumes  
we are using SPARSE packing.  Actually, there would be some bad side  
effects if it were using DENSE packing in that situation.  That is as  
follows:

The following table is from Section 14.4 of Draft 14 - If the example  
was using DENSE packing we'd be trashing SU1 with SU3 data (or vice  
versa), SU5 with SU7 data (or vice versa) and so on...  Because we  
are using the same file handle for multiple stripe units on the same  
data servers.

                     +-----+------------+--------------+
                     | SUi | filehandle | data servers |
                     +-----+------------+--------------+
                     | 0   | 87         | E            |
                     | 1   | 36         | A,B,C,D      |
                     | 2   | 67         | F,G          |
                     | 3   | 36         | A,B,C,D      |
                     | 4   | 87         | E            |
                     | 5   | 36         | A,B,C,D      |
                     | 6   | 67         | F,G          |
                     | 7   | 36         | A,B,C,D      |
                     | 8   | 87         | E            |
                     | 9   | 36         | A,B,C,D      |
                     | 10  | 67         | F,G          |
                     | 11  | 36         | A,B,C,D      |
                     | 12  | 87         | E            |
                     +-----+------------+--------------+

I explain more about some of the subtleties of this assumption and of  
SPARSE vs. DENSE packing below...

----

Further, in Draft-11 (this was prior to the new layout structures  
being introduced) we had the following text:
(Quoted from draft 11, section 13.4)
"A data server that has more than one stripe unit of a pattern to  
store each unit may store those stripes in different files, but to do  
so, will need unique filehandles in the data server list, as the  
previous example showed. While data servers may be repeated multiple  
times within the flattened data server list, if a STRIPE4_DENSE  
stripe type is used (see Section 13.5 (Sparse and Dense Stripe Unit  
Packing)), the same filehandle MUST NOT be used on the same data  
server for different stripe units of the same file.

A data file stored on a data server MUST map to a single file as  
defined by the metadata server; i.e., data from two files as viewed  
by the metadata server MUST NOT be stored within the same data file  
on any data server."

-------

The important part of that text is "if a STRIPE4_DENSE stripe type is  
used (see Section 13.5 (Sparse and Dense Stripe Unit Packing)), the  
same filehandle MUST NOT be used on the same data server for  
different stripe units of the same file."

This text appears to have been removed in draft-12 and later.

I don't recall if there was an explicit decision that this text/ 
requirement ("if a STRIPE4_DENSE stripe type is used, the same  
filehandle MUST NOT be used on the same data server for different  
stripe units of the same file.") should be removed.  Mike (or anyone  
else), do you remember?

Anyway... it is not so much the text being removed that I am worried  
about.  I am more worried about requirements it used to put on the  
layout structures.  That text effectively required us to be able to  
assign a different file handle to each stripe unit of a file's  
striping pattern.  Most importantly, if a file's striping pattern  
defines there to be multiple different stripe units stored on the  
same data server we need the capability to use unique file handles to  
access each stripe unit.

With the current layout structure (in Draft-14) I believe we have  
lost that capability.

Another way to say it is:
I interpret the lines 15651 - 15666 (as quoted from Draft-14) to mean  
that from a single data server's point of view, all stripe units in a  
single file's striping pattern stored on that data server will be  
accessed with the same file handle.

  15651	   4.  nfl_fh_list.  An array of data server filehandles for  
each list
  15652	       of data servers in each element of the  
nflda_multipath_ds_list
  15653	       array.  The number of elements in nfl_fh_list MUST be  
of three
  15654	       values:
  15655	
  15656	       *  Zero.  This means that filehandles used for each  
data server
  15657	          are the same as the filehandle returned by the OPEN  
operation
  15658	          from the metadata server.
  15659
  15660	       *  One. This means that every data server uses the same
  15661	          filehandle: what is specified in nfl_fh_list[0].
  15662	
  15663	       *  The stripe count, i.e. the number of elements as
  15664	          nflda_multipath_ds_list.  When issuing an I/O to  
any data
  15665	          server in nfla_multipath_ds_list[X], the filehandle in
  15666	          nfl_fh_list[X] MUST be used.

Therefore, you can't have a repeated value in the  
nflda_stripe_indicies list (e.g. nflda_stripe_indicies = {2, 0, 1,  
0}) if you are using DENSE packing.

Thanks,
Lisa

---

From: Mike Eisler <[email-masked]>
Date: Fri, 19 Oct 2007 21:40:36 -0700 (PDT)

> are using the same file handle for multiple stripe units on the same 

Yes, you are correct, this is broken.

[...]

> else), do you remember?

I don't recall such an explicit decision. Without looking at earlier
drafts, I suspect it got wiped out as part of the SIMPLE/COMPLEX
purge.

> With the current layout structure (in Draft-14) I believe we have  

Agreed. I think this may account for the inconsistency Suchit
discovered.

So what to do?

The simplest is to re-define nfl_fh_list to be the
same length as nflda_stripe_indices and have nfl_fh_list[X]
correspond to nflda_stripe_indices[X].

However, for SPARSE packing, combined with long index arrays,
this makes layouts not very economical.

So I think what we want is for SPARSE packing,
state that fh list to corresponds to the ds list
(same number of elements as the ds list, one element, or
zero elements in the fh list as in draft -14), and for
DENSE packing state that the fh list must be the same
length as the index array, and fh list[X] corresponds to indices[X].

The algorithm for determining the file handle in the DENSE packing
case is thus:

   stripe_count = number of elements in nflda_stripe_indices;

   fh_idx = 
     ( SUi + nfl_first_stripe_index ) % stripe_count;

   fh = nfl_fh_list[fh_idx];
   
For dense packing, we use the same example, with the same
index and data server arrays, but change the fh list from,

          36 87 67
to,

          67 37	87 36
     
The result is:

SUi     fh      data servers
0	87	E
1	36	A,B,C,D
2	67	F,G
3	37	A,B,C,D
4	87	E
5	36	A,B,C,D
6	67	F,G
7	37	A,B,C,D
8	87	E
9	36	A,B,C,D
10	67	F,G
11	37	A,B,C,D
12	87	E

---

From: Robert Gordon <[email-masked]>
Date: Sat, 20 Oct 2007 00:23:29 -0500

> length as the index array, and fh list[X] corresponds to indices[X].

yes.

Robert.

---

From: Lisa Week <[email-masked]>
Date: Sat, 20 Oct 2007 00:06:45 -0600

> drafts, I suspect it got wiped out as part of the SIMPLE/COMPLEX

Okay.  That is kind of what I expected.  It seems like this text is  
important enough to put back in.

> index and data server arrays, but change the fh list from,

Yes, I think this is a good solution.

Thanks,
Lisa

---

From: "Suchit Kaura" <[email-masked]>
Date: Sat, 20 Oct 2007 06:38:54 -0700

Hi,
 
First, I would like to thank everybody for the responses. The
clarifications help a lot.
 
I definitely agree with Lisa about the dense layout issue, in fact the
formula supplied in the example would not work 

The following was supposed to be the last part of the email.

---------

The formula for the dense layout is incorrect/inconsistent too as it we
will soon start overwriting the data in fh = 0x36 for ...

The calculation to determine the octet offset within the data file
   for dense data server layouts is:

      stripe_width = stripe_unit_size * N;
         where N = number of elements in nflda_stripe_indices.

      data_file_offset = floor(file_offset / stripe_width)
         * stripe_unit_size
         + file_offset % stripe_unit_size

-------------

In general I would like to ask a question about whether we allow the
flexibility to have a different non overlapping patterns within the same
file. 

Thus, conceptually do we imagine a layout having a pattern from let's
say starting offset in user file O till a length L. 

Can the file then have a different pattern from offset O1 till a length
L1 ? 

I was on vacation yesterday hence the delay in responding. Once again
thanks a lot ...

Regards,

Suchit

________________________________

From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, October 19, 2007 6:27 PM
To: [email-masked]
Cc: William A. (Andy) Adamson; Suchit Kaura; [email-masked]
Subject: Re: [nfsv4] Question on file layout!

	

		On 10/19/07, Mike Eisler <[email-masked]>
wrote:

			You are absolutely correct: it is inconsistent
and confusing.

			The problem is that stripe_count is being
defined two ways:

		

		isn't it requried that the length of the index array =
length of the
		dataserver array = length of nfl_fh_list (if nfl_fh_list
is not 0 or
		1)?

	No. If it were, there would be no point in having a separate
index
	array
	and data server array. Actually there would be no point in 
	having an index array at all. Keep in mind the index and ds
	arrays are in the same data structure: the device address.

	It is certainly RECOMMENDED that the index array be at least as
long
	as the data server array. Otherwise the data address has ds
addresses
	that will never be accessed, and this would be wasteful.

	In the link you see an example that shows the index and ds
array's
	of different length. The same example that is in the I-D now,
which
	Suchit reproduced.

	There is even a spreadsheet in that email that encodes the
algorithm.
	If Excel can do it, pNFS servers can do it.

Hi,

One thing that has been confusing to me (and I have just noticed this
now after studying it harder) is that the example in "Section 14.4 -
Interpreting the File Layout" of Draft 14
(http://www.nfsv4-editor.org/draft-14/draft-ietf-nfsv4-minorversion1-14.
html#file_layout_interpret) just implicitly assumes we are using SPARSE
packing.  Actually, there would be some bad side effects if it were
using DENSE packing in that situation.  That is as follows:

The following table is from Section 14.4 of Draft 14 - If the example
was using DENSE packing we'd be trashing SU1 with SU3 data (or vice
versa), SU5 with SU7 data (or vice versa) and so on...  Because we are
using the same file handle for multiple stripe units on the same data
servers.

----

Further, in Draft-11 (this was prior to the new layout structures being
introduced) we had the following text:
(Quoted from draft 11, section 13.4)
"A data server that has more than one stripe unit of a pattern to store
each unit may store those stripes in different files, but to do so, will
need unique filehandles in the data server list, as the previous example
showed. While data servers may be repeated multiple times within the
flattened data server list, if a STRIPE4_DENSE stripe type is used (see
Section 13.5 (Sparse and Dense Stripe Unit Packing)), the same
filehandle MUST NOT be used on the same data server for different stripe
units of the same file.

A data file stored on a data server MUST map to a single file as defined
by the metadata server; i.e., data from two files as viewed by the
metadata server MUST NOT be stored within the same data file on any data
server."

-------

The important part of that text is "if a STRIPE4_DENSE stripe type is
used (see Section 13.5 (Sparse and Dense Stripe Unit Packing)), the same
filehandle MUST NOT be used on the same data server for different stripe
units of the same file."

I don't recall if there was an explicit decision that this
text/requirement ("if a STRIPE4_DENSE stripe type is used, the same
filehandle MUST NOT be used on the same data server for different stripe
units of the same file.") should be removed.  Mike (or anyone else), do
you remember?

Anyway... it is not so much the text being removed that I am worried
about.  I am more worried about requirements it used to put on the
layout structures.  That text effectively required us to be able to
assign a different file handle to each stripe unit of a file's striping
pattern.  Most importantly, if a file's striping pattern defines there
to be multiple different stripe units stored on the same data server we
need the capability to use unique file handles to access each stripe
unit.

With the current layout structure (in Draft-14) I believe we have lost
that capability.  

 15651   4.  nfl_fh_list.  An array of data server filehandles for each
list
 15652       of data servers in each element of the
nflda_multipath_ds_list
 15653       array.  The number of elements in nfl_fh_list MUST be of
three
 15654       values:
 15655 
 15656       *  Zero.  This means that filehandles used for each data
server
 15657           are the same as the filehandle returned by the OPEN
operation
 15658           from the metadata server.
 15659
 15660       *  One. This means that every data server uses the same
 15661           filehandle: what is specified in nfl_fh_list[0].
 15662 
 15663       *  The stripe count, i.e. the number of elements as
 15664           nflda_multipath_ds_list.  When issuing an I/O to any
data
 15665           server in nfla_multipath_ds_list[X], the filehandle in
 15666           nfl_fh_list[X] MUST be used.

Therefore, you can't have a repeated value in the nflda_stripe_indicies
list (e.g. nflda_stripe_indicies = {2, 0, 1, 0}) if you are using DENSE
packing.

Thanks,
Lisa

		-->Andy

			are zero, one, or the length of
nflda_multipath_ds_list.

			I will fix this now.

---

From: Mike Eisler <[email-masked]>
Date: Sat, 20 Oct 2007 07:08:05 -0700 (PDT)

> Can the file then have a different pattern from offset O1 till a

The file layout specification does not explicitly support this.
A smarter pNFS server with smarter data servers could presumably
map the incoming tuple of <fh, offset, length> into any
local combination of one or tuples of 
     <offset(1), length(1)>, ..., <offset(N), length(N)>

where the sum of length(1) ... length(N) == length,
and there is no i, and j, such that <offset(i), length(i)>
overlaps <offset(j), length(j)>.

There's been no discussion (that I recall) arguing for 
explicit support (i.e. placing the intelligence in
the pNFS client) if such a packing method, and I think it is
too late for NFSv4.1 to consider it.

(BTW, I think with sparse layouts, one could implement dense packing
on the server by remapping the offsets, but again, the data servers
would have to be smarter.)

As you can see we have difficulties supporting two packing methods
as it is now.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 23 Oct 2007 17:21:47 -0700 (PDT)

I've committed changes to drsft-15 to implement the below proposal,
as well as emphasizing the need for dense packing to use different
file handles.

---

From: Lisa Week <[email-masked]>
Date: Tue, 23 Oct 2007 21:02:26 -0600

> as well as emphasizing the need for dense packing to use different

Great, thanks.

- Lisa