Subject: RE: [nfsv4] New pNFS ID available

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 1 Aug 2005 14:19:46 -0400

Dean Hildebrand wrote:
> The metadata server must accept NFSv4 I/O, BUT, must it accept NFSv4
I/O
> while a client has a layout?  I think it should be stated in the spec
that
> a client should not issue NFSv4 I/O while it has a layout for a file.

I think such a restriction could be imposed without contravening the 
minor versioning rules.  On the other hand, such a restriction is not
in effect right now and I don't really see the reason to impose it.

> It
> is then an easy check for the server to see if the client has a layout
or
> not, and where to direct the I/O request.

I think you need a lot more to effect this then simply saying the client
should not issue the I/O.  I think you have to state that the server
should (must?) return an error if it does and define an error to be 
returned.

It seems that what you are doing is checking the filehandle on a per-op 
basis (and you then still have the issue that a single request can
encompass
multiple ops in which the role is different for different ops).  This 
is very similar to distinguishing the two server by having different
fh's
in the layout.  This would allow the same fh to be used, but otherwise 
would be the same.

To get back to the original question, what is wrong with separate IP's
(and thus separate sessions)?  Why is it preferable to impose this
restriction as opposed to saying that the metadata server and the data
server must be separately addressable (even if they use the same
hardware)? 

> Is there an use case that would
> warrant it?  I can't think of one.

It seems like there is extensive serialization that you have to add 
when you are getting and returning layouts.  I have IO's outstanding
to a file and need to either wait for them to finish before getting a 
layout or have to be prepared to retry them if they fail.  Similarly,
when returning a layout.  As the spec is now, you simply have to wait
for IO done under the layout to be done before returning it, but with
this change, you would not be able to do any IO while the layout 
return was outstanding.  

One other issue is that you may not be able to get a layout for one
region (due to conflicts) while another region (possibly very large)
may be fine.  With your rule, you would have to give back the non-
conflicted layout region (with possible negative performance 
consequences) in order to do IO on the file in a region in which 
getting a layout is difficult.

I would turn your question around.  What is the motivation for this
proposed restriction?  If it is simply to allow a metadata server
and a data server to share an IP (and be distinguishable), I would
argue that we should be looking for less disruptive ways of doing
that.

-----Original Message-----
From: Dean Hildebrand [mailto:[email-masked]] 
Sent: Monday, August 01, 2005 1:23 PM
To: Noveck, Dave
Cc: Goodson, Garth; Marc Eshel; [email-masked]; [email-masked]
Subject: RE: [nfsv4] New pNFS ID available

> > > I know a single metadata server but I did not see anything that
will
>
> > > preclude it from also acting as a data server.
> >
> > You are correct.  It is a requirement that data can also be sent to
> the
> > metadata server through normal V4 commands.  Additionally, there
isn't
>
> > anything precluding the metadata server from naming itself in a
layout
>
> > to be used as a data server.
>
> The potential problem I see with naming yourself as a data server is
> that
> it makes it very hard to implement or even make much sense of cases
> where
> the spec tells you that the data server can do one set of things and
the
> metadata server a different set of things.
>
> I think to distinguish two cases.  One is that the same CPU's, memory,
> disks, network adapters are used to implement both a metadata server
> and a data server.  I don't see any problem with that.  If the
metadata
> server's IP is meta.pnfs.netapp.com and it puts a
data1.pnfs.netapp.com
> in a layout and those two addresses are really the same pile of
> hardware,
> then you have no problem.  For any request you receive, you know
whether
> it is for the metadata server or the data server and the spec tells
you
> what you can and must do.
>
> If you put meta.pnfs.netapp.com in the layout, things get much worse.
> For each op, you  might be able to make the distinction based on the
> current filehandle (assuming the layout fh's are different), but the
> prospect of a request in which some of the ops are directed at the
> metadata server and others at the data server strikes me as quite
> unappetizing.
>
> As there some reason distinct IP's will not work?  If you do need the
> same IP, I would think we need some way of establishing separate
> sessions with the metadata server and the data server so that each
> request is clearly directed at one or the other.

The metadata server must accept NFSv4 I/O, BUT, must it accept NFSv4 I/O
while a client has a layout?  I think it should be stated in the spec
that
a client should not issue NFSv4 I/O while it has a layout for a file.
It
is then an easy check for the server to see if the client has a layout
or
not, and where to direct the I/O request.  Is there an use case that
would
warrant it?  I can't think of one.
Dean

> -----Original Message-----
> From: Goodson, Garth
> Sent: Thursday, July 21, 2005 9:12 PM
> To: Marc Eshel
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nfsv4] New pNFS ID available
>
>
>
>
> Marc Eshel wrote:
> > Garth Goodson <[email-masked]> wrote on 07/21/2005
05:01:20
> PM:
> >
> >
> >>>The high level requirement that I have to be able to support
cluster
> >
> > file
> >
> >>>system is that a metadata server can also act as a data server.
From
> >>>reading this document I don't see a problem doing it but if someone
> >
> > think
> >
> >>>otherwise please let me know why.
> >>
> >>The issues outlined in Dave Noveck's previous comments still remain.
> As
> >
> >
> >>far as the spec is concerned there is a single metadata server and
> only
> >>I/O ops are to be issued to data servers (the wording still exists).
> >
> >
> > I know a single metadata server but I did not see anything that will
> > preclude it from also acting as a data server.
> >
> You are correct.  It is a requirement that data can also be sent to
the
> metadata server through normal V4 commands.  Additionally, there isn't
> anything precluding the metadata server from naming itself in a layout
> to be used as a data server.
>
> >
> >>>Is there a draft for file storage protocol? where can I find it?
> >>>
> >>
> >>It is included in the doc.  Section 5, page 26.
> >
> >
> > That is what I thought but I wanted to make sure. So I don't see how
> the
> > NFS client interface with the different layout drivers and for the
> file
> > layout driver how it establishes communications with the NFS data
> server.
> > Few things need to happen before you can send a read/write request
to
> the
> > NFS data server. In other words I should be able to implement the
NFS
> file
> > base protocol using only this document but I am sure I can.
> > Marc.
>
> We don't define how the NFS client interfaces with layout drivers,
that
> is client side implementation specific and can be done in many ways
> (although arguably useful for interoperability), but I don't believe
it
> is required to build a working file's-based client.
>
> -Garth
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
>

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 1 Aug 2005 15:25:13 -0400

Mark Eshel wrote:
> I thought we already agreed that the data server does not have to
check if 
> the client is using the right fh for data server only operations. The 
> draft states that it "SHOULD" and I was asking to down grade it to 
> "should" but it is not "MUST".

My issue is one level up.  It is not that some required check cannot be 
done.  It is is rather that when the spec tells you do something (even
if it permissively phrased), it should be clear what it is telling 
you to do (or not do).  When that isn't clear or where, as in this
case, it isn't clear who or what is being addressed, I worry that we
are going to wind up with something that isn't clear and that different
people read differently (and implement differently).  The spec tells
the data server to do some stuff and the metadata server to do some
other
stuff (even it uses "SHOULD" or "should"), and if you have some code
running that is doing something or other, in order to determine whether
that code is doing the right thing it must always be clear in which role
that code is executing.  If it isn't clear, I think we are going to have

problems.

> In any case we can add a flag in the fh 
> that will indicate that it should be used only for data server
operations. 
> like you suggested, if we wanted to do more checking on the data
server.
> Using multiple ip address for the same server complicates cluster 
> management for especially for failure detection and fail over. 

I can see that.

> Is there a 
> real problem that you see other than unappetizing ?

The thing I am most worried about is a lack of clarity and consequent
interoperability issues.  If we go this way the spec should say either,

    When a given server has the role of metadata server and also that
    of data server, each operation is to be understood (for the purposes
    of interpreting spec statements about what these two servers are
    to do), as being executed based on the current file handle.  If 
    the current file handle is one derived from a layout (and it is 
    recommended that these be distinguishable), then the operation is
    be considered as executed by a data server.  Otherwise, the
operation
    is to be understood as executed by the metadata server.  This
implies
    that the client may issue and the server should properly process
    requests in which some operations are to be performed by the
metadata
    server and others by the data server.

or else,

    When a given server has the role of metadata server and also that
    of data server, each request must be understood as being directed
    at one or the other, based on the filehandle of the first current 
    filehandle value for the request.  If the first current file handle

    is one derived from a layout (and it is recommended that these be
    distinguishable), then the request is to be considered as executed 
    by a data server.  Otherwise, the operation is to be understood as 
    executed by the metadata server.  If a current filehandle is set
    that is inconsistent with that, then the error XXX should result.
    For example, if a request is directed at the data server, because
    the first current handle is from a layout, any attempt to set the
    current filehandle to be a value not from a layout should be 
    rejected.  Similarly, if the first current file handle was for a
    value not from a layout, a subsequent attempt to set the current 
    file handle to a value obtained from a layout should be rejected.

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]] 
Sent: Monday, August 01, 2005 2:44 PM
To: Noveck, Dave
Cc: Goodson, Garth; [email-masked]; [email-masked]
Subject: RE: [nfsv4] New pNFS ID available

I thought we already agreed that the data server does not have to check
if 
the client is using the right fh for data server only operations. The 
draft states that it "SHOULD" and I was asking to down grade it to 
"should" but it is not "MUST". In any case we can add a flag in the fh 
that will indicate that it should be used only for data server
operations. 
like you suggested, if we wanted to do more checking on the data server.

Using multiple ip address for the same server complicates cluster 
management for especially for failure detection and fail over. Is there
a 
real problem that you see other than unappetizing ?
Marc.

"Noveck, Dave" <[email-masked]> wrote on 08/01/2005 09:47:21 AM:

> Garth Goodson wrote:
> ...
will
isn't
layout
the
metadata
data1.pnfs.netapp.com
whether
you
05:01:20
cluster 
From 

the 

to
NFS
that 
it

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 1 Aug 2005 15:29:33 -0400

I think this is a case in which we aren't going to agree and 
we both have better things to do than try to beat one another 
into submission.  Are there other people out there who have
an opinion on this issue and would allow us to get a consensus?

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]] 
Sent: Monday, August 01, 2005 3:24 PM
To: Noveck, Dave
Cc: Goodson, Garth; [email-masked]; [email-masked]
Subject: RE: [nfsv4] New pNFS ID available

I still think that is reasonable to allow the server to invalidate the 
device list cache on the client so it can reorganize it device name
space 
and not force it to use new ids just to avoid this SIMPLE call to the 
client. Like I said it can even be implied by the existing recall all 
layouts for a given fsid.
Marc.

[email-masked] wrote on 08/01/2005 10:48:54 AM:

> Marc Eshel wrote:
> > It will take a long time to use the 32 bit name space but it might
be
> a 
> > problem in the future where the server is up for a long time and
just 
> > replacing or recovering nodes. 
> 
> I'm having trouble imaginging a situation in which the total set of 
> nodes that we ever put in layouts is over 4 billion, but the actual
> set of nodes that we access at any one time is much much smaller
> (i.e. less than one billion, say).  In other words, to me, running
> out of the 32-bit space is an indication, not that we need to conserve
> devid'sby recycling them but that the 32-bit space is, at least 
> potentially, too small.
> 
> Given that analysis, I would say that if we think we are going to run
> out of the 32-bit space, let's just make this 64-bit.  After all, the
> motivation for devid's as a shorthand was primarily that the full
thing
> that the shorhand represents was very large (order of 1K) in the
blocks
> case.  With a 64-bit devid, we still have something 99.2% smaller
> than that, rather than something 99.6% smaller.
> 
> > Another less important reason is an 
> > implementation one. It would be nice to keep the numbers in a low
> range so 
> > they can be used as an index to the device list table.
> 
> That's on the server.  On the client, he can't depend on a restricted
> range so he has to code for the full range anyway.  On the server, the
> advantage of the restricted range is fairly small since I would
imagine
> that most pieces of the server would use an internal identifier anyway
> (e.g.. the address of the data structure) and you fetch the devid to 
> put it in the layout before sending it.  On the client, you probably
> take the layout you receive and convert the devid's to a structure
> address.  So you will have to use a hash for this but you do it only
> when you recieve the layout, but you will have to do a hash anyway. 
> 
> > At the minimum we 
> > can say that when we recall all layouts associated with an FSID we
> expect 
> > the client to refresh the device list.
> 
> This is a possibility but I think that the simplicity benefit of
saying 
> that that a devid identifies something that never changes (during the
> life
> of the server) is significant and I'd like to maintain that.
> 
> Sent: Friday, July 22, 2005 9:34 PM
> To: Marc Eshel
> Cc: [email-masked]
> > Garth Goodson <[email-masked]> wrote on 07/22/2005
05:33:06
> PM:
> > 
> > 
> >>Marc Eshel wrote:
> >>
> >>>> If data are reorganized from a storage
> >>>> device with a given device ID to a different storage device
(i.e., 
> > 
> > if
> > 
> >>>> the mapping between storage device and data changes), the layout
> >>>> describing the data MUST be recalled rather than assigning the
new
> >>>> storage device to the old device ID.
> >>>
> >>>
> >>>If the device ID changes we can recall all layout that include the 
> > 
> > changed 
> > 
> >>>id but we also need to update the device list that the client
cached.
> 
> > 
> > How 
> > 
> >>>do we do it? can we add a call to invalidate the device list on the

> >>>client?
> >>>Marc. 
> >>
> >>The goal is that device IDs aren't reused between metadata server 
> >>reboots.  With a 32 bit device ID name space, do you think this will

> >>really be an issue.  I would prefer not adding more complexity to
> handle 
> > 
> > 
> >>the invalidation of device IDs unless it is absolutely necessary.
> >>
> >>The client may have old device IDs cached, but it can invalidate
it's 
> >>cache every so often.
> >>
> >>-Garth
> > 
> > 
> > Is there a reason why you did not cc the list?
> > 
> > It will take a long time to use the 32 bit name space but it might
be
> a 
> > problem in the future where the server is up for a long time and
just 
> > replacing or recovering nodes. Another less important reason is an 
> > implementation one. It would be nice to keep the numbers in a low
> range so 
> > they can be used as an index to the device list table. At the
minimum
> we 
> > can say that when we recall all layouts associated with an FSID we
> expect 
> > the client to refresh the device list.
> > Marc. 
> 
> Sorry, forgot to hit reply to all.  I think this something we can 
> discuss more.  Your proposal is reasonable; let's see what others
think.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 3 Aug 2005 08:34:25 -0400

> > Most certainly not.  The layout is a promise by the server that data 
> > can be read using it, and if that promise ceases to ever be valid, then
> > it will be recalled.  Once the WRITE returns, if the layout has not 
> > been recalled, you can use it.  If anything, including the WRITE,
> > renders it useless, it is the responsibility of the server to recall
> > it. 

> I agree, just the guarantee is on the mapping not on the 
> contents...  I'm not sure that your definition of "useless" 
> and Craig's are the same.  

Mine is "of no use" :-)

> Reading possibly stale data using 
> the layout may not be considered useless for many 
> applications that use NFS today.  At the same time many such 
> applications may use fsync() and close() to flush the client 
> and server buffers onto stable storage and expect basic 
> level of distributed data consistency so that after client A 
> writes data and syncs it using fsync()/close() the data will 
> be visible (persistently) to any other client.  

And typically until you do the fsync()/close() some of your
written data is in the buffer cache and others may not see it.
That means they may read data that could be described as 
"stale" in a certain sense, but I definitely think it would
be going way overboard to say that the fact that I had a
write which was on the wire rendered the file handles that
others had "useless".  They are still quite useful and if
I want to make them useful to get the precisely up-to-data
data, then all I have to do is WRITE it using a synchronous
or WRITE is using an async write and then do a COMMIT.

Note here that if I do an async WRITE and do not (yet) do a
COMMIT, all other clients are not really able to rely on 
getting up-to-date data, not only those getting the data
directly from the data server.  Normally, a READ will get the
updated data but there is no assurance of that.  If the 
server goes down in the interim, a client may read the old
data.

The basic point here is there is always going to be a period
of indeterminateness with regard to newly written data.  During
this period a read may get the old data or the new data or 
some mixture.  If the write is synchronous the period of
indeterminateness begins when the WRITE is put on the wire
and ends when it executed.  For async writes, it extends all
the way until the COMMIT is executed.  This has always been
true of NFS and pNFS follows along.  The period of indeterminateness
is not any longer.  It is only that you are far more likely
to see the old data when reading using the layout.  Nevertheless,
it is possible to see the old data when doing ordinary READ's
as well.  If you think this makes the READs or the data structures 
used to do the reads (filehandles, layouts) useless, then so
be it, but I would  not describe things that way.  I think it means 
that the writing client has decided, correctly or incorrectly,
that the application can function without concurrent readers 
seeing the new data instantaneously.

> Hence, my 
> proposal to require pnfs enabled servers to sync dirty data 
> to stable storage on the data servers on COMMIT and 
> synchronous writes.

I agree with this.  When you write to a meta-data server 
asynchronously, the server may propagate the written data
to the data servers (so that they are read by those using
layouts) but is not obliged to do so until the the COMMIT
occurs.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]]
Sent: Tuesday, August 02, 2005 9:01 AM
To: Noveck, Dave
Cc: Craig Everhart; [email-masked]; [email-masked]
Subject: Re: [nfsv4] New pNFS ID available

Noveck, Dave wrote:
>>>If client A holds a write layout on a region in a file, and client B
>>>does a READ to the metadata server that contains that region of that
>>>file, the server must ("MUST") recall the layout from client A.
>>>
>>>If client A does the read, I would expect the server to return
>>>bits that are not more out-of-date than the most recent LAYOUTCOMMIT
>>>from client A for the layout. If client A wants that read to return
>>>up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
>>>A layout is identified by {ClientID,FileHandle,offset,length,
>>>layout_type,iomode
>>>}*,
>>>so the server should have enough information to figure out if the
>>>requestor holds an outstanding layout. 
> 
> 
>>Right.  Good.  
> 
> 
> Not as far as I'm concerned.
> 
> 
>>Now how about the case I cited, where the client does a 
>>write to the meta server.  Should it assume that its layout is useless, 
>>since it can't tell when it could read its data back from the 
>>storage?  
> 
> 
> Most certainly not.  The layout is a promise by the server that data 
> can be read using it, and if that promise ceases to ever be valid, then
> it will be recalled.  Once the WRITE returns, if the layout has not 
> been recalled, you can use it.  If anything, including the WRITE,
> renders it useless, it is the responsibility of the server to recall
> it. 

I agree, just the guarantee is on the mapping not on the 
contents...  I'm not sure that your definition of "useless" 
and Craig's are the same.  Reading possibly stale data using 
the layout may not be considered useless for many 
applications that use NFS today.  At the same time many such 
applications may use fsync() and close() to flush the client 
and server buffers onto stable storage and expect basic 
level of distributed data consistency so that after client A 
writes data and syncs it using fsync()/close() the data will 
be visible (persistently) to any other client.  Hence, my 
proposal to require pnfs enabled servers to sync dirty data 
to stable storage on the data servers on COMMIT and 
synchronous writes.

> 
> 
>>Clearly the NFS meta server can implement persistent 
>>storage any way it intends, 
> 
> 
> OK so far.
> 
> 
>>and the only consistency on the data 
>>is that it shouldn't hand out further layouts to a device or a 
>>set of devices until its pending data could be read by future 
>>fetches from that device or those devices. 
> 
> 
> Absolutely false as far as I'm concerned.  It also has a consistency
> constraint with regard to layouts already outstanding (in addition
> to "further layouts").  While the WRITE is pending reads to the
> data server can give you the old or the new data (or a mixture I
> guess -- no atomicity requirement).  After the WRITE completes, they
> have to give you the new data.  If the server implements its storage
> in any way such that it cannot obey those constraints then it must
> recall the layouts to prevent reads being done that cannot be handled
> correctly.
> 
> From: Craig Everhart [mailto:[email-masked]]
> Sent: Monday, August 01, 2005 6:37 PM
> To: [email-masked]
> [email-masked] wrote on 08/01/2005 05:45:04 PM:
> 
> 
>>If client A holds a write layout on a region in a file, and client B
>>does a READ to the metadata server that contains that region of that
>>file, the server must ("MUST") recall the layout from client A.
>>
>>If client A does the read, I would expect the server to return
>>bits that are not more out-of-date than the most recent LAYOUTCOMMIT
>>from client A for the layout. If client A wants that read to return
>>up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
>>A layout is identified by {ClientID,FileHandle,offset,length,
>>layout_type,iomode
>>}*,
>>so the server should have enough information to figure out if the
>>requestor holds an outstanding layout. 
> 
> 
> Right.  Good.  Now how about the case I cited, where the client does a 
> write to the meta server.  Should it assume that its layout is useless, 
> since it can't tell when it could read its data back from the 
> storage?  Clearly the NFS meta server can implement persistent 
> storage any way it intends, and the only consistency on the data 
> is that it shouldn't hand out further layouts to a device or a 
> set of devices until its pending data could be read by future 
> fetches from that device or those devices. 
> 
> 
> 
>>In the case where client A holds a read-only layout, and client B
>>does a read, there is no conflict.
>>
>>In the case where the reading client hasn't set a client ID, the
>>server should (IMO) assume that any/all outstanding write layouts
>>are held by clients other than the reading client, and recall them
>>all before servicing the read.
>>
>>-jim
>>
>>* Well, draft-welch-pnfs-ops-03 has iomode as an open issue, but I'm
>>assuming that it is included for the purposes of this e-mail.
>>
>>-- 
>>Jim Zelenka
>>Software Engineer, Panasas, Inc.
>>Accelerating Time to Results(TM) with Clustered Storage
>>www.panasas.com
>>412.323-3500 
> 
> 
>                 Craig
> 
> Craig Everhart
> +1 919 543 2169 (tie 441 2169)
>

---

From: [email-masked]
Date: Wed, 24 Aug 2005 19:20:42 -0400

> >>>> As to "cannot assume anything" with regard to asynchronous writes,
> >>>> I'm wondering if that might be too liberal.  For example, if I
> >>>> write 4K, an implementation might propagate all the odd bytes
> >>>> immediately and the even bytes at COMMIT time.  Even though nobody 
> >>>> would be that wierd, I'm wondering whether some restrictions
> >>>> might make some applications easier.  For example, the blocks
> >>>> protocol might reasonably require that all of the write that is
> >>>> within one block happen at one time, so that if the write is on
> >>>> a block boundary, you can't read a partially updated block.  
> 
> >>> That's a good property to have yet I'm not sure whether the client 
> >>> can tell what the server's atomic block size is.
> 
> >> For pnfs, you could add it to the layout or the device info.  In the
> >> case of blocks, the client is going to have to now the physical block
> >> size to access the disk, so a statement in the definition of the 
> >> storage to the effect would at least make it as big as that, even
> >> if our would still need additional info to define it as a multiple
> >> of that.
> 
> > Sometimes the metadata server's virtual memory system may impose an 
> > atomic block size which different than the physical block size on disk 
> > (one defined by allocation granularity) so I think this attribute of the

> > file system should be separated out.  Why not add it as a recommended 
> > filesystem attribute?
> 
> Please don't do that.  From a classic NFS perspective the blocksize
> has always been just an advisory piece of data.  If the client used
> the preferred block size it may get better performance, but any size
> of data would work just fine.  It has always been that the server
> made it right for its needs.
> 
> In this particular example, only the direct block method needs this
> sort of information. As was suggested above, the correct place to
> put this sort of information is in the layout.

I agree this belongs in the layout, but one has to be careful - block
size operations for an NFS idea of block size need not be atomic in the
block/volume world.  HighRoad uses blocksize in its layouts, partly in
order to get by with 32 bit block offsets, but the pNFS blocks draft
scraps that in favor of 64-bit byte offsets (it's wonderful to have
a relatively clean sheet to start from ...).  Block size can be a useful
hint to a pNFS client to tell it what to align its layout requests to if
it has the flexibility to do so.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------