Subject: [nfsv4] Large number of objects in NS; Problem definition

From: sfaibish <[email-masked]>
Date: Thu, 20 Aug 2009 14:52:52 -0400

Traditionally file system designs have envisioned directories as a means
of organizing files for human viewing, e.g NS organization. Currently,
directories typically contain a few tens to thousands of files and using  
split
of the NS and LDAP the number of objects is within the adressing space of  
LDAP.
Users of large namespaces, have begun to put millions of files/objects  
into single
directories, for example, used as simple databases. With the increase of  
the
objects in directories the namespace data increases and will in no time  
overboard
the number of split directories that will also increase proportionally.
Furthermore, large-scale applications running on clusters with tens to
hundreds of thousands of cores can burstily create objects using all  
compute
cores, amassing bursts of hundreds of thousands of creates or more.

My proposal is to revisit the data-structures used to build large  
namespaces
with large number of splits of the directories which each will contain  
billions
of objetcs (directories, files, links). As the number of NS clienmts will  
explode
at same time the pressure on the NS and LDAP servers will increase  
possibly beyond
current possible structures. Extending resizeable hash-tables and index  
tables
to a highly concurrent distributed NS service will be probaly one way to  
address
the issue but will require extenssions to the current FedFS. Other ideas  
use bitmap
encoding to indicate which of the possibly created hash partitions to split
independently, and to correct stale client handles.

This is a very general definition enough to start the discussion and agree  
on a
new draft aimed at 4.2. If needed I can explain in a more detailed draft.  
Thanks
for your understanding and patience.

/Sorin

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Network Storage Group

         EMC²
where information lives

Phone: 508-435-1000 x 48545
Cellphone: 617-510-0422
Email : [email-masked]

---

From: James Lentini <[email-masked]>
Date: Thu, 20 Aug 2009 16:13:31 -0400 (EDT)

Sorin,

Thank you for writing this up.

I think the issues you raise are orthogonal to federated namespaces. 
Regardless of how the namespace is organized, an implementation will 
have practical limits on the number of directory entries, maximum 
number of parallel file creates, etc. The issues you are concerned 
with are related to making the NFS clients and servers communicate 
efficiently at these scales (100,000+ creates, reading a directory 
with millions of objects, etc). I do agree that some form of 
parallelism could be beneficial, and for that I would suggest pNFS 
(perhaps with additional capabilities). However, I don't think a 
solution requires federation (decentralized control) of the NFS 
servers.

-james

On Thu, 20 Aug 2009, sfaibish wrote:

> Traditionally file system designs have envisioned directories as a 
> means of organizing files for human viewing, e.g NS organization. 
> Currently, directories typically contain a few tens to thousands of 
> files and using split of the NS and LDAP the number of objects is 
> within the adressing space of LDAP. Users of large namespaces, have 
> begun to put millions of files/objects into single directories, for 
> example, used as simple databases. With the increase of the objects 
> in directories the namespace data increases and will in no time 
> overboard the number of split directories that will also increase 
> proportionally. Furthermore, large-scale applications running on 
> clusters with tens to hundreds of thousands of cores can burstily 
> create objects using all compute cores, amassing bursts of hundreds 
> of thousands of creates or more.
> 
> namespaces with large number of splits of the directories which each 
> will contain billions of objetcs (directories, files, links). As the 
> number of NS clienmts will explode at same time the pressure on the 
> NS and LDAP servers will increase possibly beyond current possible 
> structures. Extending resizeable hash-tables and index tables to a 
> highly concurrent distributed NS service will be probaly one way to 
> address the issue but will require extenssions to the current FedFS. 
> Other ideas use bitmap encoding to indicate which of the possibly 
> created hash partitions to split independently, and to correct stale 
> client handles.
> 
> This is a very general definition enough to start the discussion and 
> agree on a new draft aimed at 4.2. If needed I can explain in a more 
> detailed draft. Thanks for your understanding and patience.
> 
> /Sorin
> 
> -- 
> Best Regards
> 
> Sorin Faibish
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 20 Aug 2009 16:21:10 -0400

It sounds like much more in the area of extending pNFS to directories.  There has definitely been interest in that area and I think people have been assuming that that will enable you to create larger directories, but they haven't been explicit about that as a goal, as least as I recall.

-----Original Message-----
From: Lentini, James 
Sent: Thursday, August 20, 2009 4:14 PM
To: sfaibish
Cc: nfsv4 list
Subject: Re: [nfsv4] Large number of objects in NS; Problem definition

Sorin,

-james

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Thu, 20 Aug 2009 13:44:12 -0700

In that sense, Doesn't this address the issue to a larger extent.

http://tools.ietf.org/id/draft-eisler-nfsv4-pnfs-metastripe-01.txt

How is it different from the above?

- Pranoop

> From: Noveck, Dave 
> Sent: Thursday, August 20, 2009 1:21 PM
> To: Lentini, James; sfaibish
> Cc: nfsv4 list
> It sounds like much more in the area of extending pNFS to 
> directories.  There has definitely been interest in that area 
> and I think people have been assuming that that will enable 
> you to create larger directories, but they haven't been 
> explicit about that as a goal, as least as I recall.
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 20 Aug 2009 16:59:39 -0400

It may very well and this may just be a question of presentation but handling distributed directories with billions of entries is tough.  If the document were to say it can do that than the probability it really can is less than one (Given that Mike wrote it, I'd guess .8), but this document doesn't say that so I'd say the probability is much lower than that.  Maybe Mike is just incredibly modest/conservative but if I figured out a way to handle that, the word "billions" would definitely appear in the document.

Maybe a good place is to start with this document, do some thought-prototypes and see if you think billions is in range.  If you do, then maybe some real prototypes are in order.  If Mike has already solved this problem or come very close, then that would be great.

-----Original Message-----
From: Erasani, Pranoop 
Sent: Thursday, August 20, 2009 4:44 PM
To: Noveck, Dave; Lentini, James; sfaibish
Cc: nfsv4 list
Subject: RE: [nfsv4] Large number of objects in NS; Problem definition

- Pranoop