Subject: Re: [nfsv4] Proposal for end-of-life for fedfs-utils development

From: Chuck Lever <[email-masked]>
Date: Wed, 7 Jun 2017 14:02:19 -0400

Bruce, that's a good question, and an answer is worth sharing with
the nfsv4 WG mailing list, whom I've cc'd.

> On Jun 7, 2017, at 11:55 AM, [email-masked] wrote:
> 
> So if it's not too depressing I'd be curious what went wrong--did this
> turn out to be harder than we thought to get stable, or are people happy
> enough with automounting, or did we just not do a good job of explaining
> it to people that might use it, or some combination of all those?

If you're interested in an intriguing discussion of what makes a
successful protocol, I recommend RFC 5218. Now, not in any particular
order, the main reasons FedFS has not been widely adopted are (IMO of
course):

1. Lack of vendor adoption

After the specifications were completed, we anticipated two independent
implementations. For various reasons the Solaris FedFS implementation
was abandoned. One big reason was LDAP.

NFS/Ganesha has expressed some interest in FedFS over the years, but
I'm not aware of an implementation.

NetApp abandoned FedFS before it was published as RFCs.

So we were left with just the Linux implementation, just like what
happened with SPKM.

2. LDAP is onerously complex

The LDAP components of the Linux implementation were the worst to
implement by far. This also proved to be the case in the Solaris
prototype implementation. OpenLDAP is designed for massive scalability
but aggressively shuns ease of administration.

It was suggested that we integrate with FreeIPA, which is a Linux-based
management suite that can provide an LDAP service, a KDC, and a
certificate authority. But there was never enough user inertia to make
that effort.

There was resistance to integrating FedFS directly into nfs-utils
because the LDAP components would have added complex library
dependencies.

The LDAP pieces of FedFS are specified to use TLS, but NFS operates
using GSS and Kerberos. Getting these two worlds to work together was
going to be the next step (and also, figuring out how to make the LDAP
service use GSS instead, which we should have done before completing
the standard).

However, by the time the FedFS standards were complete, there was no
longer WG interest to address its shortcomings. There were two small
I-Ds published to start down that path. They went nowhere because the
IETF's pool of LDAP expertise is difficult to consult, now that
LDAP-related WGs have been disbanded.

IMO LDAP, which was chosen early in the 2000s by the IBM prototypes
that were to become FedFS, was ultimately a poor choice for what
eventually became the public FedFS standard. This can be attributed
to changing times.

3. Lack of consensus about how to store junctions

There was never consensus in the Linux community about how to represent
junction objects on disk. NFS wanted something that would look naturally
like an empty directory to NFS versions that did not support referrals.
Samba wanted something that could behave like symbolic links, as it
had been using for its own DFS-style referrals. The filesystem folks
were not interested in creating a new distinct object that could fill
both roles.

As you are probably aware, Bruce, I asked about this every year at
LFS/MM for several years. I was always told "get back to us when you
have users."

Solaris went for reparse points in its implementation. Those are also
supported by FreeBSD. I think RPs would have been a great direction
but sadly these are not being actively pursued in the Linux FS
community.

Lack of user adoption sapped the energy out of the effort to find a
consensus. Though, if FedFS had been widely adopted before a consensus
was reached on junction object representation, we'd have a significant
data conversion problem.

4. Existing implementations are capable enough

This is mostly speculation on my part, but FedFS was a competitive
response to the global namespace capabilities of AFS and SMB, not to
any particular stated need in NFS communities.

I've discussed the use of FedFS with various large enterprises, but
quite often the underlying needs are able to be filled by some form
of pNFS. I think this class of solution is what NetApp and Primary
Data have adopted.

Or put another way, pNFS seems capable of doing most anything that a
referral-based mechanism can do. And in non-NFSv4 environments, the
automounter is good enough for most people.

Certainly we could design something from the ground up that addresses
many of these shortcomings, but I get about one query about FedFS a
year. I just wonder if it would be worth the effort.

> --b.
> 
> On Fri, Jun 02, 2017 at 11:01:05AM -0400, Chuck Lever wrote:
>> Upstream fedfs-utils has not been under active development for
>> two years or more, and there is a scant user base. I'd like to
>> propose making 0.10 the final major release of fedfs-utils.
>> 
>> The plan:
>> 
>> - Since 0.10 is in at least one major enterprise distribution,
>> I will remain available to integrate security fixes and make
>> new minor releases in the 0.10 line, as needed, for one to
>> two more years.
>> 
>> - Retire and remove fedfs-utils from upstream mirror distros
>> such as Fedora rawhide.
>> 
>> - Transfer utilities such as nfsref into nfs-utils, with
>> support for FedFS junctions removed.
>> 
>> - Announcements of the change in status will be made on
>> fedfs-utils-announce and on the wiki.linux-nfs.org site.

--
Chuck Lever

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Wed, 14 Jun 2017 23:57:02 +0200 (CEST)

Hi Chuck,

----- Original Message -----
> From: "Chuck Lever" <[email-masked]>
> To: "J. Bruce Fields" <[email-masked]>
> Cc: "fedfs-utils Developers" <[email-masked]>, "linux-nfs" <[email-masked]>, "NFSv4"
> <[email-masked]>
> Sent: Wednesday, June 7, 2017 8:02:19 PM
> Subject: Re: [nfsv4] Proposal for end-of-life for fedfs-utils development

> year. I just wonder if it would be worth the effort.

We  at DESY was interested in FedFS for two main reasons: one is to replace
AFS and second is to build a federation on independent NFS server.
The second option is not covered by pNFS as FedFS allows to run independent
pNFS instances with different administrative domains, implementations and 
namespaces, but still provide a common mount point. NetApp or Primary Data
does not provide solution for that. Even we don't :-D. 

Why we have failed? There was not clear how to set it up. LDAP management
was a mess. Schema was not standardized and not available with existing
LDAP server.

Currently we have 5 pNFS based instances with ~15PB in total. Some
nodes have direct mounts. But most of 'generic' clients automunter config
to fake FedFS as we don't know in advance which server will be used. Does
it works - Yes, does it solves all problems - No.

I would like to see a federated name space. We have the demand, but may be it's
only us.

For non NFS world we have a HTTP federation. It uses HTTP redirects, similar to
referrals. However, the tendency is to move towards distributed systems instead
of federated ones. If this project survive, then we can attempt to have a second
look at NFS federation. Life is a spiral.....

Regards,
   Tigran.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Chuck Lever <[email-masked]>
Date: Thu, 15 Jun 2017 10:56:55 -0400

> On Jun 14, 2017, at 5:57 PM, Mkrtchyan, Tigran <[email-masked]> wrote:
> 
> Hi Chuck,
> 

The last time I visited DESY, I left with the impression that
there was a requirement that directories could be populated
programmatically (ie. one directory entry could represent a
group of files, perhaps on disparate NFS servers, and something
on each client determined which member of that group would be
opened). FedFS can't do that, but perhaps pNFS could.

> was a mess. Schema was not standardized

The schema is now standardized in RFC 7532, but the pub date
for that document is recent (March 2015).

> and not available with existing LDAP server.

I recognized this could be an impediment to adoption, and
contacted the people responsible for 389ds and the OpenLDAP
community. I was told they could include the schema when
the schema was standardized and fedfs-utils had users.

fedfs-utils provides the schema in two forms, IIRC, ready
to copy to your favorite LDAP server.

> I would like to see a federated name space. We have the demand, but may be it's

I can't speak authoritatively about that, but DESY is the
only contact I've had about fedfs-utils in years.

> look at NFS federation. Life is a spiral.....

There's no plan to take down the upstream fedfs-utils
source code repository.

> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email-masked]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Chuck Lever