10.2.12

NFSv4 Myths and Legends

NFS4 exists only in the deepest depths of Hades, guarded by the three headed monster, Cerberus. In this rare mythological contact we use strong cryptographic keys to tempt the monster into allowing us the first ever human contact with this forbidding Underworld God.

Common NFS4 misunderstandings
I am not a Linux expert. These notes were written after I faced the task of installing secure NFS4 for a heterogeneous LAN of openSUSE, Ubuntu and Windows clients under a Samba 4 Active Directory PDC. I have written this in an attempt to set the ball rolling to maintain a centralised deposit of correct information on installing secure nfs4, written in plain English and on modern Linux distributions. I also hope that it will help dispel the myths I read and the dead ends I followed in setting up a test LAN. Every stage was a struggle. From configuring DNS, through preparing the keytsb, finding out that Windows 7 Home Edition can't join a domain to finally running rpc.gssd on a client and mounting a kerberized NFS4 share.

This is one of the areas where Linux really fall down. Documentation is dismal, sparse and out of date. Ask the developers to answer a simple question in plain English. There's usually a deafening silence after which they'll refer you to the souce code. Bless!

Microsoft tend not to refer you to the source code but to other mpenetrably obscure and drab articles where they tell you to point and click at something on th MMC. I suppose that's all part of the freedom which the Linux experience brings. In the end and when your first remote client has connected, oh the joy of seeing root and local users get 'Permission denied' when trying to access a chmod 777 share. Tickets please!

Myth: NFS4 must be exported from an fsid=0 pseudo root directory. e.g. I am exporting the /home directory to the clients. The /etc/exports file could be either of these examples. 

e.g. 1. How not to do it
/home is bind mounted to chmod'ed 1777 /exports
/etc/exports
/export  gss/krb5(rw,fsid=0,insecure,no_subtree_check,crossmnt)
/export/home gss/krb5(rw,insecure,no_subtree_check)
/export  gss/krb5i(rw,fsid=0,insecure,no_subtree_check,crossmnt)
/export/home gss/krb5i(rw,insecure,no_subtree_check)
/export  gss/krb5(rw,fsid=0,insecure,no_subtree_check,crossmnt)
/export/home gss/krb5(rw,insecure,no_subtree_check)
/export  *(rw,fsid=0,insecure,no_subtree_check)
/export  *(rw,nohide,insecure,no_subtree_check)

e.g. 2 . Simply use the familiar nfs3 export method which gives the same choice of security.
/etc/exports
/home *(rw,sec=none:sys:krb5:krb5i:krb5p,no_subtree_check)
Choice of 5 flavours for mounting on the client.
Please see the comment above: The linux implementation allows you. . . 

Myth: you must use the nohide option for the bind mount to be visible on the client.
You can use crossmnt on the pseudoroot or nohide on the bind mount. I never did understand what mutually excluisive meant! Or use neither crossmnt nor nohide and use the conventional nfs3 method as described above.


Myth: NFS4 can only use weak DES cryptography
You've Googled and found out that NFS4 can only use weak DES cryptography. You need the line:
[libdefaults]
allow_weak_crypto = true
added to /etc/krb5.conf

Unless you're running an ancient distro, this is not needed. This will annoy your Windows admin even more. Now he can't use the weak crypto is a security risk excuse when you ask him to allow your Linux clients to authenticate against it his delicate point and click 2k8 box. Tested using arcfour-hmac-md5 with  kernels 3.1.9-1.4-desktop with nfs-client-1.2.5-4.3.1(openSUSE 12.1) and 3.0.0-15-default (Ubuntu 11.10)

Myth: NFS4 needs only the nfs/fqdn@REALM principal in it's keytab
No. It needs the machine principal too:

Myth: The client must have a nfs/client.domain principal
No. It doesn't need to have one.
From man rpc.gssd(8)
<quote>
Previous versions of rpc.gssd used only "nfs/*" keys found within the keytab. To be more consistent with other implementations, we now look for specific keytab entries. The search order for keytabs to be used for "machine credentials" is now:
<HOSTNAME>$@<REALM>
root/<hostname>@<REALM>
nfs/<hostname>@<REALM>
host/<hostname>@<REALM>
root/<anyname>@<REALM>
nfs/<anyname>@<REALM>
host/<anyname>@<REALM>
</quote>

A real life example
hh3.hh3.site is my server and  HH6$@HH3.SITE is the machine principal for a remote Linux client.
On the client, root issues:
mount -t nfs4 hh3:/foo /bar -o sec=krb5

Here is the log on the KDC:

Kerberos: ENC-TS Pre-authentication succeeded -- HH6$@HH3.SITE using arcfour-hmac-md5
Kerberos: AS-REQ authtime: 2012-02-06T19:44:47 starttime: unset endtime: 2012-02-07T05:44:47 renew till: 2012-02-07T19:44:47
Kerberos: Client supported enctypes: aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, des3-cbc-sha1, arcfour-hmac-md5, des-cbc-crc, des-cbc-md5, des-cbc-md4, using arcfour-hmac-md5/arcfour-hmac-md5
Kerberos: Requested flags: renewable-ok
Kerberos: TGS-REQ HH6$@HH3.SITE from ipv4:192.168.1.10:45421 for nfs/hh3.hh3.site@HH3.SITE [canonicalize, renewable]
Kerberos: TGS-REQ authtime: 2012-02-06T19:44:47 starttime: 2012-02-06T19:44:47 endtime: 2012-02-07T05:44:47 renew till: 20

The nfs server must have identified itsself:
klist -ke /etc/krb5.keytab
Keytab name: WRFILE:/etc/krb5.keytab
KVNO Principal
---- ---------
   1 nfs/hh3.hh3.site@HH3.SITE (arcfour-hmac)  
   1 HH3$@hh3.site (arcfour-hmac)
Here is a Wireshark trace from the same server under arcfour. 
nfs on the wire under arcfour
Sorry, this is nfs3 but we can confirm for nfs4 too. The problem with nfs4 is the incompatibility of it's acl's:-( That's why we had to use on this particular client because it needed group rw from a 0022 umask export. This is not possible with nfs4_setfacl.

rpc processes
1. openSUSE after starting the nfs server


ps aux | grep rpc
root      1231  0.0  0.1   2356   764 ?        Ss   08:20   0:00 /sbin/rpcbind
root      3737  0.0  0.0      0     0 ?        S<   08:28   0:00 [rpciod]
root      5215  0.0  0.2   3684  1208 ?        Ss   09:26   0:00 /usr/sbin/rpc.svcgssd
root      5223  0.0  0.0   2416   344 ?        Ss   09:26   0:00 /usr/sbin/rpc.idmapd
root      5227  0.0  0.1   2976   748 ?        Ss   09:26   0:00 /usr/sbin/rpc.mountd --no-nfs-version 2 --no-nfs-version 3
(note the lack of gssd)

2. On both openSUSE (after calling rpc.gssd) and Ubuntu with all the necessary processes for successfully exporting the directory with -o sec=krb5<flavour>

ps aux | grep rpc
root      1231  0.0  0.1   2356   764 ?        Ss   08:20   0:00 /sbin/rpcbind
root      3737  0.0  0.0      0     0 ?        S<   08:28   0:00 [rpciod]
root      5215  0.0  0.2   3684  1208 ?        Ss   09:26   0:00 /usr/sbin/rpc.svcgssd
root      5223  0.0  0.0   2416   344 ?        Ss   09:26   0:00 /usr/sbin/rpc.idmapd
root      5227  0.0  0.1   2976   748 ?        Ss   09:26   0:00 /usr/sbin/rpc.mountd --no-nfs-version 2 --no-nfs-version 3
root      5294  0.0  0.0   3816   428 ?        Ss   09:31   0:00 rpc.gssd


nfs Installation details
1. openSUSE
Packages:
zypper in nfs-kernel-server nfs-client nfsidmap limal-nfs-server

Configuration (Or use the excellent Yast NFS-Server Module)
grep -v "#" /etc/sysconfig/nfs
USE_KERNEL_NFSD_NUMBER="4"
MOUNTD_PORT=""
NFS_SECURITY_GSS="yes"
NFS3_SERVER_SUPPORT="no"
NFS4_SUPPORT="yes"
SM_NOTIFY_OPTIONS=""
NFS_START_SERVICES="yes"
STATD_OPTIONS=""
NFSV4LEASETIME=""
RPC_PIPEFS_DIR=""
SVCGSSD_OPTIONS=""
NFSD_OPTIONS=""

2. Ubuntu
Packages:
apt-get install nfs-common nfs-kernel-server
(you may also need: modprobe nfs)
Configuration
/etc/default/nfs-common
NEED_STATD=
STATDOPTS=
NEED_IDMAPD=yes
NEED_GSSD=yes

/etc/default/nfs-kernel-server
RPCNFSDCOUNT=8
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS=--manage-gids
NEED_SVCGSSD=yes
RPCSVCGSSDOPTS=
RPCNFSDOPTS=

Needed on both client and server on both openSUSE and Ubuntu:
/etc/idmapd.conf
[General]
Verbosity = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
Domain = hh3.site
[Mapping]
Nobody-User = nobody
Nobody-Group = nogroup

Debugging
Run rpc.idmapd -fvvv and rpc.gssd -fvvv in separate terminals. They will tell you in no uncertain terms when you've got it wrong.
rp calls in action
rforc
It would be great if we could gather together our experiences on NFS4 installation and centralise them here. Particularly helpful would be distribution specific contributions with working examples


References:
1. The sparse and technical nfs devs wiki. Have K & R ready.

2. man rpc.gssd(8). If you don't have it installed, there's a copy here.

3. A hands-on Samba4-with-nfs guide,  screenshots and woes using these instructions. openSUSE edition.

4. The same, but Ubuntu specific. Get ready to get your apt-get ready.