Obviously, with the social distancing/lockdowns happening, I have had more time to write up blogs on things. So, here’s another one. If you have suggestions for topics, let me know and I’ll see about writing them up.

Before we get to the meat of what the title is (or scroll down to the bottom if you want the quick and dirty steps), let’s recap NFS security and Kerberos…

NFS is a protocol that allows multiple clients communicate to a single storage endpoint as a way to share data across a network. Because data is transmitted over a network, it’s important to be able to secure that data, both at-rest and in-flight.

At-rest Security in NFS

At-rest security refers to security applied to data residing on the storage system, as well as the interaction between NFS client and NFS server to negotiate things like NFS version, identity, etc.

At-rest security for NFS includes:

Export policies and rules

Permissions/ACLs

User/group owners

Name ID strings (NFSv4.0 and later)

However, when data passes over the wire, packet sniffers are able to see identifying information like IP addresses, users, groups, permissions, file names, paths, etc. All it would take is someone being able to see this information to easily create duplicate set ups to “spoof” users and groups and gain access to data. At-rest security mostly protects you from threats that don’t have this knowledge or expertise. To prevent bad actors from getting information from the transmission of data, you’d need to set up in-flight security.

In-flight Security in NFS

In-flight security refers to securing the actual data packets in transit.

For NFS, this can be done in a few ways:

For ONTAP, tunneling NFS over SSH or stunnel isn’t supported, but you can use NFS Kerberos.

For a deeper look at NFS Kerberos in ONTAP, see:

NFS Kerberos in ONTAP Primer

Kerberos Security Flavors

NFS Kerberos has 3 methods that can be used to secure things. Each provides an additional level of security, but also adds extra performance overhead.

Keep in mind that NFSv3 *can* use Kerberos, but only the NFS portion of the protocol will use it. Ancillary protocols like mountd, portmapper, NLM, etc. will still be unencrypted. The most secure version of NFS available is NFS v4.x, which combines all the NFS ancillary protocols into a single port and compound NFS calls, which can all be Kerberized.

krb5 – Encrypts the authentication only

Remember when I said if a bad actor stole information about the client, user, etc. that they could spoof that user to get access? Well, with krb5, that becomes harder to do. You can have all the information about the NFS export, but to gain access to a Kerberized mount, you’d need to have a valid username and password to get a Kerberos TGT, which would then be used to get the NFS service ticket. In addition, your client would also need to have a Kerberos ticket and keytab to gain access, so unless your attacker has KDC admin rights, they won’t be able to access the mount.

krb5i – Authentication with data checksums/integrity checking

Krb5 secures the initial authentication for NFS, but the actual NFS data packets are still transmitted in clear text across the wire. If a bad actor were to plug in and sniff those data packets in flight, they could see everything. In addition, bad actors could also act as a “man in the middle” to intercept packets and then transmit their own data if they chose. Krb5i prevents this by using checksums on the client and server to verify that all the data arriving and leaving is coming from a trusted source. The data is still visible, but it can’t be interfered with.

krb5p – Authentication/integrity checking/end to end encryption (privacy)

For the ultimate in-flight security hammer for NFS, you would use krb5p. This combines what krb5i does with in-flight encryption of all NFS packets. That means all NFS packets will be encrypted with the enctype specified in the Kerberos configuration. This can add considerable overhead for NFS performance, but will also prevent NFS packets from being sniffed easily.

How to set up NFS Kerberos in ONTAP

Setting up Kerberos in ONTAP requires a few things:

A Key Distribution Center (Microsoft AD, MIT Kerberos, FreeIPA, RedHat IDM)

DNS server or static host entries

Optional (but recommended): LDAP server for UNIX identity management

Configuring NFS clients and ONTAP Service Principal Names

The KDC is the central hub for Kerberos operations and is responsible for handing out Kerberos tickets to clients, users and services for authentication in a Kerberos realm.

DNS is used to resolve IP addresses to host names, which are then passed to the KDC as SPN requests. For example, if client 10.10.10.10 tries to get a Kerberos ticket, a DNS reverse lookup will be done to find the hostname FQDN “client1.domain.com” Then a SPN request for host/client1.domain.com@DOMAIN.COM is made to the KDC. If the SPN exists and matches the request, then the Kerberos request moves on to the next steps. If it doesn’t exist, the request fails.

LDAP is used to centralize the UNIX identities for users and groups to ensure clients and servers have the same information all the time, without manual intervention. This is less important to Kerberos and more important to NFSv4.x and NFS permission resolution.

NFS client configuration for Kerberos on newer clients is fairly straightforward; you configure DNS (so it can find the Kerberos realm name and client hostname) and then simply use one of the automated tools to “join” the Kerberos realm. This automatically creates the service principal and transfers the keytab files. Where it gets tricky is if you have to do a manual Kerberos configuration. TR-4073 and TR-4616 can be of some guidance there, as well as a bevy of articles across the web.

NFS server/ONTAP configuration for Kerberos is relatively simple; you configure DNS and the Kerberos realm and then you enable Kerberos on a network interface. When you do that, ONTAP interacts with the KDC you specified in the Kerberos realm and automates the principal creation – provided the KDC is using standard Kerberos interaction, such as Microsoft Active Directory or kadmin for Linux.

Now we’re getting to the part the title hinted about… FreeIPA Kerberos setup.

Why do I need a blog on FreeIPA Kerberos with ONTAP? You just said it was easy!

I did say it was easy – if the KDC is using standard kadmin. However, FreeIPA happens to use a wrapper over kadmin for KDC management and discourages the use of kadmin for management of service principals. In fact, by default, they lock things down pretty tight.

In FreeIPA, there is a GUI to make things simpler to use. There are also “ipa” commands to interact via the CLI, such as ipa user-add or ipa service-add. In most Linux KDCs, there was kadmin to manage things. In fact, there are *two* kadmins. One is for local management (kadmin.local) and the other is for remote management (kadmin). For more info on the differences, see:

https://docs.oracle.com/cd/E19683-01/816-0211/6m6nc66tj/index.html

The remote kadmin is what ONTAP interacts with for Linux KDCs. When you use the automated ONTAP method to add the NFS service principal, the following happens:

A prompt for a username and password to issue a kinit to the KDC to gain access to kadmin

get_principal is run via remote kadmin to see if the SPN exists If the SPN doesn’t exist, create_principal is run via remote kadmin If the SPN does exist, modify_principal is run via remote kadmin



All of these kadmin commands require the proper privileges to run successfully. In FreeIPA, remote kadmin is locked down by default to even run get_principal.

For example:

# ipa user-add krbadmin --------------------- Added user "krbadmin" --------------------- User login: krbadmin First name: Kerberos Last name: Admin Full name: Kerberos Admin Display name: Kerberos Admin Initials: KA Home directory: /home/krbadmin GECOS: Kerberos Admin Login shell: /bin/sh Principal name: krbadmin@CENTOS-LDAP.LOCAL Principal alias: krbadmin@CENTOS-LDAP.LOCAL Email address: krbadmin@centos-ldap.local UID: 1971600007 GID: 1971600007 Password: False Member of groups: ipausers Kerberos keys available: False # kadmin -p krbadmin Authenticating as principal krbadmin with password. Password for krbadmin@CENTOS-LDAP.LOCAL: kadmin: get_principal nfs/server.domain.com@DOMAIN.COM get_principal: Operation requires ``get'' privilege while retrieving "nfs/server.domain.com@DOMAIN.COM".

The ONTAP error we’d see is less clear, however. The error suggests that the SPN *already exists*.

Error: NFS Kerberos bind SPN procedure failed [ 0 ms] Creating account in Unix KDC [ 30] Successfully connected to ip 10.x.x.x, port 749 using TCP **[ 40] FAILURE: Unexpected state: Error 1142 at ** file:src/utils/secd_kadmin_utils.cpp ** func:createVifKrbAccountUsingKadmin line:227 **[ 40] FAILURE: spn already exists. Failed to reuse spn ** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using ** admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error: ** Unknown code 0 [ 41] Uncaptured failure while creating account

Obviously, the error should refer to some permissions issue. When we see this error in ONTAP, we can verify on the KDC what is happening via the /var/log/kadmind.log file. In this case, we see “unauthorized request.”

Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Request: kadm5_init, kadmin/admin@CENTOS-LDAP.LOCAL, success, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x, vers=2, flavor=6 Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Unauthorized request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

To give permissions to users to use remote kadmin, you have to modify the /var/kerberos/krb5kdc/kadm5.acl file.

In my setup, this is how I configured the ACL:

# cat /var/kerberos/krb5kdc/kadm5.acl */admin@CENTOS-LDAP.LOCAL * ontap@CENTOS-LDAP.LOCAL * krbadmin@CENTOS-LDAP.LOCAL *

Then you restart IPA services:

# service ipa restart Redirecting to /bin/systemctl restart ipa.service

Now, my user can query the principal:

kadmin: get_principal nfs/server.domain.com@DOMAIN.COM get_principal: Principal does not exist while retrieving "nfs/server.domain.com@DOMAIN.COM".

However, ONTAP gets a new (less descriptive) error when trying to interact with the KDC:

Error: NFS Kerberos bind SPN procedure failed [ 1 ms] Creating account in Unix KDC [ 29] Successfully connected to ip x.x.x.x, port 749 using TCP **[ 45] FAILURE: Unexpected state: Error 1142 at ** file:src/utils/secd_kadmin_utils.cpp ** func:createVifKrbAccountUsingKadmin line:223 **[ 45] FAILURE: Failed to create spn ** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using ** admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error: ** Invalid argument [ 45] Uncaptured failure while creating account

The /var/log/kadmind.log is also fairly useless here:

Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Principal does not exist, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_create_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid argument, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

ONTAP is just reporting what kadmin says!

Our clue comes from when we try to manually create the SPN using kadmin:

kadmin: add_principal -randkey nfs/server.domain.com@DOMAIN.COM WARNING: no policy specified for nfs/server.domain.com@DOMAIN.COM; defaulting to no policy add_principal: Kerberos database constraints violated while creating "nfs/server.domain.com@DOMAIN.COM".

Same goes for modify_principal:

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL (vserver nfs kerberos interface enable) Username: ontap Password: Warning: An account that matches the given service principal name already exists. Re-using this account deletes and re-creates the account if it is not shared by the LIFs in Vserver "NFS". This invalidates any existing Kerberos service tickets and keys for this account. Do you want to re-use this account? {y|n}: y Error: NFS Kerberos bind SPN procedure failed [ 0 ms] Creating account in Unix KDC [ 39] Successfully connected to ip x.x.x.x, port 749 using TCP [ 45] Re-using the account for spn=nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL **[ 53] FAILURE: Unexpected state: Error 1142 at ** file:src/utils/secd_kadmin_utils.cpp ** func:randomizePasswordUsingKadmin line:287 **[ 53] FAILURE: Failed to randomize password for spn ** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using ** admin spn 'ontap@CENTOS-LDAP.LOCAL' [ 54] Uncaptured failure while creating account Mar 24 11:14:54 centos8-ipa.centos-ldap.local kadmind[17208](Notice): Request: kadm5_randkey_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid key/salt tuples, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

And delete_principal:

::*> kerberos interface disable -vserver NFS -lif ipa-krb (vserver nfs kerberos interface disable) Username: ontap Password: Warning: This command deletes the service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" from the machine account on the KDC. Do you want to continue? {y|n}: y Error: command failed: Failed to disable NFS Kerberos on LIF "ipa-krb". Failed to delete the account associated with the Kerberos service principal name. Reason: cifs smb kadmin error.

So that means, even though we gave full rights to that user in the kadm5.acl file, we can’t create principals in FreeIPA using kadmin. They want to funnel you to use the IPA tools – likely for a good reason. There’s probably a bunch of other automated steps that take place with those tools, so this is in the interest of simplicity. and security.

How do we get past this?

There are two options here.

Open up the permissions on FreeIPA to use kadmin to create/add/modify/delete principals. (I haven’t figured out how to do this yet) Manually create the SPN and keytab file and then use ONTAP to transfer the keytab via URI

I went with option 2.

The TL;DR steps, as provided by Alexander Bokovoy in the comments:

- kinit admin - ipa host-add demo-ipa.centos-ldap.local - ipa service-add nfs/demo-ipa.centos-ldap.local - ipa-getkeytab -p nfs/demo-ipa.centos-ldap.local -k ./nfs.keytab -e aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96

Remember that ONTAP only supports the following enctypes for NFS Kerberos:

AES-128 and AES-256

DES and 3DES

Then, you copy that file to your HTTP or FTP server. The address to the file will be used in the ONTAP CLI command.

Note: If you’re using IIS, you may have to allow .keytab as a MIME type.

For example:

Once the file is web-accessible, you run the kerberos interface enable command and use the -keytab-uri option to upload the keytab. Unsupported enctypes will get discarded.

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL -keytab-uri http://web-server/files/ipakrb-ontap.keytab (vserver nfs kerberos interface enable) Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL". Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL". Warning: Keys for encryption types "des-cbc-crc,des3-cbc-sha1,aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96" are required for Vserver "NFS" but found keys only for encryption types "aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96". Keys for encryption types "des-cbc-crc,des3-cbc-sha1" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" are missing. Available keys will be imported. Do you want to continue? {y|n}: y Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL". Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

You’ll also want to ensure you have a valid UNIX user or name mapping rule for the krb-spn name mapping for the NFS clients. That’s covered in TR-4616 in detail.

Once that’s all done, NFS Kerberos mounts should work fine!

[root@centos8-1 sysconfig]# mount -o nfsvers=4,minorversion=1,sec=krb5 x.x.x.x:/kerberos /mnt [root@centos8-1 sysconfig]# mount | grep krb x.x.x.x:/kerberos on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=x.x.x.y,local_lock=none,addr=x.x.x.x) # su ipa-user sh-4.4$ cd /mnt sh: cd: /mnt: Permission denied sh-4.4$ kinit Password for ipa-user@CENTOS-LDAP.LOCAL: sh-4.4$ klist -e Ticket cache: KCM:1971600003 Default principal: ipa-user@CENTOS-LDAP.LOCAL Valid starting Expires Service principal 03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 sh-4.4$ cd /mnt sh-4.4$ klist -e Ticket cache: KCM:1971600003 Default principal: ipa-user@CENTOS-LDAP.LOCAL Valid starting Expires Service principal 03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 03/23/2020 14:40:16 03/24/2020 14:39:27 nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 sh-4.4$ touch krb5file sh-4.4$ ls -la total 4 drwxrwxrwx. 2 root root 4096 Mar 23 14:40 . dr-xr-xr-x. 17 root root 224 Jan 21 10:03 .. -rw-------. 1 ipa-user admins 0 Mar 23 14:40 krb5file -rw-r--r--. 1 root root 0 Mar 23 14:15 v3nokrb -rw-r--r--. 1 root root 0 Mar 23 14:16 v4nokrb -rw-------. 1 ipa-user admins 0 Mar 23 14:22 v4nokrb-ipa

I opened a bug to make this easier in ONTAP with FreeIPA (1308667), so if you run into this, open a case and have it attached to the bug.

If you have questions, suggestions or get stuck (or if I got something wrong), comment below!