How To Resolve a Volume is Busy Error on Cinder With Ceph Block Storage

Fri May 22 15:11:55 2015

Last edited Fri May 22 05:24:22 2015 Posted by Craige McWhirter on

When deleting a volume in OpenStack you may sometimes get an error message stating that Cinder was unable to delete the volume because the volume was busy:

2015-05-21 23:31:41.160 16911 ERROR cinder.volume.manager [req-6f77ef4d-bbff-4ff4-8a3e-4c6b264ac5ca \ 04b7cb61dd3f4f2f8f80bbd9833addbd 5903e3bda1e840d492fe79fb840acacc - - -] Cannot delete volume \ f8867d43-bc82-404e-bcf5-6d345c32269e: volume is busy

There are a number of reasons why a volume may be reported by Ceph as busy, however the most common reason in my experience has been that a Cinder client connection has not yet been closed, possibly because a client crashed.

If you were to look at the volume in Cinder, that status is usually available , the record looks in order. When you check Ceph, you'll see that the volume still exists there too.

% cinder show f8867d43-bc82-404e-bcf5-6d345c32269e | grep status | status | available | # rbd -p my.ceph.cinder.pool ls | grep f8867d43-bc82-404e-bcf5-6d345c32269e volume-f8867d43-bc82-404e-bcf5-6d345c32269e

Perhaps there's a lock on this volume. Let's check for locks and then remove them if we find one:

# rbd lock list my.ceph.cinder.pool/volume-f8867d43-bc82-404e-bcf5-6d345c32269e

If there are any locks on the volume, you can use lock remove using the id and locker from the previous command to delete the lock:

# rbd lock remove <image-name> <id> <locker>

What if there are no locks on the volume but you're still unable to delete it from either Cinder or Ceph? Let's check for snapshots:

# rbd -p my.ceph.cinder.pool snap ls volume-f8867d43-bc82-404e-bcf5-6d345c32269e SNAPID NAME SIZE 2072 snapshot-33c4309a-d5f7-4ae1-946d-66ba4f5cdce3 25600 MB

When you attempt to delete that snapshot you will get the following:

# rbd snap rm my.ceph.cinder.pool/volume-f8867d43-bc82-404e-bcf5-6d345c32269e@snapshot-33c4309a-d5f7-4ae1-946d-66ba4f5cdce3 rbd: snapshot 'snapshot-33c4309a-d5f7-4ae1-946d-66ba4f5cdce3' is protected from removal. 2015-05-22 01:21:52.504966 7f864f71c880 -1 librbd: removing snapshot from header failed: (16) Device or resource busy

This reveals that it was the snapshot that was busy and locked all along.

Now we need to unprotect the snapshot:

# rbd snap unprotect my.ceph.cinder.pool/volume-f8867d43-bc82-404e-bcf5-6d345c32269e@snapshot-33c4309a-d5f7-4ae1-946d-66ba4f5cdce3

You should now be able to delete the volume and it's snapshot via Cinder.

Enjoy :-)