I have had processes terminate (probably killed or crashed) and continue to use resources, but were not listed in nvidia-smi . Usually these processes were just taking gpu memory.

If you think you have a process using resources on a GPU and it is not being shown in nvidia-smi , you can try running this command to double check. It will show you which processes are using your GPUs.

sudo fuser -v /dev/nvidia*

This works on EL7, Ubuntu or other distributions might have their nvidia devices listed under another name/location.