Did it happen to you that ansible hanged in “Gathering facts”? It happened to me today, the root cause discovery was important for me, but the debugging method may be quite similar for other issues.

How to debug ansible hanging in “Gathering facts”?

First thing you should do is to execute ansible with -vvv options to increase verbosity. In my case blue font, which is the default for verbose messages, is not very visible on the screen, so I’m adding ANSIBLE_NOCOLOR=yes as environment variable. Finally, the command looks like:

# ANSIBLE_NOCOLOR=yes ansible-playbook -vvv -i ./hosts -l host01 -DC -t rsyslog -DC ./common.yml [...] SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=30 -o ControlPath=/home/user/.ansible/cp/02cc38bba9 -tt 10.234.56.24 '/bin/sh -c '"'"'sudo -H -S -n -u ro ot /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ltxvshvezrnmumzdprccoiekhjheuwxt; /usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1527099315.31-224479822965785/setup.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''

The execution hanged here, so we know that ssh to the host worked. We have the command that was executed. As we could guess it’s setup.py, because gathering facts is done by setup module. The file is probably still on the host since the execution hanged and ansible didn’t execute cleanup on files in .ansible directory. If in your case it is rather failure then hang-up and you cannot find those files just add DEFAULT_KEEP_REMOTE_FILES=yes to your ansible-playbook execution, like here:

#DEFAULT_KEEP_REMOTE_FILES=yes ansible-playbook -vvv -i ./hosts -l host01 -DC -t rsyslog -DC ./common.yml

Then login to the host01 as the user used to span ansible tasks on the host, and simply execute:

strace -f /bin/sh -c “echo BECOME-SUCCESS-ltxvshvezrnmumzdprccoiekhjheuwxt; /usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1527099315.31-224479822965785/setup.py && sleep 0”

In my case it hanged at execution of statfs system call on one of NFS filesystems. BUM! The issue found, easily verified with df /on/appropriate/path , which also hanged.

Other approach that may be helpful in different cases, especially bugs in setup.py, may be to execute the module under pyton debuger like pdb . This approach is greatly covered on Will Thames’ tech blog post