Linux is known to be very stable, but even the most stable system can encounter situations that cause it to become unresponsive. Then comes the awful feeling as you push and hold “Power”. There are a few tricks with Linux one can attempt in such a case. At this point, we are not looking to save any unsaved work. We want to find a way to avoid a reboot or at least provide the cleanest reboot possible to avoid corruption.

For the purpose of this article, Linux can be seen as a set of 2 stacks: The Kernel Stack and The User Stack. Somewhere inside the User Stack, you have programs running. More than likely X server, a window manager and some programs. You can imagine that list of programs, including the X server, as another stack inside the User Stack. This concept will help newer Linux user understand some of the task we will perform later.

1. Kill an unresponsive program with xkill

If you are using a Linux desktop, your first step should be an attempt to use your window manager’s execute command (alt+f2 in Ubuntu) feature. If you are lucky, it will popup sooner then later.

Then you can execute xkill and point the cursor at the defunct window and click it away.

Xkill works faster than the traditional X icon or right-clicking the window on the taskbar and clicking close. Xkill kills instantly without questions asked.

2. Revive a frozen Linux system with virtual consoles (TTY)

Sometimes, you will not be so lucky, the system may seem completely dead.

This could just be an illusion. Remember the stack of programs inside the user stack? Let’s work with this image for a little bit.

While it may seem like the root of the User Stack would be X. It is not.

X Window System runs on a Virtual Console (VC), also referred to as TTY. They are represented on the filesystem as /dev/ttyX. Linux has multiple VC’s available to you. Attempting to access another one can give you back control.

You can request a different VC by pressing CTRL+ALT+F{1,6} from a GUI environment or ALT+F{1,6} from another VC. The above keystroke gives you access to VC 1 through 6. Graphical interface runs on VC 7+.

When dealing with an unstable system, even requesting a change of VC can take a little time, so be a little patient.

If your attempt to change virtual console is successful, you will be prompt for your user and password. You will then have a text mode console. Your system should regain some stability here, this is in parts because the virtual console where X and everything else is running is not a priority anymore.

Once you are logged in you have a few options.

You can attempt to kill a defunct program with the kill or killall command.

You can kill the window manager, X server or login manager. This will kill all child processes and, if all goes well, your login manager will restart presenting you with your graphical login screen.

3. When everything else fails, talk with the Kernel using SysReq

Sometime, even switching to a TTY does not work. The whole system is dead. Don’t give up yet, once again, more than likely an illusion. Linux holds one more feature for situations like this.

If compiled in the Kernel and enabled in a file, Linux provides you with the means to send preset requests directly to the kernel. This is useful in a situation when the entire user stack is dead. It is known as the Magic SysRq key. Compiled and enabled by default in most distribution, only a Kernel Panic should be able to prevent this from working.

To confirm the option was set when the kernel was compiled, you can attempt to locate your current kernel’s config file in /boot. Look for the option CONFIG_MAGIC_SYSRQ. As previously mentioned it is compiled and enabled in most distributions. To confirmed it is enabled, run the following:

sysctl kernel.sysrq

On older systems, you had 2 options: 0 & 1. It was either OFF or ON. On newer ones, there is more fine control over which commands are enabled.

Any value above 1 means some commands are enabled. More details can be found here.

To set a different value temporarily, run the following command:

sudo sysctl -w kernel.sysrq=value

To change this value permanently, edit /etc/sysctl.conf. Search for kernel.sysrq, and set the desired value.

The easiest way to see if it works is to try. I also recommend testing and learning a set of keystroke while the system is stable. When you execute a SysRq call, the kernel will attempt to respond. This response can only be seen if you are in a virtual console 1 through 6. If the focus cannot be moved away from X, you will not see the responses. This does not mean it is not working.

There are numerous commands that can be sent. The full list can be seen here. I recommend memorizing the following:

Keys Description Alt + SysRq + r Takes the keyboard out of raw mode, taking control away from X Alt + SysRq + e Send SigTerm to all process, giving them a chance to quit gracefully Alt + SysRq + i Send SigKill to all process Alt + SysRq + k Kill all process in current virtual console Alt + SysRq + s Sync all mounted filesystem, flushing all data to disk Alt + SysRq + u Remount all filesystem read-only Alt + SysRq + b Reboot system instantly, does not sync or unmount Alt + SysRq + o Shutdown system

This short list of commands gives you a powerful new tool when dealing with an unstable system. When sending request, if you are unable to see the kernel’s responses, give a few seconds in between commands. Pay attention to HDD light when requesting HDD operation.

There is 2 sequence that are worth remembering.

Alt + SysRq + r e k

This sequence is good to restart an unresponsive X server. You first get the keyboard out of raw mode with Alt + SysRq + r. You then request to send a SIGTERM to all processes with Alt + SysRq + e. X server may restart with this request. If so, you have rescued your system. If not, proceed with the third request, Alt + SysRq + k. Which will send SIGKILL to all processes within the current virtual console.

If your system is still unresponsive after the above sequence, you should proceed to this next sequence.

Alt + SysRq + r e i s u b

This last command will initiate an emergency reboot.

r : Get keyboard out of raw mode

: Get keyboard out of raw mode e : Send SIGTERM to all processes

: Send SIGTERM to all processes i : Send SIGKILL to all processes

: Send SIGKILL to all processes s : Sync all mounted filesyste

: Sync all mounted filesyste u : Emergency remount read only

: Emergency remount read only b : Instantly reboot system

You are now equipped with the best tool against an unresponsive system that I know. You may follow this video from my YouTube channel for a demonstration.

I hope you learned something new about dealing with a frozen Linux system with this article. I would like to know your feedback on it. Please use the comment section and share your thoughts.