I have a headless box that will be used for CUDA computing. I installed Ubuntu server 16.04.1 but when I try to install the nVidia drivers via apt-get it also wants me to install X11 and other GUI components.

How can I just install the driver and CUDA libs without installing X11 and friends?

Ideally without downloading the run file directly from nVidia. I'd like to use debian packages if possible.