If you are using Raspberry PI as headless machine, you do not need to start the graphical server at all. It is useless when Rasbian is used as server OS and disabling it will free ~40MB of RAM.

To do so, you need to edit the /etc/X11/default-display-manager file using your favorite text editor.

$ sudo vim /etc/X11/default-display-manager

Then comment out the line

#/usr/sbin/lightdm

and add a new one

/bin/false

Save the file and reboot the machine. Voila!

Update

There is a easier way of doing it…

Just run the Rasbian configuration utility

$ sudo raspi-config

In boot option menu choose: