Good question!

The Prophet (saws) used humor to lighten the mood in many situations. He was known to always smile. When he laughed the whites of his teeth showed. When we read those ahadith, sometimes we don't really get the joke because we're living in a different time and different things are funny to us now.

The Prophet (saws) is known to have said: I always say the truth even when I am joking (hadith in Shama'il Tirmidhi). So the general rule is to not lie when you're trying to be funny. The Prophet (saws) said woe to the one who lies in the course of making people laugh. There are many instances in the hadith literature about the Prophet (saws) using humor with his Companions (raa). He (saws) encouraged people to joke with their spouses.

One incident I remember is when the Prophet (saws) was with Ali ibn Abi Talib (ra) and they had a dozen dates. As they were about to finish, Ali (ra) saw some people approaching. He quickly moved the pile of 6 pits that he had to the Prophet (saws)'s pile and jocularly told the passersby that the Prophet (saws) had kept all the dates to himself and didn't share! The Prophet (saws) in good spirits said right back: Look at Ali, he eats the dates and he eats the pits as well! (because he didn't have any pits on his side).

So the scholars use this incident for one exception: if everyone knows what you're saying can't be true, it's OK to joke about it. If someone says for example - I was on Mars last night and something happened and proceeds to make a joke about it, everyone there knows it's not true.

Other etiquettes for joking: One is not allowed to make derogatory jokes about Islam or the Prophet (saws) or the Companions as Allah says in the Qur'an in Tawbah:65-66. One is not allowed to take someone's property as a joke, nor is one permitted to frighten or scare them in the course of a joke. And of course, general rules about backbiting, slander, mockery, etc. also apply here.

And finally, the scholars have said that excessive joking (just like going to excess in any matter in life) can harden the heart. The Prophet (saws) said “Do not laugh too much, for laughing too much deadens the heart.” (Saheeh al-Jaami’, 7312). DO it respectfully, let it relax you, but don't let it distract from the akhirah.