A July trip to the Caribbean was more than just a summer vacation for John Oliver and Kate Norley.

"We were in St. Thomas on holiday," when The Daily Show correspondent, 33, decided to pop the question to his girlfriend of two years, he told PEOPLE Saturday at Comedy Central's "Night Of Too Many Stars: An Overbooked Concert For Autism Education."