People who don't collect stamps don't have a history of being burnt alive at the stake for their views. Society decided it needed a special word for such victims of religion.

By the way though, people with a fear of stamps are called timbrophobic.