Though I don't think there's an ironclad proof one way or another (unless it comes from one of the books), I think the evidence suggests it is naturally curly but that he has it straightened.

We know from Schisms that Worf makes regular trips to the barbershop, at least while serving on the Enterprise. While only the length of his hair is addressed in the episode, knowing that he has it cut (and that he cares about its appearance, warning Mot not to take off too much again) makes it extremely plausible that he would straighten or curl it too.

The reason I believe it is naturally curly and he straightens it, as opposed to the other way around, is when we see him with curly hair. I believe the top right image in your question is the best indicator, because this is a screenshot from Insurrection, in which Worf (and others) are experiencing the rejuvenating effects of the planet within the Briar Patch. (He had to deal with a Klingon pimple in an earlier scene.)

Certainly, we see Worf with both straight and curly hair in situations where he has access to 24th century comforts, but this is to be expected. (People change their hairstyles periodically, and Worf in particular struggles with his identity as a Klingon in a human environment throughout the franchise.) But I don't know of a time when he was without the comforts of the Federation and still has straight hair. (The exception being that if he goes into a situation with a straightened pony tail, it appears to stay that way if he doesn't touch it.)