(CNN) The man accused of starting a Southern California blaze that forced the evacuation of 20,000 residents sent a text to a volunteer fire chief two weeks ago saying, "The place is going to burn," the chief said Thursday.

The Holy Fire started Monday in the Cleveland National Forest's Holy Jim Canyon and has so far destroyed a dozen structures, according to fire authorities.

Holy Jim Volunteer Fire Department Chief Mike Milligan, 71, says he's known the arson suspect, Forrest Gordon Clark, for decades and has long warned that he posed a danger to the community.

JUST WATCHED Woman visits home destroyed by wildfire Replay More Videos ... MUST WATCH Woman visits home destroyed by wildfire 01:25

"I've been trying for years to get someone to pay attention and nobody has really had the opportunity to do that until now," he said.

Milligan said he was so wary of Clark that he avoided altogether going to the area of the remote Orange County canyon where the 51-year-old Clark lives. Nonetheless, Clark came to his home two weeks ago to return items he said he had "borrowed" from the fire department, he said.

Read More