Well, the first reason, I think the antlers are not so good thing for magic. I mean, the unicorn's horn are pointy thing, like a magic wand, good for beams, and antlers not.

And second reason. I just thought about female deers who have no antlers. So females couldn't do magic. I thought it's not fair.

That's why.