Philosophical arguments are made mathematical all the time. Its why you will see First Order Logic symbols thrown around on this Stack Exchange.

I think the big difference between mathematics and philosophy is that mathematics tends to start from something like a formal system, and see how much can be proven within it. Philosophy approaches the question of "what formal systems are right?" If a formal system proves something non-intuitive, Philosophers will immediately start studying the axioms of the formal system to see if they may be missing something. Philosophers admit more shades of "color" into their arguments than mathematicians can.

There are absolutely places where they blur. Consider the huge debate between ZF and ZFC in set theory. If you look at the wording choices, they are hard to distinguish from a philosophical debate. At the extreme, Proof theory and Model theory are almost more philosophical than mathematical. A read of the consequences of Tarski's definition of Truth is really almost entirely philosophy with only a pinch of Math for seasoning. Russel and Whitehead's "Principia Mathematica" could qualify as an attempt to make a religion out of pure math.

On the other side, philosophical issues like Xeno's paradox were deemed "insufficiently explained" until Calculus turned his philosophical argument into a mathematical argument. Mathematical inconsistency is often used to refute a position by using the strict meanings of FOL symbols like "AND" and "OR." These are used to force philosophers to identify where their arguments diverge from the world of mathematics. Turing's mathematical work strongly influences modern philosophy, especially with the development of AIs continuing as it is.

The real true difference is probably more linguistic than anything else. There is clearly a continuous gamut from pure philosophy to pure mathematics. However, we as humans tend to draw lines between terms because that makes them more meaningful. Mathematics tends to be colder than philosophy. Something is either formally proven, formally disproven, or an interesting outstanding problem. Philosophy is more open. Two philosophies may persist for centuries, not because nobody disproved one of the, but because people like both of them.

I think Philosophy is more willing to accept mathematics. However, mathematics avoids associating with philosophy except under well controlled conditions (such as those needed to support the richness that explodes from Formal Systems). This could account for the asymmetry in terminology you see.

Note: I say nothing about philosophers and mathematicians. The people in each group are as complicated as any person. I'm just talking about the topic described by the words.