As far as I know, no. We could make one up, here. "-phile" and "-phobe" are derived from the greek words "philia" (love) and "phobos" (fear), so we'd want to look for an greek word meaning "indifference", I think.

"adiaphoria" looks like an early contender, being what Google translate comes back with for "indifference". There is already a philosophical concept "adiaphora" (note no 'i' at the end) meaning "indifferent things", referring to (my summary) things which are neither moral nor immoral. https://en.wikipedia.org/wiki/Adiaphora

This isn't quite what we're after, and it's a bit long, anyway - it would be nice to be able to have a single-syllable suffix like "phile" or "phobe".

"neutral" gives "oudéteros", which again is a bit long.

Perhaps we could use "mesis" which means "middle" - so you could have the adjective suffix "-mesic" like "phobic", and the noun suffix "-meso" like "phobe".

Eg, an "audiomeso" is someone who neither hates nor loves music, and if you genuinely didn't give a sh*t about whether someone is gay, you could call yourself "homomesic".

EDIT: @Nathaniel, in the comments, pointed out that the noun form suffix should be "-mese", not "-meso", so "audiomese" or "homomese". This feels right to me and I thank him for it.