No, language X itself is not really the problem. What you've described here are the symptoms of a much deeper problem that will affect all the other systems your team produces as well: the team isn't really working as a team but instead as a bunch of disconnected individuals.

Your instinct that you "want [your] team to be autonomous and happy, building them up, not shutting them down" is correct; you need to solve this problem by coaching the team to fix it. I would start with trying to remove the clear code ownership that's happening where the code written in language X is "his" code. It's the team's code, and the team as a whole needs to be responsible for maintaining it and extending it.

You might start with how stories (or whatever your equivalant) are assigned. Ideally all members of the team should be regularly working on all parts of the system, rather than specific parts of the system "belonging" to particular developers. It's fine if you have some developers who are experts in a particular technologies or parts of the system while others are not, but there should be nothing in your system where there aren't at least two developers who feel they know that part well enough to be the technical lead on any particular story where it's a major component.

Pair programming and just asking for help are typical solutions for spreading knowledge. Extreme Programming handles this by allowing anybody to pick and be the lead on a story, no matter how much or little they know about it, and relies on developers not being allowed to say "no" to requests for help with that story, allowing that developer as he gets his help to learn all the gory details necessary to getting the story done. (Story assignment in XP is actually a project management function: the developer managing a story for that iteration is responsible for making sure it gets done, finding all the resources necessary to do so. This will typically result in the developer having a reasonably deep technical understanding of the story by the end of the iteration, whether she started out with that or not.)

So explain to your team that you're happy with using language X, or anything else, if the entire team has a consensus that it's a good idea and agrees that if the developer in question is kindnapped by aliens the team will carry on just fine without him. It's then up to that developer to sell his technical ideas within the team. Make it clear that even if the team doesn't agree to make X the primary development language for all new features this doesn't mean it can't be used at all; perhaps it could be used in just certain smaller areas in order to test it and gain experience with it. Encourage regular reviews both of how language X is working out where it is used and of specific subsystems (whether using language X or not) regarding how well they're working, what kind of problems are typically coming out of it, and how these can be mitigated.