Actually, I seriously dislike the words religion and religious. First, there is no such thing as generic religiosity. There are Christians and Jews and Muslims and Hindus. No one practises religion, as such. And second, precisely because the word "religion" describes the common outward format through which these very different belief systems express themselves, it cannot describe each in its specificity. This is particularly tricky when it comes to Christianity, because at its heart is a figure who was thoroughly suspicious and condemnatory of religion. "Jesus came to abolish religion," says the Washington-based poet and evangelist Jefferson Bethke. His YouTube poem Why I Hate Religion, But Love Jesus received 16 million views within two weeks of it being released. He's right: the New Testament must be one of the most thoroughly anti-religious books ever written. It makes Richard Dawkins look very tame fare indeed.

Jesus spent much of his time laying into the pious and the holy, and lambasting the religious professionals of his day. And this was not because he was anti-Jewish – as some superficial readings of his anti-Pharisee, anti-Sadducee, anti-Temple polemics would have it – but precisely because, as a Jew himself, he came out of that very Jewish prophetic tradition of fierce hostility to religiosity. Here, for instance, is the prophet Isaiah on feisty form.

The multitude of your sacrifices – what are they to me? says the Lord.

I have more than enough of burnt offerings, of rams and the fat of fattened animals; I have no pleasure in the blood of bulls and lambs and goats. Stop bringing meaningless offerings!

Your incense is detestable to me. New moons, sabbaths and convocations – I cannot bear your worthless assemblies.

This is the sort of theology to which Jesus looked for inspiration. And partly, it was this uncompromising anti-religiosity that got him nailed to a cross.

You may think I am being slippery with the word "religion". So let's take, for instance, the French sociologist Emile Durkheim's influential definition of religion as that which divides up the world into the sacred and the profane. But here again, the Jesus stories have him as thoroughly anti-religious, not least in those narratives that surround his birth. For the idea that God might be found not in the spiritually antiseptic space of some sacred temple, but in the smelly cow shed out the back, is about as hostile an idea to religion as one can possibly imagine. When it comes to Christianity, just being a little bit religious is being too religious. Religion is a pejorative term. So the answer is yes.

Of course, I'd say you cannot be too Christian. That's a different kettle of fish. And if "being too Christian" makes you think of Christian fundamentalists, I'd want to insist that they are simply not Christian enough. Indeed, that it's their lack of faith that makes them cling to a bogus form of certainty and literalism. Mostly, Christian fundamentalists worship a book. They like the safety of having pat answers. But this is just another form of idolatry of which the Hebrew scriptures regularly warn. Worshipping a book and worshipping God are two totally different things. Falling down before a baby, with all the inversion of power that this implies, takes courage not intellectual suicide. It is about the world being turned upside-down, the mighty (including the religious mighty) being cast down and the weak being held up. It is about placing something other than oneself at the centre of the world. And no, I don't think there can be too much of this.