One day before the outbreak of the first world war, a precocious boy called Gerhard Scholem burst into a room at home and began the rite of symbolically castrating his father. “Papa, I think I want to be a Jew,” he exclaimed. He was planning to learn Hebrew, study the Bible and become a Zionist. His father, an assimilationist German businessman who despised his Jewish heritage, was appalled: “You want to return to the ghetto?” he asked. “You’re the ones who are living in the ghetto,” his son snapped back. “Only you won’t admit it.”

Scholem meant that his father had established the family in a gilded bourgeois Jewish prison within a hostile German society – his friend, Walter Benjamin, who grew up in a similarly privileged west Berlin milieu, described it as “something of a ghetto held on lease”. These rebellious sons turned out to be unwittingly prescient.

Such Oedipal confrontations were common in German-speaking lands during the late 19th and early 20th centuries, as the privileged sons of Jewish businessmen rebelled against their fathers’ devotion to bourgeois accumulation and deluded patriotism for a Wilhelmine polity that denied Jews equal rights. Some rebels such as Gerhard’s brother Werner (who would die in 1940 in Buchenwald) became communists. Others, Scholem for one, were attracted by the Zionist hopes advanced by political activist Theodor Herzl and philosopher Martin Buber. Scholem kept a portrait of the former on his bedroom wall, and felt a jolt of electricity when he heard Buber lecture and identify Jews as Orientals for whom the priority was mutuality and community, processes and relationships – against “the atomised, petrified western man of the senses”.

He dreamed that the Jews could replace, as George Prochnik puts it, “the attitude of impotent suffering with rambunctious perilously naked self-expression”. Instead of being strangers in European lands, Jews could go home and become themselves – not hobbled melancholics of the diaspora, nor spiritless worshippers of degrading consumer capitalism.

David Ben-Gurion (centre), Israel’s first prime minister, giving an address in Tel Aviv, 1948. Photograph: AP

This at least was the messianic dream that led Scholem in 1923 to quit Germany for Palestine, where the young philologist and scholar of that mystical thread of Judaism called Kabbalah spent the rest of his life. That was the year of Hitler’s beer hall putsch, only 10 years before the Nazi leader was elected German chancellor and 19 before the Wannsee conference implemented the Final Solution. Scholem figures as an antithesis of Stefan Zweig, subject of Prochnik’s previous book, the cosmopolitan humanist who couldn’t abandon his idealised vision of European culture: Scholem is a hero to the author because of the virtuosity with which he developed alternative, non-European Jewish, visions.

In Jerusalem, Scholem changed his first name, becoming Gershom – the name given by Moses to his son after the escape from Egypt. It was after that escape that Moses said: “I have been a stranger in a strange land,” with its implication (encoded in one meaning of the name Gershom) that the prophet was home after the woes of exile. But, for Scholem, anarchically esoteric exegete that he was, Gershom also meant “Stranger is his name”. Prochnik tells us he revelled in this paradox, glossing it thus: “Once a stranger, now home; forever a stranger, by destiny”. He was never at home, not quite.

Certainly if Scholem had been a stranger in Germany, he was to be differently alienated in his new home. Prochnik takes us through the history of British mandate-era Palestine to the creation in 1948 of the state of Israel, whose birth pangs were witnessed by Scholem. From the heights of the newly founded Hebrew University on Mount Scopus, the professor looked down on his disappointing people like a new Moses on Mount Sinai – finding that, instead of Zionism meaning the transcendent salvation of the Jewish people, in practice it involved the recrudescence of godless western materialism, dubious religious conservatism and heartless treatment of the Arab population. He stayed but became something Guardian readers can identify with, a remoaner.

Yet Scholem also became a source of pride to the new Israel, symbol not of its martial valour or economic chutzpah but its intellectual excellence. Prime minister David Ben-Gurion reportedly shut his office for five days in 1957 and went to bed to read Scholem’s magnum opus on the false Jewish messiah, Sabbatai Sevi. Imagine, by way of parallel, Theresa May suspending Brexit negotiations to curl up with volume three of philosopher Derek Parfit’s On What Matters.

But that’s only one thread of this ardent, beautifully written book. The other is Prochnik’s parallel rebellion and Zionist awakening in the late 1980s when he quit the US, converted to Judaism, learned Hebrew and settled in Jerusalem. He describes his upbringing in Fairfax City, Virginia, as spiritual violation. The despoliation he witnessed as malls consumed America’s wilderness resonated somehow with the ruins “of European history that my father’s family had fled”. Israel promised, or so it seemed, escape from both ruins. Like his hero, Prochnik was a first-born son sticking it to the old man for letting “the flame of his Jewish identity burn down as low as it could go”.

And so, one day in the late 80s he flew to Israel clutching his battered copy of Scholem’s On the Kabbalah and Its Mysticism. In Jerusalem he married Anne, an artist and teacher, and raised three children, all the while struggling with his writing career and with his place in, and commitment to, Israel. Like Scholem, only more so, he was both beguiled by and estranged from its realisation of Zionism – rising consumerism, irksome dress codes, the unresolved Arab question.

Harvesting wheat in a field near Modi’in, Israel. Photograph: Abir Sultan/EPA

As he delved deeper into Scholem’s mystical Jewish thought, Prochnik explored the notion of the Shekinah, roughly the glory of the divine presence interpreted in Kabbalism in feminine terms. Just as Aristophanes had imagined in Plato’s Symposium human nature split into gendered halves who yearned for their original wholeness, so Scholem suggested that “a part of God Himself is exiled from God”, namely the feminine part, who is the spiritual personification of exile and Jewish exile in particular. For Kabbalists and for Scholem, humanity’s task of bringing God’s masculine and feminine aspects back to their foundational unity was akin to Zionism’s dream of overcoming Jews’ exile.

One imagines Prochnik looking up from his exciting esoteric readings, to be confronted with Israel’s sometimes disappointing reality, gender-wise. Jewish religious practice, at worst, was hardly premised on rediscovering that foundational unity: “Orthodoxy effectively cut women out completely from non-domestic religious activity,” Prochnik writes. “I felt there had to be a more meaningful role for women than just replicating male functions in a ritual territory demarcated and dust covered by men.”

Other demarcations slowly impinge on Prochnik. At one poignant moment, he wonders why Arab boys cleaning tables at a restaurant aren’t at school. While he is graceful in admitting his omissions of empathy, the book reads as if he sometimes lost sight of how, to repurpose Walter Benjamin’s remark, Israeli civilisation, like every other in human history, has its barbarous flip side.

His estrangement came to a head with the 1995 murder of Yitzhak Rabin by ultraconservative student Yigal Amir, who was opposed to the prime minister’s support for the Oslo peace accords that entailed Israeli withdrawal from West Bank settlements. The villain of Israel’s recent history, for Prochnik, emerges as the nation’s current prime minister Benjamin Netanyahu, who, before Rabin’s murder, made “the breasts of conservatives swell with a sense of their own vigilante machismo while sending psychopaths into a frenzy”.

He doesn’t quite indict Netanyahu for creating favourable circumstances for Rabin’s assassination, but when “Bibi” was months later elected prime minister, it was time for Prochnik and Anne to leave for the US. He writes: “the very thing that once drew us was what we needed to renounce”. Worse, their marriage, founded on the joy of their Israeli adventure, couldn’t survive the rupture. What remains, however, is Prochnik’s adoration for Scholem, and his unrealised and perhaps unrealisable notion of Zionist transcendence.

• Stuart Jeffries’s Grand Hotel Abyss: The Lives of the Frankfurt School is published by Verso.

Stranger in a Strange Land: Searching for Gershon Scholem and Jerusalem s published by Granta. To order a copy for £21.25 (RRP £25) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.