Lambeth Palace, mucked about with down the centuries and later badly damaged by German bombs, is something of a muddle, architecturally speaking. Its looming gatehouse, for instance, is early Tudor, built of the same blood-red brick as Hampton Court; while the Great Hall, ransacked by Cromwell’s men during the civil war, is 17th-century gothic (“a new old-fashion hall”, as Pepys had it). As for the building in which the archbishop of Canterbury lives, it is 19th-century neo-gothic, and resembles an Oxford college complete with quadrangle – except, that is, for a few older remains, among them Lollard’s Tower, which dates from 1435 and once housed ecclesiastical prisoners whom the authorities hoped to persuade to renounce their heresy.

Justin Welby’s study, the room where he “reads and works and thinks and prays”, is in yet another tower, also Tudor: a gloomy, somewhat cell-like, wood-panelled affair with – pull back the heavy velvet curtain – a view over the chapel below, where services are held three times a day. It’s unexpectedly touching to be invited to interview him here. There is something so intimate about the sight of his desk, on which there stands not much more than a crucifix, an icon and a Bible, open at Psalm 73 (“My flesh and my heart faileth: but God is the strength of my heart, and my portion for ever”), though we’ve been talking for almost half an hour before he reveals that it was in this room that Thomas Cranmer is reputed to have written The Book of Common Prayer. How does that make him feel, I ask, picturing his long ago predecessor thrusting his right hand into the pyre that would kill him. (Cranmer was executed by Queen Mary, to whose half-sister, Elizabeth, he was godfather, and whose father, Henry VIII… well, you know the rest.)

“It feels extraordinary and… bizarre,” he says, wonderingly.

Isn’t it quite batty to find oneself, of all things, archbishop of Canterbury? He giggles. In private, Welby can be quite deliciously camp when he wants to be.

“That’s a very good way of putting it, though I might not say ‘quite’.”

The awe soon creeps back into his voice. “Sometimes, it’s exhilarating and beautiful. You get invited to places you never dreamed of. But there’s also a deep sense of responsibility. On bad days – this is another thing of which to repent – I probably grumble, though I try not to. There is a lot to be thankful for.”

In Nigeria, a militiaman was told to take Welby out and execute him

The archbishop of Canterbury, as leader of the Church of England, is supposed to be an “instrument of communion and a focus of unity”, which is, he says, rather a strange thing to think about when you’re shaving: “Do I look like a focus of unity, I’ll think, as I stand in front of the mirror. Do I see here an instrument of communion?” What it comes down to in the end, though, is that he has a job to do, one that calls for the patience of Job and the restraint of David, for which reason he tries not to let himself get too carried away with its trappings. He favours black rather than the traditional purple, and refuses to have a driver, travelling around London – movies, galleries – by bus and tube. “I’ve got a very forgettable face,” he says. “I really have. I can move around unseen very easily.” Does he take off his dog collar? “You bet I do.”

Welby has written a book, Reimagining Britain: Foundations for Hope, in which he sets out to identify the values that might help us through the months and years ahead, a period he would rather describe as one of opportunity and challenge than of threat and fear. It’s a volume with which it’s hard to disagree if you are on the liberal left – it is about such (to me, uncontroversial) things as inclusion and the necessity for increased funding of certain institutions – though when I tell him this he only says, rather sweetly: “That’s probably because it’s not very interesting.” Nevertheless, it is important to understand that it is emphatically his book, not some statement on behalf of the church.

(No fewer than three aides listen to our conversation, incidentally, which is three times the number who sat in when I interviewed Tony Blair and David Cameron.)

“Mainly, I write for myself,” he says. “It’s how I work out what I’m thinking. But the catalyst was the Brexit vote.” He was here, at Lambeth, when the referendum result came in. Was he surprised? “No, not especially. It’s a reality. You deal with life as it comes. But it accelerated thinking I was already doing about the way we live, about how we might build a country where everyone flourishes.”

There are, he thinks, two views of the future in play at the moment: “One is sort of Voltaire’s Candide, and says all is for the best in the best of all possible worlds, and everything’s going to go brilliantly. The other is that it’s all catastrophe and disaster. But of course neither of those is true.”

Unsurprisingly, for someone who is meant to lead a flock (however small, these days), he is less interested in external events than in people, and how they are treated: by the state, by one another.

“One of my great inspirations comes from Catholic social teaching, like Pope Leo XIII in the 1890s, who tried to define the policy of the church in the light of the Industrial Revolution in his encyclical Rerum Novarum [‘Of New Things’]. Pope Francis’s encyclical Laudato Si’ [about global warming] was very much in that tradition. Both are deeply based in scripture, but they apply to everyday events. They take the church from being shut within its walls and only talking to itself, to leading us to say how we treat people. Because people are made in the image of God and they are of infinite value.” He flashes me a smile that may, or may not, be beatific. “That’s where I come from.”

Perhaps, I say, the church should simply give up trying to grow, and concentrate instead on social issues full time. Everyone knows how rapidly most congregations are shrinking; why try to stick a finger in the dam?

Welby speaks alongside fellow bishops and archbishops at Lambeth Palace in London, 2014. Photograph: Luke Macgregor/Reuters

“Ah, but it’s not an either/or,” he says. “There is the common good. The church is a partner in most food banks, in night shelters, in debt counselling. There are chaplains in every hospital, prison, regiment, sea-going vessel, squadron. Every day, we bury the dead and comfort the bereaved. We prepare people for marriage, and baptise children. We run 4,700 schools. Every single person in a parish is deeply welcome, whether they believe or not. But we should be seeking to grow, too, to bring people to God.”

And how’s this going? At this, his expression combines – it is as if he has strolled straight out of the pages of Trollope – archness and disappointment in almost equal measure. “Well, in some places it’s going better than others, it would be fair to say,” he says, crisply. “Over the last 10 years, we have created 1,300 new churches, but because of demographics and secularisation, the overall trend is slightly negative.”

For the church, problems come and go – though a few issues might best be described as eternally pending. When Welby became the 105th archbishop of Canterbury in 2013, the question of whether women would be ordained as bishops was still pressing, the church’s General Synod having voted down the legislation that would have made it possible only the year before. In 2014, however, it passed all three houses of the Synod, with the result that there are now 13 female bishops in England. Now the two major crisis points are the church’s position on same-sex marriage – while gay Christians are “welcome and supported”, marriage is still considered to be “a lifelong union between a man and a woman” – and allegations of historical child sex abuse, cases that were in many instances covered up by the church.

In 2016, it was dealing with about 3,300 safeguarding issues, a figure that includes historical allegations and non-sexual abuse. Last month, the Independent Inquiry into Child Sexual Abuse devoted three weeks of its attention to the Church of England, with a particular focus on the Diocese of Chichester (in 2015 Peter Ball, the former bishop of Lewes and of Gloucester, was convicted of two counts of indecent assault and one count of misconduct in public office, and sentenced to 32 months in prison; an independent review in 2017 found that the church hierarchy, including the former archbishop of Canterbury, George Carey, had concealed abuse carried out by Ball over a 20-year period). Both Welby and his immediate predecessor, Rowan Williams, appeared at the inquiry; Welby spoke, again, of his deep sense of shame over this matter, and for the need for the church’s culture of “clericalism”, in which excessive deference was shown to senior figures, to be strongly challenged.

Does he feel there is still a long way to go over all this? “Yes.” Are more allegations likely to emerge in the future? “Yes.” He is frustrated by the pace of change. “We have made progress. There is a safe-guarding officer in every parish; every diocese has at least one paid safe-guarding adviser who has a statutory right of independent decision from the bishop. But getting the culture deeply embedded is going to take time. I was pleased the inquiry happened. I thought we needed it. You can’t read about what happened without a profound sense of horror. It’s a terrible thing, and I am sorry we did it. There were the great sins: the abuse, and then the cover-ups. We do the most youth work [of any organisation] in the country, and that’s why it’s so shaming, so horrible. We failed.”

And what of his position on George Bell, the former bishop of Chichester? Bell, who died in 1958, was, and still is, a heroic figure, for his support of the German resistance during the war, and for his opposition to area bombing by the allies. But in 1995, an allegation of child sexual abuse was made against him – a complaint that was not passed to the police by the church until 2013 (after a second complaint was made to what was now Welby’s office). The police concluded that, had Bell been alive, he would have been arrested, and the diocese went on to pay compensation to the victim. But the church’s handling of the case was widely criticised, mostly for its lack of due process. Last year, an independent review of the case by Lord Carlile said that the church had rushed to judgment, perhaps because it wanted to avoid repeating the mistakes of the past. The church then announced that it had passed new evidence about Bell to the police. The row, however, rumbles on. In January, a group of leading historians, among them Professor Sir Ian Kershaw, urged Welby to withdraw comments he made following the publication of Carlile’s report, in which he said “a significant cloud” still hung over Bell. The archbishop’s comments were “indefensible”, they said; the allegation against Bell is uncorroborated, and in their eyes contradicted by considerable circumstantial material. Welby has so far refused to do so. Is he likely to shift his position?

“Not for the moment. Following Lord Carlile’s report on what was a badly handled inquiry, we had further information which is being investigated, and that will take a long time. Nothing could be better for the church, and for Chichester, than if we were able to say there was nothing to it, but you can’t do that until it is properly investigated. We say that if someone makes an allegation, we always take it seriously.”

It is interesting, though, that he has also said, in connection with Bell, that evil acts don’t cancel out good acts, or vice versa (I’m paraphrasing). Isn’t this something that society as a whole is wrestling with at the moment, almost every day? (I’m thinking of #MeToo, among other things.)

“Yes. I think you have put your finger on something that is hugely difficult, and that it is evident we don’t know how to deal with. Wrong behaviour must be confronted. It needs to be called out. #MeToo is very necessary. Such misuses of power are disgusting. But… do you do a balance sheet? Do you say, one bad deed wipes out any number of good deeds? Or do you try to ignore the one bad deed? None of these are good answers. They’re terrible answers. What I’m trying to feel for is the idea that every human being sins, and needs forgiveness. It is really complicated. Some people’s bad deeds do overwhelm anything good about them. But Bell’s heroism in the war… we can’t forget that.”

Forgiveness, he says, cannot be handed out like sweets; Christian thinking offers no opportunity for what the dissident German pastor Dietrich Bonhoeffer, hanged by the Nazis in 1945, called “cheap grace”. Repentance involves suffering for the consequences of one’s actions. “But we [the Christian church] don’t write people off completely. It’s not simple, because human beings aren’t simple.” He begins every day with “a lengthy period of confession”, and “it’s probably the last thing I think about at night, too”. So he does things of which he’s ashamed, too? “Yes, of course.” His eyes widen. “But I’m not going to tell you [what they are].”

Many people don’t quite believe Welby when he trots out the church’s line on gay marriage; they sense he is too liberal, and possibly too kind, to really go along with this stuff – and sure enough, in his book, he talks of the church struggling to keep up with the rest of the world on this matter. Doesn’t he get frustrated, toeing the line just to keep the peace? Wouldn’t he, as the symbolic head of the Anglican Communion worldwide, like to be able to put more pressure on, say, the church in Uganda, where gay people are on the receiving end of horrific human rights violations? “Um, let me think,” he says. “No, I don’t find it frustrating, and I’m quite open about it. Homophobia is wrong, utterly wrong. But the Church of England is one of 39 provinces, not some imperial power. We have no right to tell others what to do.”

For the time being, relative calm reigns on this “neuralgic” issue – certainly by comparison to when he first became archbishop. “People are tired of the arguments. But it could all blow up again. It easily could. I am realistic. We are not out of hospital, but we are in a slightly better condition.”

So far as his book goes, it is, for Welby, simply the case that there is no use pretending society is other than it is. “People are all sorts of things. They fall in love with the right people, and the wrong people. Some marriages are wonderful, and some are terrible. To take a personal example, you may discover that you don’t come from the family you thought you came from.” He laughs. “You’ve got to start where people are, not where you want them to be.”

Recent archbishops of Canterbury have not brought with them particularly interesting backstories; George Carey, for instance, was chiefly remarkable for the fact that he had not attended Oxbridge. But Welby is a little different. When he took over in 2013, his childhood, spent alone with his father Gavin, an alcoholic who suffered from unpredictable rages, was widely written about (Justin, who attended Eton, lived with his father following his parents’ divorce when he was three). In 2016, however, it was announced that a DNA test had revealed Welby’s father to be not Gavin Welby, but Sir Anthony Montague Browne, Winston Churchill’s last private secretary, with whom his mother, Jane, also a former alcoholic, had had a fling before she was married. At the time, Welby played down the news. “I know that I find who I am in Jesus Christ, not in genetics,” he said, when the story broke. But he also grasped, I think, that there would be some who would feel as I did when I heard the news on my car radio (I remember the moment vividly): that here was a religious figure to whom people might be more easily able to relate.

Did he feel, in 2016, as if he was a character in a Graham Greene novel?

“I wish! No, it’s just life – a life that lots and lots of people have. Someone only recently said to me, ‘I’ve just discovered I’ve got a half-sister, because my dad had a fling 50 years ago’. It’s not unusual. There have probably been previous archbishops in the same position, only they didn’t happen to know.”

Does his childhood – he spoke on Desert Island Discs of one Christmas, a “grim day”, when his father did not get out of bed, and he had to forage in the fridge for his lunch – explain his faith, to some degree?

“I often ask myself that. One of the key connections is that I have found in [God] a father who is utterly loving, trustworthy, just, who demands everything and yet who gives everything.”

The archbishop photographed last week at the Four Candles pub in Kent, tasting a craft ale brewed to raise funds for the local parish. Photograph: Gareth Fuller/PA

His favourite Bible story is that of the Prodigal Son, which he loves for its ambiguities, its sense of unfinished business (when the parable ends, the older son is still outside his father’s house; we do not know if he is reconciled to him, or to his returned brother). Then again, amateur psychology will only take you so far. “My wife grew up in a most stable and wonderful family, and she is equally committed to Christ.”

He became a Christian at Cambridge, though only with some reluctance; he understands, absolutely, how preposterous the whole thing sounds to most people, because he felt so too, then; he answered God’s call “kicking and screaming”. Slowly, though, his faith grew; when, in 1983, his baby daughter, Johanna, was killed in a car crash at seven months old, the experience only seemed to strengthen it.

After Cambridge, however, he worked perfectly happily as an executive in the oil industry. It wasn’t until the late 1980s that he was called to the priesthood, a feeling that was – he won’t deny it – quite “horrifying”. When the then bishop of Maidstone asked him, during the discernment process, why he wanted to be ordained, he said: “Well, I don’t really.” What would he do if the Church of England didn’t want him? “I’ll go back to London and take my wife for the most expensive meal I can afford, to celebrate,” he said. He loved his existing job: “I’m probably the only one who thought so, but I was quite good at it.” If the church turned him down, that would be a sign it wasn’t the right thing to do – and part of him hoped for exactly that outcome.

But the church didn’t turn him down, and having studied for the priesthood in Durham, he began life as a curate in Nuneaton. This was a shock: “Infinitely more stressful than working in finance for an oil company. It was an urban priority area, which meant deprivation. But from the first day, too, you found people opened up their lives to you. They would tell you what they were going through. You could pray with them even when they were outside the church. Just by turning up and being half human, you can make a big difference – let them know they’re not forgotten.”

In 2002, he joined Coventry Cathedral as Canon Andrew White’s co-director of international ministry (White is the priest who was later known as the vicar of Baghdad), and the pair made many trips together, crossing the Iraqi desert between Ramadi and Fallujah at 110 miles an hour; conducting a service in St George’s Anglican church, Baghdad, when it was surrounded by tanks. Later, he focused his attention on Africa, doing reconciliation work with the Ogoni in Nigeria, where conflict had claimed 100,000 lives. Twice, he phoned his wife Caroline at home in Coventry and asked her to pray for him, fearing he was about to die (on one occasion, in Nembe, a militia leader had actually told his men to take Welby outside and execute him; his life was saved by an elder). During all this, he is said never once to have removed his dog collar. Is he a very calm person? “Oh, golly, no. I wish.” Is he brave? “No.” Why did he do these things? “I did them because they were there.”

And now he must do things, not because they are there, but because they are part of this strange, slightly unreal job of his. For instance, though he will not be conducting the marriage service of Prince Harry and Meghan Markle – “the dean of Windsor is in charge: it’s a Royal Peculiar, in the technical term” – he will “do the vows”. We meet shortly before it has been officially announced that he has baptised Markle in the Church of England, but he can’t help himself: “Yes,” he says, when I ask if he’s already performed this particular task. “I think that’s known now.” (An aide yelps that it isn’t quite yet.) I suppose he’s going to tell me she’s just another person, no more special than any other child of Christ. “Well, it’s never just another person. Every person is special. Baptism is a huge privilege, and when the person is filled with sincerity and a deep sense of the presence of God, it’s fabulous.” He beams.

After this, we all stand up, and process back down the stairs to the 19th-century part of the palace – Hogwarts, he calls it. He walks, cross swinging, lightly and briskly across a red carpet, the aides bringing up the rear, the communications team PA, Fiona Barton, already waiting for us ahead. Again, I think of Trollope – what his eye might have made of this scene, at once so anachronistic and yet so modern (the PRs, the mobiles). “High rank and soft manners may not always belong to a true heart,” the chronicler of Barchester once wrote. In the case of Welby, however, you guess that they do (I am fairly certain this is not just misplaced hope on my part). He seems all right, I say to Barton, once he has disappeared from view. “Oh, he’s a lovely, lovely man,” she says, unhesitatingly.

When I reach the gatehouse, I turn back for one last look at the palace – and there she still stands, carefully photographing the swollen buds of a magnolia tree on her mobile phone. She is smiling – as unharried, in this moment, as any tourist.

• Reimagining Britain: Foundations for Hope is published by Bloomsbury (£16.99). To order a copy for £14.44 go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99