There is a moment in Steven Spielberg’s Empire of the Sun (1987) when an American plane strafes the Japanese camp where young Jim (Christian Bale) is interned. Then, in slow motion, the pilot sweeps past the burning buildings and turns his head and waves a sunny Hollywood hello to the enthralled watching boy. We’re not just inside Jim’s besotted response to American power here; the film really partakes in his vengeful rapture. My groan of disbelief, on first seeing it in the cinema, may have been audible outside in Leicester Square; in that instant, in the collision of the real and fantasy, I had seen the core of schmaltz within. Yet, now I feel that I was wrong: it is precisely in that collision that Spielberg’s greatness lives.

The apparently unexceptional, business-as-usual blockbuster-maker now seems deeply odd, intriguingly askew. He manipulates us to make us believe. In Indiana Jones and the Last Crusade (1989), Indy asks Denholm Elliott’s character, “Do you believe?”, and this question resonates through all Spielberg’s sci-fi and fantasy films, turning as they do on our ability to awaken our faith and move beyond the literal “facts” to the truth.

Like those of David Lynch, his exact contemporary, Spielberg’s films veer wildly between kitsch and cruelty. They are mad journeys, open to anything that is possible. He is a barometer of American concerns, bearing the pressure of the nation’s fears and fantasies. In Spielberg’s movies, especially those sci-fi and fantasy films where history is not there to trammel his imagination or expose his wayward touch, we find the pure madness of modern America: the delusions, the illusions, the confounded dream.

Despite Jaws and Jurassic Park, Duel's truck continues to be Spielberg’s most convincing monster

For 45 years, Spielberg has wholeheartedly committed himself to the marvellous. His sci-fi films, his quests and dystopias, find their home properly on the big screen. There are few other film-makers whose work loses as much by being watched on a TV or computer. His is a cinema where size itself rises to poetry. He endows us with the grand, entangling us in mechanisms of clockwork threat; he’s a master of making us feel hunted. He lets us long for hiding places, or opens before us light and vastness, as the eyes lift to watch the skies. He is not anyone’s idea of an intellectual, but remains instead a great intuitive film-maker, plot and spectacle sparking perceptions. Watching E.T. the Extra-Terrestrial (1982), we don’t think ET’s thoughts: like his boy hero Elliott, we feel his feelings. Spielberg moves us, and increasingly it’s the suspect power of the work of art that has become his real subject.

Dennis Weaver in Duel. Photograph: Allstar/Universal TV

It’s all there already in his debut film, Duel (1971). This TV movie begins inside an automobile’s point of view, Dennis Weaver’s Everyman identified with the machine that contains him. A truck pursues Weaver, intent on crushing him. Spielberg’s great theme in much of his work is simply survival. Despite Jaws (1975) and Jurassic Park (1991), that truck continues to be Spielberg’s most convincing monster, a mechanical beast holding an unseen man within. In a sense, Spielberg has been a humanist working in a “post-human” world ever since, his embrace of technology and special effects disrupting and threatening the fragile human identities his films would hope to extol.

Duel restages the contest between David and Goliath, a battle that seems the soul of many of his films. He is an aficionado of the unequal struggle, eager to affirm again, and again, the little guy’s triumph. Yet we can feel sometimes that, in aesthetic terms, he prefers the Giant to kill the Jack. With their efficient brutality and power, the tripods in War of the Worlds (2005), the velociraptors, the great white shark, starkly impress us and him. Spielberg’s films are always apt to find a home for a good enemy. In Jurassic Park, Spielberg’s aim was to make the dinosaurs as “real” as possible, to ground the “as if” of fantasy in scientific truth. He wants us to feel that what’s on screen – the grazing flocks of leviathans, the hunts, the attacks – are truly happening. The intention proves impossible, ripped to shreds by the velociraptors, who descend into the role of screen heavies, no longer beasts but conscious movie monsters. The film sets out to instruct us that nature can overwhelm us, but instead overwhelms us with art.

Invasion scene from War of the Worlds. Photograph: Allstar/Paramount

Critics routinely suppose that a sense of wonder informs science fiction, and yet how rarely is any sense of joy present there. In his sci-fi films, Spielberg proves remarkable in his desire to reach for joy, a celebratory sublime. Nowhere is that yearning more apparent than in his two child-like masterpieces of alien contact, Close Encounters of the Third Kind (1977) and E.T. Both offer an experience suffused with spilt religion. Spielberg swept me up in an epiphany when I saw the first film, aged 12; if I could have walked into the screen and absconded in the Mother Ship, I would gladly have done so. Now it seems to me a gloriously weird film, de-centred, caught between the generic mechanisms of horror, the downbeat exposure of domestic estrangements, and the rapt pieties of The Robe, Richard Burton’s 1953 sword and sandal epic about a Roman’s crisis of conscience when ordered to crucify Jesus. In Close Encounters it’s precisely its invitation to believe that turns out to be its subject. In E.T. the same impulse works at an even more intense level of manipulation; we’re caught in the maddening moment from Peter Pan where the audience must clap to show their belief in fairies and so resurrect expiring Tinkerbell. In E.T., it’s a hard heart or a clear head that resists the call.

This could look like we’re being practised upon, dupes of unconscionable skill. Spielberg can seem close to Hitchcock in this talent to influence us. In Hitchcock’s case, one senses the film-maker looking on and looking down on you. Yet, in Spielberg’s best films, we know somehow that he is making the film so he may sit in the audience with us. If he is a manipulator, he’s an honest one, mesmerising himself with his own rhetoric, tearing up over the dying wretch, ready to join in the booing of the villain. We feel his own art moves him, and that he’s almost ready to forget, with us, that it’s art at all.

It’s no surprise that in the fabulous Indiana Jones films, Spielberg should revive the turn-of-the century boy’s stories, King’s Solomon’s Mines and The Lost World, even down to the imperialist politics. He shares with those books the consoling notion that life offers us adventures. He reclaims an ideal of American heroism, compromised by Vietnam and the Watergate years. The Reagan-era optimism of these films has its political resonances, and yet remains true optimism. In the 21st century, Spielberg followed the US again in his return to darkness. The sentimental would flip into the despair that is its inescapable other side.

Haley Joel Osemt in A.I. Artificial Intelligence. Photograph: Allstar/Cinetext/Warner Bros

In the science fiction films, the shift first manifests itself in A.I. Artificial Intelligence (2001), Spielberg’s direction of what might have been his friend Kubrick’s last film. It examines our relation to technology as well as our splintering relation to each other. In this eerie remake of Pinocchio, love itself is terrifying. An artificial boy, David, replaces a couple’s lost son; the mother activates a program in the robot child that triggers a deep, undying love in the boy for her. Yet she soon rejects him, another instance of a primal fear of abandonment throbbing through Spielberg’s work. In the 15 years since it came out, it has almost stopped feeling like sci-fi, and, with its robots, its climate-change ravaged world, its smart rooms, has become merely a drama about what our internet-embedded lives feel like now. I’ve watched A.I. and loathed it, and watched it and been amazed by it. If it is bad, then it’s bad in a peculiarly confident way. The film tries to tap into Kubrick’s remoteness, his essentially cold eye, but in the end becomes something impossibly visionary, a wildly Spielbergian creation. It can seem like a bizarre children’s film, and then a moment later like a horror movie, and then back again; it grants us “fact” and then “fairytale”, and then mixes the two. The juxtapositions and collisions in its structure, its crazy veering in tone, are at the centre of Spielberg’s aesthetic. The film afflicts us with scenes of terrible violence against the robot characters, but after all they’re “only destroying artificiality”, pouring acid on or eviscerating machines.

The actors in the film make uncanny doubles for the robots they play. The film actor is anyway more or less a robot, being mechanical, no longer organic, and – like the mummy-fixated boy at the movie’s heart – someone that we may or may not find it in ourselves to love. A.I. wishes to humanise the robot, as E.T. humanises the alien. This film – like many of Spielberg’s – is fascinated by what can’t be seen, only felt or intuited. The invisible element that makes us ourselves. The robot boy David’s quest is for something that cannot be found in the world or on a cinema screen: that is, the soul. This can only emerge from fantasy, from dreaming, a reverie caught in the permanence of film. In the shape of the fabricated boy, the movie presents to us an age-old paradox of art. After we’re dead, the artificial humanness we construct in robots, the traces of ourselves we leave in Facebook profiles or films will survive us, as this film survived Kubrick and will survive Spielberg, and even as this small article will survive me. When benevolent aliens rescue David from a millennia-long quest to become real, they turn themselves into cinema screens of his memories, projecting the film we’ve ourselves seen. It’s one of the most eccentric tributes ever to the wonderful, the harrowing survival of art. David ends the film as a “real boy” – real in so far as he too requires art’s feigning, he too may love an image.

Tom Cruise, Samantha Morton and Steven Spielberg on the set of Minority Report. Photograph: Twentieth Century Fox

Since then, scarier visitors have replaced A.I.’s benign aliens. Spielberg comes across as a perplexed optimist, an essentially sunny person on whom the pain of the world has forced itself. The sometimes strident insistence on purposeless suffering in his films of the last 20 years or so feels as though it springs from the fact that such suffering is a discovery to him. There’s a kind of thrown pride in his knowledge of grief and defeat. Tom Cruise has been the actor key to these forays into darkness, very good in the excellent Minority Report (2002) and very unappealing in the monumentally glum War of the Worlds (2005). In the latter film, all memories of Close Encounters are effaced as we watch Martians murder hordes of people, see cities destroyed, civilisation erased. As in most of his films made with cinematographer Janusz Kami´nski, light itself seems sick. The world unravels, but then at the end of the film, the aliens conveniently catch cold and die, and Cruise’s nuclear family turns out to be still happily intact. On a leaf-strewn Boston street, while countless millions have died, they have survived. The film expects us to feel good about this. Yet here the clash between sentimentality and violence is too strong, the impact of those deaths too unforgotten. It’s all that’s bad, all that’s individualistically American, about a film-maker that nonetheless is one of America’s justifications for itself, a sign of its confident ability to impose on the world its compelling dreams of itself.

• The Spielberg season is at the BFI, London SE1, throughout June and July. bfi.org.uk.