The rapture inspired by Skyfall in critics and public alike might have surprised Bond fans of the past. For the franchise's 23rd instalment lacks what some would have considered its quintessential ingredient.

What used to distinguish 007 from previous thriller heroes was his unique brand of ironic detachment. Ian Fleming's books demanded to be taken straight. The earlier films mocked their source material's vanity, as well as the thriller genre, love, death and Her Majesty's secret service. Their studied cheesiness mocked the mockery itself.

In Skyfall, Daniel Craig's Bond delivers a scattering of old-style quips, but the chronic flippancy from which they used to spring has disappeared. Indeed, the film's lack of larkiness is the point of one of the cracks. Ben Whishaw's Q, favouring practicality over hilarity, offers Bond only a gun and a radio tracker. When this produces a raised eyebrow, he says: "Were you expecting an exploding pen? We don't really go in for that any more." Thus frugally equipped, our hero confronts a world pervaded by guilt, doubt, grief and foreboding rather than the joshing sadism of his previous outings.

The asperity of that world is no novelty for Craig's 007. In Casino Royale, he suffered the humiliation of being tortured in the nude. Even more startlingly, he declared himself unambiguously in love. Quantum of Solace provided him with the psychological driver for his behaviour that had previously been considered unnecessary.

Still, James Bond is not the only screen hero to have sobered up. When his mirthfulness was at its height, it infected his big-screen beefcake peers. In the 1990 version of Total Recall, Arnold Schwarzenegger tries to outdo Bond in homicidal gibes. Impaling an enemy on a drill, he remarks: "Screw you!" When his wife tells him he can't hurt her because they are married, he shoots her in the forehead and says: "Consider that a divorce." This summer's reworking of the story, on the other hand, was glumly earnest, offering social and political allusions in place of flippancy.

The cheery Batman of 1966 has become the grim and agonised Dark Knight. Prometheus aspired to a portentousness of which Alien felt no need. The teen-flick turned sombre in The Hunger Games, while The Amazing Spider-Man spent so much time grappling with existential angst that he had little left for derring-do. Inception presented more of a mental puzzle than a white-knuckle ride. Even Harry Potter felt obliged to exit amid such unalloyed grimness that there were fears he might scare the children.

Cinema still plays host to gross-out, farce and facetiousness; yet it is darkness, deliberation and doom that are doing some of the best business.

The medium is taking itself more seriously, as well as the world. Hitchcock took a famously light-hearted view of his craft; nonetheless, Fox's forthcoming biopic is no mere cocktail of gossipy anecdote. It depicts its subject as struggling against Hollywood taboos to pursue his artistic goals. The real-life Hitch was apt to shrug "It's only a movie." Yet Anthony Hopkins intones on his behalf: "All of us harbour dark recesses of violence and horror. I'm just a man hiding in the corner with a camera, watching."

The small screen, too, has taken on a heavier tone. Lightweight crime favourites such as Midsomer Murders and Agatha Christie's Poirot have been upstaged by denser fare from Scandinavia. A flood of demanding drama from US subscription channels has upped the medium's smarts. History and science have elbowed their way to the front of the schedules. Art galleries and museums attract record crowds. Uncompromising lectures on recondite topics have made TED a surprise internet hit. Some sports writers have taken a heuristic turn. Weighty public debates sell out, while people queue all night in the rain for tickets to literary festivals.

The Institute of Ideas has had to move its annual Battle of Ideas festival to bigger premises to accommodate growing demand, with more than half of those attending in their teens and early 20s. "There's a palpable frustration with the assumption that everyone who's under the age of 25 has got the attention-span of a gnat and isn't interested in events and ideas," according to Claire Fox, the institute's director.

Over the past few years, the "knowledge networking business" Editorial Intelligence has emerged as the cerebral equivalent of a physical fitness club. It hosts discussions on the issues of the day at both member-only salons and free events. "I think that people are desperately seeking information," says the organisation's founder, Julia Hobsbawm. "Information is the new currency. The more you have of it the wealthier you are."

Phenomena such as these get less attention than complaints about low educational attainment, the preoccupation with celebrity or the supposedly mind-rotting effects of social networking. Yet alongside apparently relentless dumbing down, a new hankering for seriousness seems also to be emerging.

It is tempting to thank the economic cataclysm of 2008. A developed world that had been partying mindlessly on money it didn't have was brought down to earth with a bump. People who might have assumed they had no need to think now realised that they did. Thus, when the crisis hit, "subprime" overtook "Kate Moss" on Google Trends, even as the supermodel's love life was dominating tabloid discourse. Yet this explanation is not quite sufficient.

When irony ruled: Roger Moore as Bond in 1974's Man With the Golden Gun. Photograph: SNAP/Rex Features

For the sobering up predates the crash. Bond was refashioned in 2005. That was also the year in which Christopher Nolan's glum Batman put in his first appearance, and The Thick of It arrived to show that screen entertainment could get serious without becoming humourless. However, in his 2005 budget speech, then-chancellor Gordon Brown was still boasting that the economy had grown for 50 consecutive quarters and was forecast to carry on doing so. The darkening mood must therefore have deeper roots.

Mark Batey, chief executive of the Film Distributors' Association, believes that in the days of Sean Connery's Bond, audiences still trusted in authority, whereas people generally seem less deferential now. "Unarguably Chris Nolan was ahead of the game on this with films like Inception and the original Dark Knight."

Claire Fox has other thoughts, and traces the current appetite for reflection back, with surprising specificity, to the fall of the Berlin Wall in 1989. The idea that the ideological conflict between capitalism and communism had reached an end led to a greater pragmatism, she argues, which in turn spread pessimism about our ability to reshape our world: "Once you look round and think, 'Is this it?', it's a bit dispiriting."

Meanwhile, the media's preoccupation with trivia began to irritate. All of these things came together to trigger a reaction, says Fox: "I think there's a sense in which people start to think that this isn't satisfactory: it's too shallow."

Beneath humanity's mood swings, a self-correcting pattern can be detected. The laughing cavaliers beget Cromwell's roundheads, who in turn beget the Restoration's libertines. Edwardian buoyancy morphs into Great War despair. This delivers the roaring 20s, which bring forth the despondent 30s. Frivolity, it can be conjectured, is intrinsically wearing and eventually boring: it produces a backlash of its own accord. By this reading, we should have been due for a period of pensiveness about now, even without the debacles that have beset us.

Yet if we are really wising up, it is happening this time in the face of a legacy of unusually robust inanity. During the second half of the last century, the intellectual infrastructure that had sustained the importance of being earnest was hollowed out. The slow withdrawing roar of religious conviction and elite guidance brought forth a tidal wave of indifference.

Relativism, postmodernism, solipsistic materialism and other anarchic creeds took on the force of dogma. Crises no less grave than our own were met with sneering, nihilism or facetious disregard. The smutty, smirking James Bonds of those days reflected the fervent vacuity of their era.

When the joke wore off, it happened quite suddenly. In 2004, Neal Purvis and Robert Wade were commissioned to write a successor to Die Another Day. "They felt the film prior had become too fantastical. It was getting ridiculous," says Martin Campbell, the director of Casino Royale. According to Campbell, there were fears that the resulting change of tone would antagonise audiences. Yet Casino Royale took nearly $600m at the box office, the highest figure in the franchise's history. Clearly, the newly astringent Bond had struck some kind of chord.

Skyfall looks poised to do even better, in our even gloomier climate. Yet although something in the air may have changed, the ground beneath it remains the same. Our new taste for gravity has not been matched by an overhaul of our intellectual outlook. So how serious is the new seriousness? Might it be just another frivolous fad, the latest amuse-gueule for irredeemably decadent palates?

Films may have become darker, but they are not necessarily more intelligent. Few show the clear moral purpose that once used to infuse screen heroics. The Dark Knight's anguish did not develop into a coherent worldview. Inception did not actually make sense.

In Skyfall, director Sam Mendes toys, like Nolan, with grand issues. These range from the sacrifice of individuals for the greater good and the emotional consequences of childhood trauma, to the ethics of WikiLeaks and age discrimination in the workplace. But he only toys. Judi Dench's M has a grim-sounding message for the political class about "the shadows"; but it's not clear what, if anything, it means.

In the real world too, our newfound seriousness has not cut very deep. The long-awaited crisis of capitalism has produced plenty of attitudinising, but profound discussion of how we might change our world has hardly been to the fore. There are those who see our apparent thirst for seriousness as no more than an illusion born of cultural fragmentation. Says Will Self: "There are so many potential cultural sources that all levels of brow can be happily accommodated, including those that deceive themselves that they're higher than they really are."

Even Claire Fox is unsure how much things have really changed. "I don't want to be complacent," she says. "These things can be fashionable. Often I go to book festivals and they just turn authors into celebrities."

We may believe we like to be stirred; perhaps we just want to be shaken.