Last autumn, Barack Obama hosted an evening of classical music at the White House. Beforehand, he said, "Now, if any of you in the audience are newcomers to classical music, and aren't sure when to applaud, don't be nervous. Apparently, President Kennedy had the same problem. He and Jackie held several classical music events here, and more than once he started applauding when he wasn't supposed to. So the social secretary worked out a system where she'd signal him through a crack in the door. Now, fortunately, I have Michelle to tell me when to applaud. The rest of you are on your own."

Obama was having fun at the expense of the No Applause Rule, which holds that one must refrain from clapping until all movements of a work have sounded. No aspect of our modern concert ritual causes more bewilderment. The problem is not that the Rule is so arcane that even a law professor turned commander-in-chief cannot master it. Rather, it's that the etiquette and the music sometimes work at cross-purposes. The noisy codas of the first movement of Beethoven's "Emperor" and the third movement of Tchaikovsky's Pathétique demand applause, even beg for it. The word "applause" comes from the instruction plaudite, which appears at the end of Roman comedies. Those climactic chords are the musical equivalent of plaudite: they almost mimic the action of putting one's hands together.

If the president ever clapped in the "wrong" place, he was intuitively following instructions in the score. This explains why newcomers exhibit anxiety on the subject; it even appears that fear of incorrect applause can inhibit people from attending concerts, although they may be merely inventing excuses. Children pose a particular problem. In literature handed out by music-education associations, the suppression of enthusiasm in children is a major concern. Programme booklets sometimes contain a list of rules, rendered in the style of God on Mount Sinai: "Thou shalt not applaud between movements of symphonies or other multisectional works listed on the programme." And one may only applaud: "Appropriate applause is the only acceptable audible response from the audience."

The underlying message of the protocol is, in essence: "Curb your enthusiasm. Don't get too excited." Should we be surprised that people aren't as excited about classical music as they used to be? This question of etiquette is only part of the complicated social dilemma in which classical music finds itself. But I do wonder about the long-term effect of the No Applause Rule, as I wonder about other oddities of concert life: the vaguely Edwardian costumes, the convention-centre lighting schemes, the aggressive affectlessness of many professional musicians.

Whether the format should change is by no means an easy question. I don't plan to offer prescriptions. Indeed, in my view, the chief limitation of the classical ritual is its prescriptive quality; it supposes that all great works of music are essentially the same, that they can be placed upon a pedestal of a certain shape. What I would like to see is a more flexible approach, so that the nature of the work dictates the nature of the presentation – and, by extension, the nature of the response.

Mozart played to the crowd

The classical concert of the 18th century was radically different from the rather staid and timid affair of today. Famous evidence comes from a letter that Mozart wrote to his father after the premiere of his "Paris" Symphony: "Right in the middle of the First Allegro came a Passage that I knew would please, and the entire audience was sent into raptures . . . and as I knew, when I wrote the passage, what good effect it would make, I brought it once more at the end of the movement — and sure enough there they were: the shouts of 'da capo'." This kind of behaviour seems in line with what you find in jazz clubs, where people applaud after each solo, as well as at the end of each number.

In the Romantic era, composers began to reject the idea of music as boisterous entertainment. Schumann, in the guise of his alter ego Florestan, wrote: "For years I have dreamed of organising concerts for the deaf and dumb, that you might learn from them how to behave yourselves at concerts, especially when they are beautiful. You should be turned to stone pagodas." Mendelssohn, in his "Scottish" Symphony, asked that the work be played without a break, to avoid "the usual lengthy interruptions".

Wagner played a pivotal, if inadvertent, role in the transformation of audience behaviour. At the premiere of Parsifal in 1882, he requested that there be no curtain calls after act two, so as not to "impinge on the impression". But the audience misunderstood these remarks to mean that they shouldn't applaud at all, and total silence greeted the final curtain. "Did the audience like it or not?" Wagner asked. Two weeks later, he slipped into his box to watch the flower maidens scene. When it was over, he called out "Bravo!" – and was hissed. Alarmingly, Wagnerians were taking Wagner more seriously than he took himself.

In the first decades of the 20th century, mid-symphonic applause was still routine. When Elgar's First Symphony had its first London performance, the composer was called out after the first movement. Around 1900, though, a group of German musicians and critics began promoting a code of silence, à la Bayreuth. Hermann Abendroth was among the pioneers: in Lübeck, where he led concerts from 1905 to 1911, he told his audience not to clap between movements. By the 1920s, several leading conductors were discouraging excess applause. At first, many listeners resisted, regarding this as a display of arrogance on the part of superstar maestros. Olin Downes, chief critic of the New York Times, campaigned against the Rule in the 30s and 40s. After describing how Koussevitzky had gestured disapprovingly toward his audience when they clapped after the third movement of the Pathétique, Downes exclaimed: "How anti-musical it is! Snobbism in excelsis!"

This may go too far. In many instances, the Rule seems in keeping with the music. I wouldn't want applause between movements of, say, Messiaen's Quartet for the End of Time. Elsewhere, though, it has a perverse effect. Emanuel Ax, not a showboating pianist, complains on his website: "I am always a little taken aback when I hear the first movement of a concerto which is supposed to be full of excitement, passion, and virtuoso display (like the Brahms or Beethoven Concertos), and then hear a rustling of clothing, punctuated by a few coughs; the sheer force of the music calls for a wild audience reaction." It is the sound of people suppressing their instincts.

Worse is the hushing of attempted applause. People who applaud in the "wrong" place are presumably not in the habit of attending concerts regularly. They may well be attending for the first time. Having been hissed at, they may never attend again. And shushing is itself noise. I often hear a "shhhh!" from another part of the hall without having heard whatever minor disturbance elicited it. In an ironic twist, these self-appointed prefects have made themselves more of a nuisance than those whom they are righteously reprimanding.

Tweet your enthusiasm

Perhaps it is unnatural to expect utter stillness in a public space. We may be imposing habits of home listening on the concert hall. Seated before our stereos, we've grown accustomed to brief bands of silence between movements. This may explain why resistance to the Rule subsided rather quickly. Increasingly, individuals gathered in one place to have solitary, inward experiences. Where listeners were once swept away by music, they now spoke of music sweeping over them, like an impressive weather system over which they had little control.

During the applause debates of the 1920s, the pianist and conductor Ossip Gabrilowitsch said, "It is a mistake to think you have done your part when you buy your tickets." There ought to be more give-and-take between performers and audience, he is saying. Passivity is too easily mistaken for boredom. Performers, for their part, overdo the detachment. American orchestral musicians appear to have taken classes in how to show no emotion whatsoever – with the occasional exception of a slight smirk during the composer's bow or a flicker of a smile during the soloist's encore. Music is an art of mind and body; dance rhythms animate many classics of the repertory. But in modern classical music, the body seems repressed.

I am both a lifelong classical-music lover and a member of a generation – the so-called Generation X – that, according to scary graphs recently published by the League of American Orchestras, has yet to show the midlife surge of interest in classical music that previous generations displayed. I went to college with extraordinarily smart people, who knew their art, literature and cinema. But few of them knew classical music. I bring such friends to concerts, and although they are pleased to be there, I often sense a slight disappointment. They admire the music, but the evening in some way falls short. And I ask myself whether the experience could be modified so that their admiration might turn to love.

There is no lack of proposals aimed at demystifying classical music: theatrical lighting, videos, explanatory messages on handheld devices, invitations to Twitter, and so on. I have my doubts about many of these, unless the music is by Messiaen, in which case tweeting seems apt. Yet, for me, the introduction of gadgetry destroys one very distinctive quality of the concert hall – its largely non-electronic nature. In a totally mediated society, where electronics saturate nearly every minute of our waking lives, surrendering to the natural properties of sound can have an almost spiritual dimension.

Perhaps concerts should be more old-fashioned – more local, communal. Institutions might work on strengthening the bond between performer and public: remarks beforehand, gatherings afterward, and, certainly, a relaxation of the Rule. I'm with Ax when he says, "I think that if there were no 'rules' about when to applaud, we in the audience would have the right response almost always." At the same time, the prevailing atmosphere is too humdrum, too perfunctory. We are not doing justice to the music's uncanny presence. There are too many opportunities for distraction. It's disheartening to see people burying their heads in programme booklets. Why not lower the lights and train the spotlight on the musicians?

People often ask whether classical music has become too serious. I sometimes wonder whether it is serious enough. Certainly, it has acquired a veneer of solemnity, but too often that veneer is a cover for business as usual. I dream of the concert hall becoming a more vital, unpredictable environment, in thrall to the wildly diverse personalities of composers and performers alike. The great paradox of modern musical life, whether in the classical or pop arena, is that we both worship our idols and, in a way, straitjacket them. We consign them to cruelly specific roles: a certain rock band is expected to loosen us up, a certain composer is expected to ennoble us. Ah, Mozart; yeah, rock and roll. But what if a rock band wants to make us think and a composer wants to make us dance? Music should be a place where our expectations are shattered.

 This is an edited version of a lecture given to the Royal Philharmonic Society in London on 8 March. The full text is available at royalphilharmonicsociety.org.uk