If sonata form is about the process of reaching a destination, variation form is more about taking pleasure in the journey itself. While sonata form explores material through argument – contrasting ideas thrashed out and eventually synthesised – variation form is more about one unopposed idea being explored, mined, twisted and turned in a monologue of elaboration.

When instruments began to sing alone without voices and when that music began to be written down, variation was the commonest form by which short musical ideas were extended. A catchy tune could be pinned down and strung along by repetition and embellishment. By the classical era, sonata form had been invented and eventually became ubiquitous, but writing sets of variations remained a rich source of expression for composers.

Variation sets are usually lighter in intellectual substance than a composers's greatest works, decoration more than architecture, but occasionally a composer will reach beyond the superficiality of the form to find the greatest heights of inspiration and a Goldberg or a Diabelli is born. It is as if the greatest minds, when released from the necessity to think through a complex problem, can show something of the blinding, unwitting intricacy of their subconscious as they deal with something simple and straightforward. It's often called genius.

Paganini's Caprice No 24 has been used by so many composers as a theme to be varied for a number of reasons. It is written in the clean, white-note key of A minor – a pure starting point – and exposes the bare bones of its tonal simplicity: a textbook example of classical harmony, it shifts rhythmically from tonic to dominant and back, like a tennis ball over a net. In the second half of the theme, we hear a circle of fifths, the harmonic progression without which much of Bach and most of Vivaldi – not to mention countless popular songs, such as "Windmills of Your Mind" and Jerome Kern's "All the Things You Are" – would never have been written. As well as giving the music a strangely poignant effect, it allows it to shift instantly and infallibly to a more expressive vein. Finally, Paganini's theme cuts a dashing rhythmic shape as its melody repeatedly turns on itself with a swagger and clip of the heel. All points that make this musical material eminently suitable for variation.

Liszt, in 1838, while Paganini was still alive, merely took the Caprice and transcribed it for piano, but soon afterwards many others (notably Brahms but curiously not Schumann, despite publishing some Paganini transcriptions) used the theme as a springboard for their own musical ideas. And thus it became iconic, a Madonna and Child that all those who painted had to paint. By the early 20th century, composers and pianists such as Ignaz Friedman and Mark Hambourg (not to mention the virtuoso violinists Eugène Ysaÿe and Nathan Milstein) had taken the theme and covered it with their fingerprints.

When, in 1934, Rachmaninov began to write his Rhapsody on a Theme of Paganini, his composing career was in decline, at least in the eyes of audiences and critics. Since leaving Russia for the west, he had written little and his Fourth Piano Concerto (1926) had flopped. His (over)ripe Romanticism was out of fashion; modernism held all the cards. But then, in the postwar, palette-cleansing neoclassicism of composers such as Stravinsky and Prokofiev, perhaps Rachmaninov saw new possibilities. The Rhapsody on a Theme of Paganini was his take on this aesthetic – or at least, his way of using a classical theme as a launching pad.

Many who have no affection for Rachmaninov still admire this piece. It is a formal miracle: a set of 24 variations held together as if it were a mini piano concerto, but with the carefree unfolding of a rhapsody. The overall structure is completely satisfying, and its composite parts – the small decorative, inventive details of its variations – are scintillating.

In addition to Paganini's theme, there is the constantly recurring spectre of the Dies Irae, the plainchant melody from the requiem mass, famously brought into the concert hall by Berlioz 100 years earlier in his Symphonie fantastique and a much-used theme in its own right. In Rachmaninov's work, this musical symbol of doom and judgment is like cement in the mosaic, giving stability, strength and a certain seriousness to the kaleidoscopic variations on Paganini's theme.

Lutosławski's Paganini Variations returned to the work's origins. The Polish composer cooks the same meal as Paganini but adds a pinch of spice to each course, transforming the plain harmonies into piquant, exotic flavours. It was originally written in 1941 for two pianos and presents a dizzy dialogue of antiphony as snapping rhythms and pianistic glitter are tossed from one keyboard to the other in an exchange of witticisms. An orchestra-and-piano version, composed in 1978, saw the dialogue less clearly defined, but the palette of different instruments allowed Lutosławski to dazzle with unexpected colours and varied textures.

Variation is at the heart of all music, and even the withholding of variation in minimalism's strictest repetitions is a pun: the listener's perception changes when the music doesn't. And there is a broader human theme here, too, because every breath, every heartbeat for us is the same, but every thought and word and deed that those functions allow unfolds in constant variation. The French put it well: Plus ça change, plus c'est la même chose.

Stephen Hough performs Rachmaninov's Rhapsody on a Theme of Paganini and Lutosławski's Variations on a Theme by Paganini on the opening night of the Proms on 12 July. bbc.co.uk/proms. Follow Stephen on Twitter at @houghhough

Stephen's Hough's new ipad app, exploring Lizst's B Minor Sonata, will be available from 12 July. Preview it exclusively here