The Vatican has just lent its greatest counter-reformation altarpiece, Caravaggio's Entombment, to the artist's anniversary exhibition in Rome, but it would be surprising if Pope Benedict XVI made the short journey to the Quirinal Palace to enjoy the show. It's not so much that the artist was a homicidal genius denounced by near-contemporaries as an anti-Christ who had come to destroy painting – Caravaggio received a papal pardon for his capital crime, and most art crimes are a matter of aesthetics, which are debateable. It is more that he first made his name in Rome by creating the sexiest boy pictures ever made. Several were commissioned by a cardinal who, like the artist, may have been a pederast, while others were acquired by another cardinal (the pope's nephew) who was probably homosexual.

It is 400 years since the squalid, lonely death of Caravaggio, and the one-time fugitive from justice and serial malefactor is now revered as a supreme cultural hero. As with Van Gogh, Pollock and Bacon, Caravaggio's fame is fuelled by the seeming symmetry between his tempestuous life and his raw, revolutionary art. He is routinely called the first modern artist. Fifty years ago an art historian said anyone would be forgiven for thinking that Caravaggio's contribution to civilisation lay somewhere between Aristotle and Lenin; now we could throw in Nietzsche and the Marquis de Sade.

Michelangelo Merisi da Caravaggio (1571-1610) was born into a prosperous family in the small town of Caravaggio, near Milan. His father Fermo worked as a builder and architect. Caravaggio was apprenticed to a former pupil of Titian in Milan, but left for Rome in 1592. A pilgrimage to the Eternal City to study modern and ancient masters was de rigueur for an aspiring artist, but Caravaggio may have left after a brush with the law. In Rome he was a dogsbody, moving between studios, barely eking out a living, with some of his pictures sold on the street by hawkers. He specialised in portraiture and a novel north European genre: still life.

The prevailing period style is now known as mannerism – busy, diffuse compositions featuring idealised figures in complex pseudo-Michelangelesque poses; saccharine colour schemes; plunging perspectives. But by the end of the century Roman art was becoming marginally more naturalistic, with a greater emphasis on clarity, cogency, human drama and emotion. Caravaggio both created and exploited this taste for naturalism. His breakthrough came with freshly coloured and luridly lit genre paintings with one or more half-length figures of imposing scale. They were modelled from life using friends, lovers, street-people, gypsies, prostitutes – and his own swarthy self. These low-lifers co-existed with dazzling still lives. Caravaggio's iconoclastic credo was that it is as difficult to paint a good picture of flowers as it is to paint figures. He caught the eye of the supremely cultured Cardinal Francesco Maria Del Monte – a lover of music and alchemy, and friend of Galileo – who made him a salaried retainer.

A stunning sequence of single, seated good-time boys interacting with prominent still-life elements are a double-edged homage to his great namesake Michelangelo Buonarrotti. These sybarites could almost be Michelangelo's Sistine ceiling ignudi brought down to earth, fashioned from high-blush flesh and blessed with dirty fingernails. Whereas the ignudi hold or marshal bunches of acorns (papal symbols) and make movements that are sublimely indeterminate, Caravaggio's own semi-clad figures knowingly proffer or pose with fruit and drink, implicating the viewer. Two represent Bacchus, the wine god.

Boy Bitten By a Lizard (c1593-94) is gently moralising, mildly titillating. A curly-haired, plump-lipped boy with a white rose in his hair reaches forward to grab some cherries and gets nipped by a lurking lizard on the middle finger of his right hand: the would-be biter of (forbidden?) fruit gets bitten. His white, toga-style shirt peels off from his right shoulder, which has jerked upwards into his cheek, seemingly dislocated by the shock. Raking light from the upper left slaps his shoulder and face. What sweet sadism! The lizard bite (neither poisonous nor dangerous) serves to intensify rather than jeopardise the boy's beauty, by opening up his face and torso and by exposing a charming vulnerability that elicits amused and erotic compassion rather than revulsion. His flailing fingers pluck the air like those of the sultry boy musician in The Lute-Player (1595-96) painted for Del Monte. Both seductive and scary, these haunting creatures are ancestors of what the gay writer and film director Pier Paolo Pasolini called ragazzi di vita – boys of life.

Caravaggio's transformation from a stylish genre painter whose works were prized by cultivated connoisseurs to a great and popular religious dramatist is as sudden as it is unexpected (at a much lower artistic level, it's as if Jeff Koons morphed into Francis Bacon). Caravaggio's first church contract, for three pictures on the life of St Matthew in the Contarelli Chapel, was signed in July 1599, thanks to Del Monte's string-pulling. It was unveiled a year later to huge acclaim, and Caravaggio never again lacked for prestigious church commissions, even after his flight from Rome in 1606 having killed a man in a sword-fight. He painted only two more genre pictures. Wherever he went, local artists aped his style.

In The Calling of St Matthew, the main action, as in all Caravaggio's pictures, is uncomfortably close. It takes place at the front of a shallow stage blocked off behind by a dark featureless expanse (in this case a wall). Caravaggio still uses fashionably dressed figures who could have stepped out of one of his genre scenes, yet their pretensions are stripped bare by men with a different dress code, and armed only with light.

Matthew the foppish tax-collector sits at a table counting his money or gambling with rakish male friends and colleagues. At stage left, Christ and Peter have shuffled in, shoeless and dressed in drab togas. They're an odd couple. Peter stands weak-kneed right in front of Christ, practically pushing him to the wall. Of Christ, we glimpse only the noble head, a bare foot and raised right hand, which points in the direction of the fashionistas. This hand is strangely floppy (the limpest of the three pointing hands in the picture), and is an allusion to the languid hand of Michelangelo's Adam. It is surely meant to be like the hand of a spear thrower relaxed after release, for a blade of light passes diagonally overhead, angling towards its target of Matthew's face. The ambushed victim points quizzically to himself, as if to say: "Who, me? Why me?"

Caravaggio is justly regarded as the master of chiaroscuro, creating mood, emphasis and relief by contrasts of light and dark. This was part of his Milanese heritage, for Leonardo da Vinci had lived and worked there a century earlier: Caravaggio would have studied his Madonna of the Rocks altarpiece, now in the National Gallery. But whereas Leonardo tried to soften the transitions, Caravaggio's contrasts of colour and light are aggressively hard-edged, almost heraldic in their geometrical clarity and frontality, and played out over dark, skyless backgrounds. Rubens, Rembrandt and Velázquez were all profoundly influenced, just as, more recently, were countless photographers and film-directors.

The bold immediacy of Caravaggio's Roman pictures owes as much to his method of painting directly from the live model at speed without making preparatory drawings. He incised the main outlines into his darkly painted grounds. For his multi-figure compositions he may have created tableaux vivants in a blacked-out studio, using lamplight (as re-enacted in Derek Jarman's film Caravaggio and Simon Schama's Power of Art). The National Gallery website informs us that Caravaggio's technique "was as spontaneous as his temper". Yet little is left to chance or rushes of blood. These are not snapshots. Most gestures and poses are slow and ritualistic rather than instinctive or off-the-cuff.

Unlike the Venetians, who had pioneered painting from the live model, Caravaggio rarely selected perfect specimens. This perceived lack of idealisation was the most controversial aspect of his altarpieces: for some, it exemplified the counter-reformation insistence that art should give the unvarnished truth in as forceful a manner as possible; but for artists such as Poussin, Caravaggio had gone too far and tried to destroy art by flouting decorum. Several altarpieces were rejected or criticised by the authorities, but the rejects were instantly snapped up by private collectors. Rubens persuaded his employer, the Duke of Mantua, to buy The Death of the Virgin (1601-03), rejected because the Virgin looked too dead and was reputed to have been modelled on a drowned prostitute. Yet Rome's poor seem to have appreciated seeing themselves centre stage. When the Madonna of Loreto (c1603-06) was unveiled, with its bedraggled pilgrims (one with filthy feet) praying before the Virgin and child, it was noisily applauded.

Some of Caravaggio's proletarian imagery, and especially the flaunted dirty feet and furrowed brows, do come close to caricature. The rejected altarpiece in the Contarelli Chapel of St Matthew and the Angel (destroyed in Berlin in 1945) is a brilliantly silly example. Matthew is a balding, burly, illiterate peasant with Artexed brow and dirty feet, one of which dangled over the altar; the casual intimacy of his relationship with the angel who guides his writing hand doesn't help matters. It was immediately bought by a wealthy friend of Cardinal Del Monte's, who paid for Caravaggio to paint a more decorous replacement that resembled the suave and educated tax-collector in the Calling.

The "proletarianism" and sardonic humour of so much of Caravaggio's Roman work must owe something to the Accademia della Val di Blenio, founded in Milan in 1560. This was an anti-establishment wining and dining club dedicated to promoting a fabricated dialect that was claimed to be the ancient language of Swiss wine porters working in Lombardy. Members wrote burlesque literature in the rough-and-ready dialect, and in 1568 Giovan Paolo Lomazzo, a painter friend of Caravaggio's first teacher, was appointed "Abbot" of the society. In St Matthew and the Angel, and in many other religious works, Caravaggio has resurrected, as it were, the ancient dialect of biblical wine porters.

During the 18th and 19th centuries Caravaggio was everyone's whipping boy – not just an iconoclast, but a heretic and even an atheist. John Ruskin saw nothing but "horror and ugliness, and filthiness of sin". Of course, these vices have come to seem like exciting virtues to modern audiences, with one scholar praising Caravaggio's "derisive irony" as though he were the father of Duchamp, Dada and the spaghetti western.

When confronted by the vast tarpaulins of darkness that envelop Caravaggio's late works, painted as he scurried around Naples, Malta and Sicily, one does wonder how far he believed in personal salvation. The spotlighting of his great New Testament figures is forensic: it freezes and targets rather than transfigures (he would have loved laser sighting mechanisms). The professionalism of the executioners – and the lack of ministering angels – is spine-chilling. David with the Head of Goliath was painted soon after Caravaggio had been disfigured by assassins in Naples, and not long before his death from malaria during his desperate rush to Rome for a papal pardon. The great shepherd boy holds up the bleeding head as nonchalantly as Caravaggio's own Bacchus held out a glass of red wine (Goliath's head may be a self-portrait). This David is not in the least bit perturbed or moved, as some critics have claimed: his decidedly phallic sword blade presses against his own groin.

Caravaggio's art can be best understood as representing a "Machiavellian moment". He lived in an age when the term Machiavellian had become common currency, often as a term of abuse. But it was still granted huge explanatory force: the St Bartholomew's day massacre of Protestants in Paris in 1572 was attributed more to Machiavellian political manoeuvrings than to religious ideology. The metronomic intensity – and, to some extent, monotony – of Caravaggio's work stems from that fact that he is as interested in the natural and political history of religion as the spiritual history. His David is a narcissist, opportunist and realist before he is a prophet and author of the penitential Psalms.