Before the arrival of Lou Reed, it could be said that rock'n'roll remained a part of the entertainment industry, still in the business of ingratiating itself with its target audience even as it tried equally hard to alienate their parents. Reed, who has died aged 71, made alienation the dominant mode and an article of faith for subsequent generations.

With his band, the Velvet Underground, and in his subsequent solo career, his hymns to transgressive behaviour shaped the approach of such near-contemporaries as David Bowie, Iggy Pop and Patti Smith before exercising a pervasive influence on such successors as Talking Heads, the Sex Pistols, Joy Division, REM, the Strokes, and almost certainly whichever indie band finds itself on the cover of the next issue of the New Musical Express.

After the affectless New York drawl heard on Walk on the Wild Side and Perfect Day, his fearlessness was the most distinctive quality that Reed brought to the business of making rock'n'roll. Unafraid to jolt his audience by presenting them with noise elements borrowed from the jazz and classical avant-garde or with images of himself as a representative of a sexually ambiguous underworld, eventually he would find that he had created an audience of outsiders who coalesced around his example.

For five years following the release in 1967 of The Velvet Underground and Nico, an album ostensibly "produced" by Andy Warhol and featuring the artist's peel-off banana on the cover, the music of Reed and his colleagues was listened to only by a devoted few. Those few, however, eventually emerged as the shapers of a new and powerful strain of popular music. Once the punk and new wave explosions had occurred on both sides of the Atlantic, it could be seen that Reed had changed the mood and the direction of pop in a way that might legitimately be compared with the impact of Elvis Presley, the Beatles or Bob Dylan.

Before Reed got to work, pop's idea of darkness was the romantic despair of Roy Orbison's Running Scared, the adolescent misogyny of the Rolling Stones' Under My Thumb or the acid put-down of Dylan's Positively 4th Street. Right from the beginning, the Velvet Underground's songs plunged into the real darkness of hard drugs, sexual degradation and psychosis, the mood of Venus in Furs and the even more startlingly explicit Heroin recalibrating rock'n'roll's sights.

The prevailing darkness was reflected in the way Reed and his fellow musicians rebooted the music's primal look: they wore black, black, and more black, a blatant rejection of the vivid palette of psychedelia and its love-and-peace attitudes. The look flourishes today among young groups seeking to evoke the illicit thrill of Warhol's Exploding Plastic Inevitable multimedia shows, in which the Velvet Underground played while the actress Edie Sedgwick gyrated and the poet Gerard Malanga cracked a bullwhip, all against a backdrop of Warhol's movies.

Reed's more profound ambition was to use rock's immediacy as a vehicle for a certain kind of literary approach. "Let's take Crime and Punishment and turn it into a rock'n'roll song," he said. As well as Dostoevsky, his heroes included Raymond Chandler, Hubert Selby Jr, William Burroughs and Edgar Allan Poe. His later career included collaborations with artists from various fields, including theatre pieces with Robert Wilson, films with Wim Wenders and works with the composer Laurie Anderson, who was his companion for the last 20 years.

Reed was born to a middle-class Jewish family in Brooklyn, the son of a tax accountant and a former beauty queen. Growing up from the age of 11 on Long Island, he was sent to a psychiatric hospital for electroconvulsive therapy, intended to cure him of a general hostility to his parents and what they believed to be homosexual instincts. Attracted to the sounds of rock'n'roll, doo-wop vocal groups and avant-garde jazz, he had taught himself to play the guitar and made his first record with a doo-wop group, the Jades, at the age of 14.

In 1960 he enrolled at Syracuse University, where he studied film directing, creative writing and journalism, and fell under the influence of the poet and short-story writer Delmore Schwartz. On graduating in 1964, and intent on a career in music, he went to work as a songwriter for Pickwick Records, whose principal business was churning out budget-priced discs for sale in supermarkets. When one of his compositions, titled The Ostrich and intended to promote a made-up dance craze, threatened to become a hit, a band was quickly assembled to help promote it, and it was there that Reed met John Cale, who would become his most important musical collaborator.

The Welsh-born Cale, whose instruments were the viola and the bass guitar, had studied composition with Aaron Copland and Iannis Xenakis and performed with John Cage and La Monte Young's Dream Syndicate. His discovery that Reed played The Ostrich on a guitar with all its strings tuned to the same note encouraged him to pursue their collaboration. Rehearsing in an apartment on the Lower East Side with Sterling Morrison, a friend of Reed's from Syracuse, on rhythm guitar and the sculptor Walter De Maria on drums, they went through a series of names: the Primitives, the Warlocks, the Falling Spikes. Once De Maria had departed and his replacement, Angus MacLise, had been succeeded by Maureen Tucker, they settled on calling themselves the Velvet Underground, after a pulp paperback devoted to aberrant sexual behaviour.

Returning from a visit to London with copies of records by the Who and the Small Faces, Cale advocated a shift from their conventional Dylan-influenced folk-rock style to something more adventurous and confrontational. With Reed adding the abstract noise elements of free-form jazz, Cale alternating between hammering out piano ostinati and using his viola to produce grating drones, and Tucker thumping her tom-toms from a standing position, the group began to produce the morose, sludgy ensemble sound that was like nothing else in popular music, as well as rendering Reed's vocals tantalisingly indistinct.

Both rudimentary and highly sophisticated, their music appealed to Warhol, who was looking for a rock group to add to his menagerie of soi-disant superstars and was taken to hear them at a Greenwich Village coffee house. They soon became familiar figures at the Factory, where they were introduced to the German actress and model Nico (born Christa Päffgen), who had recently made a record in London. At Warhol's suggestion she became their fifth member, adding an element of glamour and mystery as she intoned Reed's songs, notably Femme Fatale, I'll Be Your Mirror and All Tomorrow's Parties, in her deep, virtually expressionless voice.

It is hard to exaggerate the degree to which they were swimming against the tide of popular fashion, or the extent of the mingled scorn and indifference with which their debut album was received. Supposedly, San Francisco and Los Angeles were the crucibles of the new music emerging to accompany the celebrations of the Summer of Love, while New York had been temporarily relegated to the status of a black hole from which no cultural light escaped.

Nico left the band after the first album, but the remaining Velvets carried on as a quartet to record a second album, one that would be just as striking and, ultimately, as influential as its predecessor. The centrepiece of White Light/White Heat (1968) was the majestic 17-minute Sister Ray, probably the closest any of the studio records came to reproducing the controlled anarchy of their live performances.

Tensions between Reed and Cale led to the Welshman's enforced departure before their third album, The Velvet Underground, in which Reed adopted a gentler overall tone – except for a song called Murder Mystery, in which he experimented with parallel narratives delivered simultaneously from either side of the stereo sound-picture. A fourth album, Loaded, contained two songs that would become classics, Sweet Jane and Rock & Roll, but by the time of its release in 1970 Reed had left the group, disillusioned by its failure to make any sort of commercial impact.

He returned to his parents' home, working as a clerk at his father's firm for more than a year before re-emerging at the end of 1971 to begin a series of albums under a new contract with RCA Records. The first was not a success but the second, Transformer, was the one that finally brought him attention: produced by his new friend Bowie, it contained a hit single, the seductive Walk on the Wild Side, in which he portrayed a series of picaresque Warhol characters. Its packaging aligned him with the new mood of glamour and decadence to whose adherents, thanks largely to Bowie's enthusastic patronage, he became an object of veneration.

He retained his status as glam-rock gave way to punk, but his recording career became characterised by its inconsistency: sometimes infuriatingly sub-standard, occasionally brilliant. The moments of brilliance were usually those most likely to lose him his following, such as a song-cycle of epic morbidity titled Berlin (1973), and Metal Machine Music (1975), a double album of guitar noise which could have been explicitly devised to ensure the termination of his recording contract.

Street Hassle (1978), The Bells (1979) and The Blue Mask (1982) all contained pieces in which he stretched himself in interesting directions, but with New York (1989) and Magic and Loss (1992) he hit his full stride once more, the songs Dirty Blvd and What's Good proving his continuing ability to invest the two-chord rock'n'roll song with an irresistible freshness. At the other extreme came The Raven (2003), a series of pieces inspired by Poe, demonstrating the extent of his extraordinary range.

He collaborated with Cale once again in 1990 on Songs for Drella, a threnody for Warhol, and in 1993 the four original Velvets re-formed for a brief series of European concerts that ended, appropriately enough, in acrimony. Reed, Cale and Tucker played together once more, while being inducted into the Rock and Roll Hall of Fame in 1996, the year after Morrison's death.

Reed's final years were full of activity, including several theatre pieces with Wilson, the publication of his lyrics and photographs in book form, an adaptation of Wedekind's Lulu plays with Metallica, concert performances and a film (directed by Julian Schnabel) of Berlin, and appearances on records by Gorillaz and the Killers and on tribute albums to Buddy Holly and to his great friend the songwriter Doc Pomus. Ornette Coleman, Jimmy Scott, Antony Hegarty and the Blind Boys of Alabama were guests on his records, and he took part in the most unlikely of projects, a live orchestral rendering of Metal Machine Music. He never renounced the habit of terrifying journalists who failed to ask questions that engaged his attention.

What Reed possessed was authenticity. When he sang about going up to Harlem to buy heroin or about transgressive sex, his writing came from first-hand experience. Drugs and drink were a constant companion for most of his adult life, and between his first and second marriages — to Bettye Kronstadt in 1973 and Sylvia Morales in 1980 — he lived openly for several years with a transvestite named Rachel. And he could turn it all into an unflinching poetry.

He gave up drink and drugs in his 50s, opting for T'ai Chi and studies with a Tibetan lama. He underwent a liver transplant earlier this year and died in the Long Island home he shared with Anderson, whom he married in 2008 and who survives him.

• Lou (Lewis Allen) Reed, singer, songwriter and guitarist, born 2 March 1942; died 27 October 2013