James Dillon has written some of the UK's most important music - and last night won another major award. Why is he better known abroad? Andrew Clements reports. Meanwhile, Tom Service names five other neglected composers

By his own admission, James Dillon likes to stay on the fringes of Britain's musical life. He's never seen at high-profile premieres, and is extremely reluctant to be photographed. He was once associated with composers such as Brian Ferneyhough and Michael Finnissy, in what became known as the "new complexity" movement, but Dillon's music has always been very much his own, showing only the most superficial connections to that of his contemporaries. Over 70% of performances of his music take place abroad (his own estimate), but he is unquestionably one of Britain's leading composers. Last night his Fourth String Quartet received this year's Royal Philharmonic Society award for chamber-scale composition.

Unprecedentedly, it is the third time in the last 10 years that one of Dillon's works has won the RPS prize. This is a measure not only of how his work is valued by his peers, but also of how its complex rhythms and teeming, densely detailed surfaces (which give him a superficial connection with a composer such as Ferneyhough) pack such a direct, expressive impact. Dillon's musical language may have the rigour of 20th-century modernism behind it, but it also carries the weight of the musical tradition, and seems fully aware of its debt to the past.

Born in Glasgow in 1950, Dillon was virtually self-taught musically, yet he is now one of that rare breed of composers who manages to support himself entirely through his commissions. "I live cheaply," he says. "I've never had money." In his teens, his first interest was pop music, or more specifically rhythm and blues, and he formed his first band at the age of 13. "It was really the Rolling Stones first, and on the back of their recording sleeves I discovered names like Muddy Waters, Lightnin' Hopkins and the Delta blues. It was also a time when lots of dance was coming through from the States - Tamla Motown and Stax, and so I quickly moved away from the Stones."

From the start, Dillon was playing guitar and writing his own songs - "I'm a total control freak, which meant I had to produce my own material" - and left school as soon as he had done his O-levels. He did a foundation year at Glasgow School of Art, but knew he really wanted to be a musician. "I didn't know how I would do it, and I remember odd experiences in which I imagined musical ideas that I didn't know how to place."

The way forward came by happy accident, when Dillon was living in a communal house in Cornwall. "Someone turned up with some LPs that he had swapped for dope, and among them was a version of The Rite of Spring and some Webern played by the Italian Quartet. I was completely perplexed by the Webern. I realised that either this music was completely crazy, or that I had to learn a lot more about music. For me, the biggest source of information was the back of record sleeves, just as it had been for rock'n'roll, and there I read about something called the Second Viennese School - I didn't even know what the First Viennese School was. I've always been rather curious, and I soon realised here was an open field which had a resonance for me, a place where things were not circumscribed."

Dillon moved to London as soon as he could. "I thought I could get lost there, just do what I wanted, which was to work, and get away from the scene I was in, which was dominated by pop music and drugs." He was lucky, too, in arriving in the capital at the same time Pierre Boulez was starting his Roundhouse concerts with the BBC Symphony Orchestra, programming brand new works alongside the modernist classics. For Dillon, it was a perfect way in. "The irony was that I knew the Roundhouse as a rock venue where I had seen the Doors and the Stones, so I felt more at home there than I ever did at the South Bank."

He switched to playing classical guitar and discovered John Dowland at the same time as he was discovering Xenakis and Stockhausen. "I wasn't interested in the period the music came from, only in whether it spoke to me." By then he had decided that he wouldn't go to music college. "I thought it was too late, although I did wonder how I could get to study with Messiaen, how I could find the money to go to Paris. His was the name that kept coming up, and in his music of the 1960s I found an amazing balance between intellectual rigour and sensual speech. It fascinated me that you could drag this language into a space that I could recognise."

From the Greek Xenakis, too, he learnt it was no longer necessary for a composer to feel part of the central European tradition, that coming from the west of Scotland, on the Atlantic fringe of Europe, was perhaps an advantage.

Dillon's first public performance was a "little piano piece" at the inaugural Huddersfield Contemporary Music Festival in 1978. "Up to that point I had been writing things that were either never performed or were a bad experience," he says. After that, though, there were no miscalculations, just a constant stream of works, many of which he has tended to group in cycles or series. This was a consequence of his growing frustration with the structure of traditional concerts, he says, which he describes as "a parade of unconnected works with all the disruptions between them. One of the things that I did carry over from my experience of popular music was the notion that you don't break the enchantment, that going to a concert is like stepping into the magic circle. This was utopic of course, and the ideal was that one of these cycles would occupy a whole concert, but of course it has never happened."

The most important of these cycles is Nine Rivers, a series of nine vocal, electronic and ensemble works that dominated Dillon's output through the late 1980s and 1990s. It's one of the most significant achievements in British music in the last quarter century, but because of the sheer logistics and expense of mounting a performance, it has yet to be heard complete. A full cycle is, however, planned for Strasbourg in 2008 ("I'll believe it when it happens").

His first opera, Philomela, based on Ovid's Metamorphoses, was premiered in Porto in 2004, and also seen in Paris and Strasbourg, but so far there are no plans for it to be staged in the UK. It would be a good project for Scottish Opera, I suggest, but Dillon is doubtful and seems to have given up hope of receiving any recognition in the country of his birth. If you move abroad, even to England, he says, then Scotland forgets about you.

The intensely thoughtful, quietly spoken person one meets now is hard to square with the prickly reputation Dillon acquired in the early part of his career. "If I look back at the early 1980s, part of what I was doing then was reacting against what was around me, and part of that was a frustration with the way in which British music had gone, so there was a certain bloody-mindedness if you like. But it's only in retrospect I can see that I was being confrontational. Britain has always had a problematic relationship with modernism. Though it was on its last legs by the time I was beginning, modernism still produced some extraordinary works."

Yet he admits that he still feels uncomfortable if he senses he is becoming part of something; he is far happier being an outsider, away from new-music politics. The commissions come in steadily enough, and a new piano concerto will be performed at this year's Proms; a piece for the Orchestre de Paris is in progress. Even so he is never entirely pleased with what he has composed. "There are moments in pieces when I feel that my concentration hasn't lapsed. There are things that work quite well. As Beckett said, you fail and try to fail better - I can live with that."

· James Dillon's piano concerto Andromeda will receive its first performance at the Proms on August 10. The RPS Music Awards will be broadcast on Radio 3 on May 10

Laptops, ensembles and a piece called XXX Live Nude Girls!!!

Brian Ferneyhough

Now professor of music at Stanford University, California, Coventry-born Ferneyhough has reached the heights of contemporary-music fame: on the continent at least. There are precious few performances of his music here, although you can hear the UK premiere of his fifth String Quartet at this year's Aldeburgh festival (box office: 01728 687110). He's credited with starting "new complexity", the movement - even if it's a label he now rejects - thanks to the sheer calligraphic density and difficulty of his scores.

Michael Finnissy

Probably Britain's most prolific composer, Finnissy has one of the most diverse catalogues of any composer working today. A leftwing radical, he takes an uncompromising intellectual stance, but the heady textures of pieces such as his Verdi Transcriptions and Gershwin Arrangements for solo piano are sensuous and attractive. Apart from a dedicated following at the Huddersfield Contemporary Music Festival and concerts at the British Music Information Centre's Cutting Edge series, his music is shamefully under-represented on concert programmes in the UK.

Richard Barrett

Another leftwing radical, 46-year-old Richard Barrett has made his career in Germany and Holland, and he's as active a laptop improviser as he is a composer of chamber and orchestral pieces. The ensemble Elision has created a performance tradition for his music in Australia, but he hardly ever appears in the repertoires of new music ensembles such as the London Sinfonietta in this country. Now lives and works in London, so perhaps his time is coming.

Jennifer Walshe

Irish-born Walshe, 32 this year, trained in Scotland and Chicago and now lives in Berlin. Her work uses radical forms of theatre and often explores pop culture - typical titles include XXX Live Nude Girls!!! and He Wants His Cowboys to Sound Like How He Thinks Cowboys Should Sound. A rising star in Germany and also in America, we don't know enough of her music here.

Joanna Bailie

Thirty-three-year-old Bailie, born in London, is another young composer who has made her career anywhere but in the UK: Holland, then New York, and now Brussels, where she set up Ensemble Plus-Minus. Her fastidious, ambiguous music gets the odd performance by Plus-Minus and the ensemble Apartment House. There's a rare chance to hear a performance of her Five Famous Adagios this Saturday at LSO St Luke's, London EC1 (box office: 020-8576 1227).