Some time in the late 1990s, the music video director Mark Romanek approached his friend Trent Reznor to compose the score for his feature debut One Hour Photo, a thriller in which lab technician Robin Williams becomes obsessed with a suburban family. It was not the happiest time for Reznor, who was straining to complete the third Nine Inch Nails album and, he says, was about to "slip off a cliff of addiction". While things were still at the demo stage, he received an apologetic call from Romanek saying the studio was pressuring him to use a "real composer".

The physical evidence of how much has changed since those days is 34cm tall, gold-plated and stands on the mantelpiece of Reznor's house in Beverly Hills. He picks up the Oscar for best score that he and Atticus Ross won earlier this year for their work on David Fincher's The Social Network and hands it to me. It's heavier than it looks. "That's one of the surprises when your name is read out," he says. "Fuck, don't trip up! Fuck, how many steps? Fuck, it's heavy!"

The 46-year-old Reznor is a much warmer, droller character than he appeared during Nine Inch Nails's anguished first decade when he sang lines such as "I hurt myself today/ To see if I still feel." "I'm not at war with myself as I once was," he says. "Life isn't miserable. I don't miss that." Now father to a one-year-old boy, with singer Mariqueen Maandig, Reznor wears jeans, a hoodie and a shaggy, stay-at-home beard, although he still dyes his hair raven-black. We sit out in his hillside garden beside two dozing greyhounds. In the distance, the skyscrapers of downtown LA look foggy and unreal, like a painted movie backdrop awaiting an alien invasion.

Reznor has spent the last 15 months working on Fincher's high-stakes adaptation of The Girl with the Dragon Tattoo, Stieg Larsson's bestseller about a journalist and a computer hacker looking for a woman who has been missing for 40 years. Already, Reznor's score has been nominated for a Golden Globe. The two men admired each other's work long before they teamed up: Fincher used Nine Inch Nails's Closer in his second feature, 1995's Seven, which saw Brad Pitt and Morgan Freeman hunting serial killer Kevin Spacey. But then both Reznor and Fincher have an affinity with grime and shadow; both are intrigued by tormented machismo and physical distortion; and both have a streak of gruelling perfectionism.

In the 1990s, Reznor compiled a berserk collage of other people's songs for Oliver Stone's Natural Born Killers, and a more restrained collection for David Lynch's Lost Highway. But when Fincher approached him two years ago to write his first full-length score, for The Social Network, Reznor was exhausted from touring and needed a break. He said no. A few months later he called the director to apologise and ask him to bear him in mind for any future projects. Fincher replied: "I'm still waiting on you to do this."

Reznor's only problem was that he had no idea how to go about it. "I thought maybe I should call Hans Zimmer," he says, referring to the German composer who scored Gladiator and The Dark Knight, "and see if I could hang out with him and make coffee for him for a while and take a crash course in how the fuck you score films. But instead, I sat with Fincher and said, 'I'm not going to bullshit you. I don't really know how to do this. What do you want?'"

Fincher showed him 45 minutes of the rough cut. "It struck me I had no idea what type of music would even go in this," says Reznor with a dry chuckle. "It's just fairly unlikable guys bitching about stuff in rooms. No epic battles or UFOs." Fincher said he wanted to hear "the sound of creativity", so Reznor and Ross produced a mass of material based on the characters' conflicted feelings about power, class, technology and money. To Reznor's amazement, most of it ended up in the finished movie. "I was adhering to the one lesson I'd heard: music is in service to the picture. It's powerful but it's a supporting role."

The Girl with the Dragon Tattoo was a much longer process. Reznor gave Fincher an initial 90 minutes of music last autumn to use while he was shooting the movie; the finished score runs to three hours. Reznor walks me into what he calls his "adult playpen of knob-twiddling": a small garage converted into an Aladdin's cave of instruments, mixing desks and synthesizer modules, their lights winking in the dark. An assistant boots up the movie's eight-minute trailer on a large screen. The first few minutes of tense conversations and snow-shrouded Swedish landscapes are accompanied by wintry drones and tinkling bells. As the editing gets faster and the clips more violent, a brutal metallic beat drives the trailer to its climax. It's unusually exhilarating to see a movie on this scale without the standard orchestral sound.

Not so long ago, Hollywood's default setting, says Reznor, was still "that type of orchestral movie score that all kind of sounds the same". Rock bands or electronic musicians like Nine Inch Nails were limited to appearing on compilations of songs "from and inspired by" blockbuster movies: cynical cross-promotion exercises with little relation to the score. But in recent years, directors have sought out the likes of Radiohead's Jonny Greenwood (There Will Be Blood), the Chemical Brothers (Hanna) and Nick Cave (The Assassination of Jesse James By the Coward Robert Ford). In Reznor's experience, the old guard have been surprisingly welcoming. "There wasn't any snobbiness, like: 'That's not a real score.' I had Hans Zimmer come up to me and say, 'The fact you've won all this stuff is great, in terms of opening up perceptions of what can be incorporated into a movie score.'"

Trent, I want more modulation

The bleak themes and frozen landscape of The Girl with the Dragon Tattoo inspired Reznor to start off with what he calls "Eraserhead drones" (one track on the Lost Highway soundtrack was self-deprecatingly titled Various Ominous Drones). "It gave us a few starting points, revolving around emptiness and dread. Then we started inserting chunks of texture and melody."

As they sent work-in-progress off to Fincher, who was on location in Europe, the director would respond with gnomic emails. "He'd say, 'Maybe if it had a little more modulation in it.' What the fuck does that mean? Often I'll take an email and say, 'OK, let's go around the room and see if anyone can guess what this means.' I think part of it is him challenging me. He's not going to tell me to move the kickdrum one-eighth of a beat later. It's going to be: 'How does this character's thinking apply to the kick drum?'" So he's making you think like a director? Reznor nods. "Or he could just be fucking with us the whole time. Which I also appreciate."

Fincher's own ideas were, however, the hardest to execute. It was his suggestion to open the movie with a monstrously heavy version of Led Zeppelin's Immigrant Song, sung by Karen O from the Yeah Yeah Yeahs. Likewise the demented take on Grieg's In the Hall of the Mountain King that appeared during the Henley Regatta sequence in The Social Network. "Imagine that song playing three or four hundred times a day for weeks and weeks. 'What if it has a little bit more –' 'Aarrrrrgggh!'" He pulls a beleaguered face.

But in both cases, the stress was justified. "In my own real-job world, I'm at the top of the pyramid of creative control – but it's been fun to not be that guy and to have a boss. The only way that works is if there's mutual respect, and there is. You learn to trust his instincts. Nine out of 10 times – no, let's not go that far. Four out of five times, Fincher's right."

Since winning the Oscar, Reznor has had a flurry of offers; but he's wary about becoming a full-time movie composer. "If every director was as smart and challenging as David Fincher, OK, that's a discussion. But they're not." Either way, he's not short of work. How to Destroy Angels, his new band with Maandig and Ross, release their debut album next year. He has also been approached to direct his first movie, and is developing a 10-part HBO mini-series very loosely inspired by his 2007 album Year Zero, a dystopian protest record – although he's having problems finding the right screenwriter.

"I chose not to write it myself because I don't know how to do that. But now I'm thinking I should have." He shrugs happily. "Somebody's got to do it, right? How hard can it be?"

• This article was amended on 23 December 2011. The credit list in the above box included people who worked on the Swedish adaptation of The Girl With the Dragon Tattoo. This has been corrected.