Last year, Tom Becker won the Waterstone's prize for children's fiction with his first novel, Darkside; last week he won another award, the Calderdale children's book prize. The talk among agents and publishers has been about his suspenseful prose, his great potential. But few people have been talking about a more salient fact: that the book's concept and story was generated not by Becker, but by focus groups.

The company behind Darkside is Hothouse, a London-based business that aims to give children what they say they want from stories, rather than what adults think they want. Becker's book was the company's first attempt at book-by-focus-group, and it is part of a successful supernatural horror series aimed at boys aged up to 12, published by Scholastic. In April, Puffin books launched a new series, Fright Night, also conceived and delivered by Hothouse.

Hothouse uses a market research company to put story ideas to children, who are observed from behind a one-way mirror. Using dummy covers, short excerpts and blurbs to prompt conversation, researchers ask the children their opinions on which characters, plots and ideas they enjoy most. Each child is also visited at home by a researcher, who finds out what kind of books they already own and read. Drawing on this research, Hothouse commissions a team of writers accordingly.

You could ask whether Hothouse is publishing books that will endure, or merely pushing products. Consensus, surely, takes us somewhere far away from art. Creatives are meant to work in isolated, inspired flashes, fuelled by long nights of the soul - not by means of spreadsheets and a scientific research process.

But publishing isn't the only area of the arts dabbling in focus groups. EMI's new owner, Guy Hands, had musicians choking on their Red Bull this January when he revealed that focus groups would play a more significant part not only in promoting his company's records, but in creating them.

"When you look at which car companies are succeeding, it's the ones which work with their customers," he said. "Are clothes not creative? Is fashion not creative? Is food not creative? The only real difference is that these industries have learned to work with the customer and not force-feed them."

The theatre has long used previews to test plays on audiences before opening night. Much can be done to improve a production even in those late stages, and many experimental companies are now looking to try their ideas on a live audience at the very start of the artistic process. At BAC, the renowned south London fringe venue, monthly "scratch nights" offer theatre practitioners the chance to try out work in development - anything from a loose collection of scenes to a vague notion they had in the bath. More important than the performance itself is the after-show event, nattily titled A Beer for an Idea, where audience members are invited to give feedback in the bar.

"It's a really good trade-off," says Harun Khan, who gave a scratch performance of a work in progress, The Twins. "It ups the ante because the artists are taking more risks, the audience get to see something born - or flop - and directors get an audience response without having to risk financial or critical disaster." Past successes have made it a popular event: Stewart Lee's Jerry Springer - the Opera emerged from the BAC programme, as did his What Would Judas Do?.

"You can tell the audience what you want to know from them, and you can be specific whether it's which characters they identify with, or how clear they found the story," says Khan. "Sometimes you just want a really open response."

BAC can rely on an audience with a taste for the avant garde, but what constitutes development strategy there could look like crowd-pleasing elsewhere. "There is a danger of that," admits Khan. "But creativity and good ideas don't belong just to artists and creatives. The more people allowed into your process, the more potential you have for getting one of those great ideas - even if it's from someone who doesn't normally go to theatre. The more open you are, the bigger your resources."

Andrew Therkelsen, who works for market research company Discovery, says he has noticed a growing interest in focus groups from the art world. "Editors, producers and those in more creative genres always tended to shy away from using research in this way, mainly because it doesn't always get it right. But increasingly, they are using it to define and develop their ideas. Traditionally, any audience feedback done in the creative world is reactive rather than pre-emptive. But we're saying we can help decide the best outcome before you do it."

Could focus groups help democratise art? Reg Wright, CEO of Hothouse, points out that one of the great advantages of listening to young readers is that they have a surprisingly good feel for where a story should go. "We've had children come up with great ideas for plots," he says. "They may not be sophisticated, but they'll make it their own. Our job, in the end, isn't to implement what they say, but to interpret what they want."

Wright's company has also developed Rapport, an online focus group that gives publishers access to the opinions of 10,000 readers. He believes it is good for publishers to be challenged on their assumptions about what readers want. "We are not our target audience," he points out. "You see publishing houses full of 28-year-olds who say, 'We know what people want.' No: they know what they want. At least we admit we don't know anything. The kids' feedback is a great example. Even what parents think their kids would like is often wildly wrong."

In the film industry, production companies hire whole audiences to give their verdict on a movie before the final edit. Ridley Scott's Blade Runner was famously lumped with an ending he hated after focus groups found his original ending too negative. But then the critics hated it, and it wasn't until 10 years later, when Scott released a director's cut, that the happy ending was removed and the film got the acclaim it deserved. Sometimes it doesn't pay to give people what they want.

David Livingstone, president of marketing at Working Title films, makers of Notting Hill and United 93, argues that "the director isn't a puppet" and that test screenings fulfil a vital function. "When you make a film, there are a lot of points of view. But 400 people in a screening is a consensus." He adds that some directors are better at interpreting audience response than others. "Those who have more experience in TV tend to be better at knowing how to use that information."

Of course, television already has the pilot - the single first episode that is tested on an audience to see if it's worth going ahead with the rest of the series. However, piloting a show is prohibitively expensive. Instead, broadcasters are increasingly looking to pre-emptive focus groups, and encouraging commissioners and programme-makers to pay more attention to their results. The BBC used focus groups before launching Holby City; even independent producers now commission their own market research, although it remains costly enough that only the larger indies can afford it.

At a Royal Television Society seminar, Clare Salmon, a former marketing director of ITV, said she was hoping to usher in a "new era of marketeers and programmers working together creatively". ITV are considered to be at the forefront of audience research; they use "segmentation", categorising their viewers by attitude rather than by the more traditional model of age and gender. By dividing audiences into 16 different segments, based on lifestyle and beliefs, researchers can help suggest what kind of storylines and characters will appeal - or flop.

"It's not about turning programme-makers into analysts," says Salmon. "It's not about saying that a certain segment likes gardening and vicars, so we must put a gardening vicar in every programme we make for that audience. But we can help commissioners avoid collisions where they pick on something that's an attitudinal divide." She uses The Real Good Life as an example of a show that flopped because it failed to take account of the tension between its subject matter - how to live ethically - and a "segment" audience that believes environmental threats are overstated.

However, producer Alex Graham, who worked on Who Do You Think You Are? for the BBC, believes creatives need to keep a healthy distance from focus group material. "It's like internalising the rules of a sport before you play it," he says. "There's a point where you put it all in a drawer and lock it away. The wrong amount of information at the wrong time can cripple creativity stone dead."

Perhaps the best examples of this are the paintings created by Vitaly Komar and Alexander Melamid, two Russian conceptual artists who emigrated to the US in the late 1970s. In 1994, they collaborated on USA's Most Wanted, a painting created in response to a survey of 1,001 Americans that asked people what they considered to be good art. The result, a kitsch landscape featuring two deer and a hippopotamus (the poll put a premium on wildlife), is dominated by a mountain lake and is "the size of a dishwasher" - the favourite size, apparently, for a work of art.

The pair went on to do similar projects in a range of other countries, from Poland to China, and the results were unnervingly similar. Large expanses of water featured on almost all their canvases. From which we can deduce that good art is, well, blue. Perhaps it's no wonder market research has never held much sway in the art world.

Emma Dexter, director of exhibitions at Timothy Taylor Gallery and a former Tate curator, says: "The point of art is to imagine the unimaginable, and the worst possible scenario would be the idea of an artist making what people thought they wanted to see. Increasing globalisation means we should treasure the sense of locality and specificity about artists' work, rather than looking for a lowest common denominator." She does, however, think that galleries and museums can and do benefit from focus groups: "Learning what people come to it for, how they experience it, how they move through a building."

Therkelsen, of Discovery, agrees. "At the high end of art, that's always going to be led by the artists - quite rightly. But where we can help is in the more mainstream, consumer-led entertainments." As Tom Becker's awards prove, the method can work. Perhaps "popular art" will not always be a paradox ·

Emma John sits in on a focus group creating a show

6.30pm Eight people meet for the first time in a London studio, on the wrong side of a one-way mirror. Drawn from the databases of Criteria Fieldwork, they're as diverse as you could wish. A policewoman who loves musicals. A graphic designer who won't go to the theatre unless there's a Hollywood star. A cool-looking dude who raves about an avant-garde, site-specific work he recently saw, until a middle-aged woman next to him interrupts: "You've just described my worst nightmare." It's like Big Brother already.

6.55pm: Andrew and Vicki of market research company Discovery take the group through an exercise they call "psycho drawings". Everyone is asked to recall their best ever theatrical experience, then draw how they felt about it. Doodles include an alarm, a firework and a disturbing cartoon of a brain transplant (the avant-garde dude, naturally).

7.15pm: Now Andrew wants the group to identify what makes bad theatre. Poor acting, underestimating an audience, plays that go on too long all take a kicking. As, inexplicably, does Jeremy Irons.

7.30pm:The group brainstorms subjects for a play. Global warming and terrorism are high on the list; current issues are considered the most exciting because, as one woman says, "You feel you're paying attention to important topics, which makes you feel good about yourself."

8pm: The musical theatre lovers want an R&B love story for the Asbo generation. "It could show how kids have no real role models now," says the policewoman.

8.15pm: Our Asbo play has scrapped romance for a tougher theme: a young lad from the estates who keeps getting in trouble, but whose love of music redeems him. Someone points out that they've just written 8 Mile: the Musical. The ideas are gathered together, and passed on to playwright Ben Yeoh, who has had plenty of experience writing synopses for Radio 4 and the Gate Theatre in London. Here's what we came up with:

Asbo: the Musical

Jake always wanted to be a singer like his dad, who left home when he was young, but he has been getting into trouble on his estate. One fight leaves Steve hurt: Jake, though blameless, ends the night in the cells. He meets Fats, who knew Jake's dad and tells him how his dad wanted to be a great soul singer but died on tour. Fats convinces Jake to live the dream his father couldn't.

On his release, Jake impresses at an open mic night, and a record producer asks to meet him. On the way to the meeting, Steve's brother - desperate to show his crew that he is a man - knifes Jake in revenge. Jake is rescued by bouncers and signed by the record producer; Steve's brother ends up in jail, his talents wasted.

· This article was amended on Friday July 4 2008. Clare Salmon, whom we described as the marketing director of ITV, left that post in December 2006. This has been corrected.