An all-female Julius Caesar has just hit the stage, but it's a rarity in theatre. In a special report, Charlotte Higgins asks leading figures why women are still underrepresented at every level of the business – and what needs to change

"It's a big world in here" is the ringing phrase the Young Vic theatre in London has adopted as its motto. It's a nice play on Shakespeare, for, as he said, all the world's a stage (not for nothing did he call his own theatre the Globe). On stage, real people act out human desires and dilemmas in front of a live audience: at its best, theatre is the art form that best represents the world.

Except that it doesn't. Some months ago, the staging of two all-male Shakespeare productions at the Hampstead theatre in London uncorked an explosion of frustration from women actors, writers and directors. There was a sense of basic injustice – actor Janet Suzman talked of a "really frustrating" career where there "aren't bloody well enough parts for women"; deeper concerns were also expressed.

This failure to represent women, argued the actor, writer and director Stella Duffy, was deeply entwined with society's wider failure to put women's voices on an equal footing with men's. A sense of responsibility to the world was, she said, being ducked – particularly by our larger national stages. In an impassioned blogpost, she wrote: "When we do not see ourselves on stage we are reminded, yet again, that the people running our world (count the women in the front benches if you are at all unsure) do not notice when we are not there. That they think men (and yes, white, middle-class, middle-aged, able-bodied men at that) are all we need to see."

After I wrote an article quoting Suzman and others, Elizabeth Freestone, artistic director of Pentabus theatre, wrote to me. While an artist-in-residence at the National Theatre, she had done her own research into women in theatre, which she offered to show me. She had also done fascinating work on Shakespeare, unlocking some of the root causes of this imbalance.

The Guardian teamed up with Freestone, and we extended and updated her research. Her headline figure had been that there was a "2:1 problem" in English theatre, or two men for every woman; this was borne out by our new findings, too. Women are seriously underrepresented on stage, among playwrights and artistic directors, and in creative roles such as designers and composers. On the other hand, women are a substantial majority when it comes to the audience. According to Ipsos Mori figures produced for the Society of London Theatre in 2010, women make up 68% of theatregoers.

We looked at the top 10 subsidised theatres in England – those best placed to provide leadership – and at their record in the financial year 2011-12. (One important detail to bear in mind: the National Theatre and the Royal Shakespeare Company received at least six times more subsidy than the next-best-funded theatre, the Royal Court in London.)

The 2:1 problem begins at board level. Our sample had an average of 33% women on their boards; only one, the Royal Court, has a majority female board. Women accounted for 36% of the artistic directors; executive directors were much better represented at 67%. Of the actors employed by the 10 theatres, 38% were female, with the National coming out worst at 34%.

Of directors, only 24% employed were women; and when we examined creative teams (directors, designers, sound designers, composers), 23% were women. We found, too, that women in creative roles were less celebrated. In 36 years of Olivier awards, women have won only twice for director (Deborah Warner, in 1988 and 1992) and four times for playwright (Caryl Churchill, Timberlake Wertenbaker, Pam Gems and Katori Hall).

We were aware that our statistics did not dig into the subtleties of women's careers. Josie Rourke and Kate Pakenham, the artistic and executive directors of the Donmar Warehouse, tell me their office is almost entirely staffed by women in their 20s and early 30s. But will they rise through the ranks? "For me," says Rourke, "a huge part of the conversation is how you stick at it in your 30s, and what motivates you to move from middle to senior management." For those in freelance roles, the theatre is a hard place to sustain a career and children: no pension, no maternity leave, a nomadic lifestyle, unsociable hours. "It will take huge wisdom and honesty for theatre to investigate its culture," says Rourke.

The weight of history is not on the side of female playwrights: the canon is overwhelmingly male. Even so, we were surprised to learn that one of our top 10, Chichester festival theatre, employed no female directors and produced no plays by women over the period we looked at. Artistic director Jonathan Church suggested this was an anomalous sample; he also pointed to the financial pressures on regional theatres and the need, when programming contemporary or 20th century work for a big auditorium, to put on "names" that have had West End success, such as Coward, Ayckbourn, Stoppard, Hare, Rattigan: all men.

There is currently a blooming of extraordinary female voices in theatre, among them Lucy Kirkwood, Lucy Prebble and EV Crowe. Yet we found that women writers accounted for only 35% of the new plays produced – another expression of the "2:1 problem". There is, however, some cause for hope: 41% of the plays commissioned by our theatres, but yet to reach the stage, are by women. Nicholas Hytner, artistic director of the National, sees this as a hugely positive sign: "I won't put a date on it, but in not too many years the gender balance of directors and writers will be 50-50. I can say that with confidence because I look at directors and writers in their 20s and 30s and it is 50-50."

Still, there are nuances behind such figures. Of the seven plays by women produced at the Royal Court in this period, all but one were staged in its smaller auditorium: meaning a smaller fee, fewer royalties, a lower profile. If these women are being nurtured towards bigger careers, so much the better; but will they be allowed to make the leap? The playwrights I spoke to talked of careers that had sparkled in their 20s and stalled in their 30s. The really big commissions had never come, and they found their male peers outstripping them. Some had turned to screenwriting: still tough, but, according to playwright Zinnie Harris, "In TV, I haven't encountered the feeling that you're not going to get to the top because you are a woman."

Only two original plays by women have ever been staged in the largest auditorium at the National: Rebecca Lenkiewicz's Her Naked Skin in 2008, and Moira Buffini's Welcome to Thebes in 2010. According to writer Tanika Gupta, who has a new play at the Swan in Stratford next year, there is an underlying doubt about the material women are capable of taking on. "The argument is that women can write very good domestic drama, but are not intellectually rigorous enough to do the big plays. We need to be given the chance to write those state-of-the-nation plays."

Harris adds: "It is somehow harder for people to embrace a play written by a woman, whatever its quality. There is something slightly unseemly about filling stages with our voices, whereas men have a sense of filling Chekhov's or Ibsen's shoes. The woman who raises her voice becomes shrill and hectoring; the man becomes authoritative." She believes the media is at fault, too. "When plays by women don't work, they are over-condemned. With men, they are seen as a step on the way to developing an interesting voice." The statistics bore out what we had suspected: women playwrights write more roles for women than their male counterparts. Women wrote 49% of their parts for women; men wrote 37%.

A clear message began to emerge about the importance of women running instititutions. Female artistic directors had staged many more plays by women than their male counterparts. Roxana Silbert at the Birmingham Rep came out top: 32% of the plays she has directed are by women. She was followed by all the remaining women. Then came David Lan at the Young Vic (15%), followed by all the remaining men. Neither Hytner nor Gregory Doran, artistic director of the RSC, has ever directed a play by a woman.

It is clear that history comes into play here: both men have had careers that focus on the classics, with Doran a specialist in the 16th- and 17th-century repertoire. When I put this to Hytner, he said he believed his own record was "irrelevant: there are all sorts of things I have never directed because I am not very good at directing that kind of thing". Freestone disagrees: "Profile and visibility matter. Those in charge of national organisations have a responsibility to show leadership." Vicky Featherstone, who in April becomes artistic director of the Royal Court (England's most important theatre for new writing), says: "It is a no-brainer that there should be equal representation of men and women in the theatre. It is absolute common sense and I expect nothing less."

Our research was not intended to browbeat individuals. Rather, it was meant to focus debate on fact rather than anecdote, and to encourage theatres to take gender into account. As Freestone puts it, "This is about asking: are you thinking about gender balance? Do you ask the question? If you have 10 writers under commission do you think about it if they are all men? If you never think to ask, that's when you are in trouble." She argues that much more needs to be done. "The theatre world remains strangely passive in the face of overwhelming evidence of its failure to address the gender imbalance both on and off stage. Programming, commissioning and casting decisions are routinely made without any consideration of gender."

Why is this 2:1 ratio so stubborn? Like so much else in English theatre, it goes back to Shakespeare. He was, of course, writing for all-male companies; and, though he wrote transcendent parts for women, there aren't very many. Of his 981 characters, 826 are male and 155 female: 16%. Women have less to say, too: of roles with more than 500 lines, only 13% are female. The most wordy of Shakespeare's heroines, Rosalind, has 730 lines. Hamlet, his most loquacious hero, has 1,539.

This means less work for women actors, and fewer opportunities for them to develop their skills via the many workaday parts available to men. The Shakespeare problem has persisted because, until relatively recently, much of British theatre relied on the repertory system: a company of actors performing a handful of different plays in a season. Shakespeare would frequently sit at the heart of such a company; many new plays would therefore tend to be written for a similar gender balance. There is something deeply culturally engrained about this: it runs so deep that we have become used to not seeing women equally represented, arguably aided by a culture of complacency. The Shakespeare inheritance has meant, says Freestone, "we've been caught thinking that 30% women is good enough. I'm not saying there's been institutional sexism, but there has been a sort of blindness to female actors because of the burden of the classical canon."

The answer to this is, surely, gender-blind casting, especially in the classics, where colour-blind casting has ceased to be a matter for comment. Phyllida Lloyd, whose current all-female Julius Caesar at the Donmar has provided a focus for recent debate, is clear: "If I were running the RSC, I would make it 50% male and 50% female actors – and then I would work out how to do the plays," she says. "It wouldn't be a stranglehold, it would be liberating."

When I speak to Doran, he tells me he has invited Lloyd to run a company of actors at the RSC on precisely those lines. "Watch this space," he says. Hytner believes he cannot impose strictures, that directors' casting decisions have to be based on their own instincts: "I am very interested in gender-blind casting and often think it is excellent. But I can't tell writers how they should write and directors how they should cast. What I admire is that the Donmar Warehouse responded to Phyllida Lloyd's desire to do Julius Caesar that way." When the National staged Timon of Athens this year (Shakespeare's most male-heavy play), five male parts were given to women – though the share of female lines increased only from 0.67% to 14%.

For many, this is not enough. "I really believe that a more imaginative approach to casting the classics will unlock all kinds of creative interpretations, and naturally feed in to all other areas – male playwrights writing female parts, more confidence in female creative teams," says Freestone. Duffy argues that fringe theatre is better balanced, and that the national companies should be leading by example – "and they are not, and I find it heartbreaking". She adds: "We have a responsibility to make the world a fairer place, and sometimes you have to do a little social engineering to make that happen." In the end, she says, "I don't know why people don't just suck that up – and get on with it."