"Where are all the women?" is a common cry at technology conferences. The appointment of Marissa Mayer – Google's 20th ever employee and its first female engineer – as the new Yahoo! chief executive does little to answer that question.

For as long as I can remember, there has always been a handful of women at or near the top of Silicon Valley companies. In the US, Carol Bartz, the last Yahoo! CEO but one (it's been a tumultuous time for a pioneering internet company that everyone fears is fading), led the computer-aided design company Autodesk from 1992 to 2006; and Carly Fiorina rose to prominence at Lucent before leading Hewlett-Packard from 1999 to 2005. Other notable company leaders include Kim Polese (Marimba, SpikeSource), Donna Dubinsky (Palm, Handspring, Numenta), current HP chief executive Meg Whitman (eBay), and June Rokoff, who in the early 1990s was Lotus's vice-president of software development. In Europe, the first female tech business leader that springs to mind is Martha Lane Fox (Lastminute.com), probably the best known, but before her was Rikke Helms (Borland, Dexterra) and Dame Steve Shirley (FI Group); the most prominent female academic computer scientist is Dame Wendy Hall, busy founding web science in Southampton.

Around 2000, both the Association for Computing Machinery (in the US) and the British Computer Society BCS (here) became concerned about what they saw as an alarming trend: the number of women entering computer science was shrinking. There was no obvious reason why this should be the case. The number of women entering other sciences, particularly medicine and biology, was on the up. Computer science does not require great physical strength or size. And there are role models, from Marissa Mayer and the others listed above to early heroines Ada Lovelace and Grace Hopper, current engineers Susan Landau and Rebecca Mercuri, and community builders Anita Borg, and Emma Mulqueeny (Rewired State). Yet at the coding fest for teens Mulqueeny ran last summer, Young Rewired State, the percentage of girl coders was still depressingly small.

At last weekend's HOPE 9 (Hackers on Planet Earth) in New York there were the usual minimalist queues for the women's lavatories – and a panel to discuss how to get more women to participate in the hackspaces. During the debate, a German woman recounted her experiences visiting hackspaces wherever she travelled. Sometimes, she said, she could sit for half an hour and no one would speak to her. The best advice from the panel organisers, Johannes Grenzfurthner and Sean Bonner was: stop complaining that there are no women around and just invite all your friends to participate. This might work for Young Rewired State, too: allow kids, male and female, to sign up in groups as well as individually so that shy kids would be sure to turn up on the day. And probably parents would be reassured if all the sponsoring locations made sure there were women as well as men acting as leaders.

For the broader problem, it's clear that a more profound change needs to take place, both in how we view mathematics, science and computers: not as boys' subjects or toys but as valuable subjects in which everyone should be literate. We have not come far enough since 1970, when my New York private school got its first computer – and only the boys had access to it. It matters: for modern computer systems to be usable by all and sundry, the make-up of the population that designs and implements them needs to reflect that of the population at large.

As for Mayer, she had probably gone about as far as she could at Google, where the two founders are likely to continue to rule. Yahoo! still has a large, persistent audience, substantial revenues, and a well-known brand name. There are enough previous CEOs who have failed to restore the company's former status as an Internet leader, that if Mayer fails she's lost nothing – and if she succeeds she'll be seen as having achieved a miracle. Good luck to her.

• Follow Comment is free on Twitter @commentisfree