Arthur C Clarke’s adage that “any sufficiently advanced technology is indistinguishable from magic” may or may not be true, but what is definitely true is that computer software has magical properties. It’s pure “thought stuff”: a programmer has an idea; they encapsulate it as a string of symbols that are then fed into an inanimate machine. And then the machine executes the instructions encoded in the symbols. And it obeys those instructions faithfully, unquestioningly and without tiring. Which is why being a programmer is a bit like being Napoleon – except that, unlike Bonaparte, the programmer doesn’t have to worry about victualling the troops.

As with any other line of work, there’s a spectrum of ability in programming that runs from barely competent to genius. At the top end are people who are not just 10 or 20 times better than the average, but a million times smarter. So to call them programmers is like calling Christian Dior a dressmaker; they are more like artists whose medium just happens to be software. One of them, Bill Atkinson – the guy who created MacPaint way back in the 1980s – once said that “the art of creating software that is usable by individuals [by which he meant normal human beings like you and me] is a communication skill. It is not a programming skill... The most exciting thing for me is when I see people amazed and pleased at the newfound power they got from a program – where they say: ‘Wow, I can do this!’”

One of the distinctive things about the tech giants that now mediate our online lives is that they not only attract top-rank computer scientists and software engineers, but that this particular kind of human capital is arguably their most precious asset. This may seem a strange thing to say about corporations that have colossal physical assets in terms of technical infrastructure (all those server farms and silos of data), but last week the New Yorker ran a fascinating essay that vividly illustrates the point.

The article (by James Somers) focuses on a friendship between two Google programmers – Jeff Dean and Sanjay Ghemawat – who have played an outsized role in the company’s ascent to global domination. Their first contribution came when in 2000 they identified a hardware problem that was seriously crippling its search results – just at the moment when the co-founders were negotiating for the contract to provide Yahoo (then a major internet company) with search-engine services.

Having located the fault, they then effectively redesigned Google’s server architecture. The company had from the outset bought cheap PCs for its server farms: good for growth, bad for reliability. So Dean and Ghemawat set about making the system more fault-tolerant (and therefore resilient). “Working 90-hour weeks,” Somers writes, “they wrote code so that a single hard drive could fail without bringing down the entire system. They added checkpoints to the crawling process so that it could be restarted midstream. By developing new encoding and compression schemes, they effectively doubled the system’s capacity.”

Google employees sign letter against censored search engine for China Read more

Server farms are full of spinning hard disks. Since the outer edge of a spinning disk moves faster than the inner one, other Google engineers had moved the most frequently accessed data to the outside, so that bits could flow faster under the read head, but had left the inner half empty. Dean and Ghemawat used that space to store preprocessed data for common search queries. “Over four days in 2001,” writes Somers, “they proved that Google’s index could be stored using fast random-access memory instead of relatively slow hard drives; the discovery reshaped the company’s economics.”

And so on. Google has 11 grades of engineer – from IT support staff to senior fellow. Some of the world’s leading experts occupy level 10. But there are only two senior fellows – Dean and Ghemawat, recognition that, if you aspire to global dominance (as Google definitely does), you need technological Mozarts, not session musicians.

And when you’ve got them, you need to retain them – which can be tricky because they’re not in it for the money: engineers of this calibre can work anywhere. What turns them on are technical challenges and hard problems. But also, for some, it’s important that the company’s mission resonates with them. That’s why recent protests by Google staff against the firm’s treatment of sexual harassment, working with the Pentagon and developing a crippled search engine for the Chinese market should worry its bosses. It should worry Facebook’s bosses, too: it’s inconceivable that its recent scandals aren’t getting to its leading techies. They won’t want to get to the point where working for Facebook is seen as the moral equivalent of working for a tobacco firm. Even magicians crave admiration sometimes.

What I’m reading

How Facebook is fuelling populist rage in France

A report by Frederic Filloux.

Nick Bostrom’s ‘vulnerable world‘ hypothesis

Sobering (and clever) essay on the celebrated futurist’s blog.

Quartz’s Dave Gershgorn

On how the myth of the lone inventor is often just that – a myth.