We live in an age of desperation. Never in living memory has the competition for job openings been more intense. Never have job interviews been tougher.

For some job seekers, Google is the shining city on the hill. It's where the smartest people do the coolest things. In the US, Google regularly ranks at or near the top of Fortune magazine's list of 100 Best Companies To Work For. But unsexy firms also find themselves with multiple well-qualified applicants for each position. That is very good for the companies that are able to hire. Like Google, they get to cherry-pick the top talent in their fields. It's not so good for the applicants. They are confronting harder, ruder, more invasive vetting.

This is most evident in the interviews. There are, of course, many types of questions traditionally asked in job interviews. These include the "behavioural" questions that have almost become clichés: "What is your biggest failure in life?" Questions relating to business: "How would you describe Holland & Barrett to a person visiting from another country?" And finally, there are open-ended mental challenges, such as how you would weigh an elephant without using a scale – something for which Google is particularly known, an attempt to measure mental flexibility and even entrepreneurial potential. The answer? Nudge the beast on to a barge. The elephant's weight will cause the barge to sink several inches in the water. Draw a line on the barge's hull to mark the water level. Then direct the elephant back on to land. Load the barge with 100lb bags of sand (or whatever is handy) until it sinks to the line marked on the hull. The elephant weighs as much as the sand.

The style of interviewing at Google is indebted to an older tradition of using logic puzzles to test job candidates at technology companies. Consider this one: the interviewer writes six numbers on the room's whiteboard – 10, 9, 60, 90, 70, 66. The question is, what number comes next in the series?

Most of the time, the job applicant stumbles around, gamely trying to make sense of a series that gives every indication of being completely senseless. Most candidates give up. A lucky few have a flash of insight.

How would you weigh an elephant without using a scale? Photograph: Getty Images

Forget maths. Spell out the numbers in plain English, which gives you the following: ten nine sixty ninety seventy sixty-six. The numbers are in order of how many letters are in their names. Ten is not the only number you can spell with three letters. There's also one, two and six. Nine is not the only four-letter number; there's zero, four and five. This is a list of the largest numbers that can be spelled in a given number of letters.

Now for the payoff: what number comes next? Whatever number follows sixty-six should have nine letters in it (not counting a possible hyphen) and should be the largest nine-letter number. Play around with it and you'll probably come up with ninety-six. It doesn't look like you can get anything above 100, because that would start "one hundred" requiring 10 letters and upwards. You might wonder why the list doesn't have 100 (hundred) in place of 70 (seventy). "Million" and "billion" have seven letters, too. A reasonable guess is that they're using cardinal numbers spelled in correct stylebook English. The way you write out the number 100 is "one hundred".

At many of these companies, the one and only correct answer is 96. At Google, 96 is considered to be an acceptable answer. A better response is 10,000,000,000,000,000,000,000,000,000,000,000,000,000, 000,000,000,000,000,000,000, 000,000,000,000,000,000,000,000,000,000, 000,000,000. Aka "one googol".

That's not the best answer, though. The preferred response is 100,000,000,000,000,000,000,000,000,000,000,000,000,000, 000,000, 000,000,000,000,000,000,000,000,000,000, 000,000,000,000,000,000,000,000. Ten googol.

Puzzles such as this have drawbacks as interview questions. The answer here is a simple matter of insight: either you get it or you don't. There isn't a process of deduction to relate, and thus no way to distinguish someone who solves the problem from someone who already knew the answer. At Google, of all places, anyone applying for a job knows how to use a search engine. It's expected that candidates will Google for advice on Google interviews, including the questions asked. Consequently, Google encourages its interviewers to use a different type of question, more open-ended, with no definitive "right answer".

You and your next-door neighbour are holding car-boot sales on the same day at the same place. Both of you plan to sell the same item. You plan to put your item on sale for £100. The neighbour is going to put his on sale for £40. The items are in identical condition. What do you do, assuming that you're not on especially friendly terms with this neighbour?

The bit about not being on especially friendly terms should tell you that a strategic response is expected. As should the fact that this question is frequently asked at the more aggressive Wall Street houses. The friendly solution is to pull the neighbour aside and say, "You're throwing money away by offering it for £40." But this plan is not considered an especially good answer. Suppose the big spender finds two identical items on sale for £100. He's equally likely to choose either one, and the other may go unsold.

You simply want the neighbour's item off the market. You might offer to pay the neighbour not to market his item. A better answer is simpler: buy the neighbour's item. He'll be pleased to sell his item immediately. He's not likely to be offended or to raise the price. You can haggle, like any other buyer, and may get it for less than £40. Why should you want his item? When you put something on sale for £100, you hope to make a decent profit, compensating for the time you've invested in selling it and factoring in the chance that it won't sell. Anything that diminishes the chance of your item's selling in effect costs you a significant fraction of that £100.

If you were a cartoon character, which one would you be and why? Photograph: Sportsphoto/Allstar/Cinetext Collection

The numbers in this puzzle were chosen so that the neighbour's price is comparable to the economic damage he's doing to you. By buying the item, you get the right to keep it off the market when that suits your purposes, plus the right to sell it at any price the market will bear. Anything you get from selling the second item is pure profit. The best plan is to hide one item until the first one sells. Then put the second item on sale at a reduced price, according to how late in the day it is.

As the job market collapsed in 2008, employers took to career-fair or phone interviews in which the interviewer poses so-called screener questions or litmus tests. These are simple questions or criteria that (supposedly) weed out the "wrong" people. Many companies ask trivia questions about the firm itself. Morgan Stanley asks interviewees to name a recent story they read in the Financial Times – apparently a lot can't – or to give the square root of 0.01 (it's 0.1).

Bloomberg LP is big on proofreading. Some applicants are given a test in which they have to count the number of times a given letter appears, upper case or lower case, in a paragraph. It's a lot harder than it looks. (Don't believe it? Count the hs in this paragraph. There are 15, and hardly anyone gets all of them.)

"If you were a cartoon character, which one would you be and why?" This is a question that Bank of America has asked aspiring personal bankers. "I said Yogi Bear," one applicant recalled. "I can't remember why I said this, but the hiring managers were all applauding it." He was given the position immediately. Questions such as this, sometimes downright silly, have become more common. Since being a maths or tech wiz is irrelevant to running most businesses, mainstream companies have redoubled their efforts to find the perfect match of candidate and corporate personality. Online retailer Zappos has asked, "On a scale of one to 10, how weird are you?" The preferred answer there is somewhere in the middle, CEO Tony Hsieh explained. A one is "probably a little bit too strait-laced for us" and a 10 "might be too psychotic".

The unanswered question is whether today's ever-more-polymorphous interviews succeed in identifying better employees. The usual justification for creative thinking riddles and personality assessments is that they test broad, general abilities not tied to a specific set of skills. Whether they do that is hard to say. What is certain is that "pet" questions take on a talismanic quality for some interviewers. Just as athletes don't change their shirt during a winning streak, interviewers keep asking the same question because of a few remembered instances where it supposedly "worked". The fact that many of the most admired, innovative companies use such interview questions seems to speak for itself ("You can't argue with success").

Try this one. You're locked in a pitch-black, empty room with bare walls and no electric lights. You've got a book of matches, a box of tacks and a candle. How would you attach the candle to the wall for a light?

The best answer: empty the box of tacks. Take the box top, turn it upside down and tack it to the wall. The box top projects out like a little drawer. Then put a tack to attach the candle to the box's bottom. The tack's point, projecting through the box bottom, serves as a pricket. Finally, slide the box bottom into the box top on the wall. The nested top and bottom will be sturdier than either alone, and safely support the weight of the candle.

The great physicist Richard Feynman once applied for a job at Microsoft (so runs the guaranteed-apocryphal story). "Well, well, Dr Feynman," the interviewer began. "We don't get many Nobel prizewinners, even at Microsoft. But before we can hire you, there's a slight formality. We need to ask you a question to test your creative reasoning ability. The question is, why are manhole covers round?"

"That's a ridiculous question," Feynman said. "For one thing, not all covers are round. Some are square!"

"But considering just the round ones, now," the interviewer went on, "why are they round?"

"Why are round manhole covers round?! Round covers are round by definition! It's a tautology."

" Uh, right. If you'll excuse me, Dr Feynman, I would like to consult with our human resources department."

The interviewer left the room for 10 minutes. When he returned, he announced, "I'm happy to say that we're recommending you for immediate hiring into our marketing department."

This joke pokes fun at one of the most famous brain-teaser questions, long associated with Microsoft and alleged to have been devised by Steve Ballmer himself. It expresses deep ambivalence about this style of interviewing. Feynman (a childhood hero of Sergey Brin's) shows more creative thinking than Microsoft's so-called right answer (the "right answer" to the manhole question by the way, is because a round manhole cover, unlike a square one, can't fall in the hole).

This "higher level of abstraction" figures in many of Google's interview questions. Here's an example. You want to make sure that Bob has your phone number. You can't ask him directly. Instead, you have to write a message to him on a card and hand it to Eve, who will act as go-between. Eve will give the card to Bob, and he will hand his message to Eve, who will hand it to you. You don't want Eve to learn your phone number. What do you ask Bob? This question is usually asked of software engineers, who instantly recognise the names "Bob" and "Eve". In computer science textbooks, it's conventional to speak of "Alice" sending a coded message to "Bob" (it sounds a bit more human than saying, "A sends a message to B"). The rote villain of the textbooks is a snoop called "Eve" (for "eavesdropper"). Coded messages are vitally important on the internet – they're the basis of e-commerce and cloud computing. Eve's many guises include hackers, spammers and phishers. It is not too much of an exaggeration to say that this interview question presents, in a kernel, the central problem of our wired world.

It also reveals very different ways of thinking about problems. There is a technically brilliant solution. Every textbook discussion of Bob and Eve segues into an exposition of RSA cryptography, the type used by PayPal and other forms of electronic commerce. Suffice to say that RSA involves some heavy computation. That's OK, because it's always done by computer. The smart interviewee is led to wonder whether there's any way to explain RSA to Bob as part of a message that could fit on the back of a business card. This is something like telling your grandmother how to build an iPad so clearly that she could make one.

It's possible to explain a bare-bones implementation of RSA to a naive Bob who doesn't know how to code. The candidate who succeeds in drafting his RSA message to Bob will feel he's knocked the ball out of the park. Not so fast. He's just given the "Microsoft answer". Eve or no Eve, Bob is sure to balk at following complicated instructions just for the mundane task of confirming a phone number. Google interviewers expect engineers to know RSA, of course, but they're especially impressed by those who come up with a simpler, more practical answer. Tell Bob to call you (ideally, give a specific time). If your phone rings, bingo. If not, that tells you he doesn't have the right number. That's all the question asks for ("You want to make sure that Bob has your phone number…"). Why do things the hard way?

This question tests something rarer than education – the capacity to ignore what you learned when it isn't helpful. In business there's no one to tell you what part of your education applies (if any). There is an overwhelming temptation to use whatever intellectual tools are at hand and to pat yourself on the back for using the highest-powered tool possible. Google doesn't want people who instinctively do things the hard way because they can. They want those with a knack for intuiting simple solutions that work.

Who's smarter: a computer science PhD or your mum? After years of interviewing at Google, Paul Tyma, an engineer, resolved to find out. Suppose you're given a million sheets of paper (runs one of Tyma's interview riddles). Each is the record of a university student. You're to sort them in order of age (number of years old). How would you do it?

Tyma posed this question to his mother, who knew nothing about computer science. Mrs Tyma's answer was more efficient than those of many of the highly educated applicants Tyma has interviewed. She'd make stacks. Take the first record off the top of the pile and look at the age. If it's a 21-year-old, it goes in the 21-year-olds' stack. If the next record is a 19-year-old, it goes in the 19-year-olds' stack. And so on. You have to look at each record only once, and when you're finished simply collect the stacks in order of ascending age.

That procedure is about 20 times faster than quicksort, the algorithm that many Google applicants suggest. But Mum's stacks happen to be a much better tool for this particular task. Sometimes "creativity" is just common sense.

How would you answer the following? If you had a stack of pennies as tall as the Empire State Building, could you fit them all in one room? This may sucker you into thinking that it is one of those interview questions where you're intended to estimate an absurd quantity. Hold on – the question doesn't ask how many pennies. It asks, will the stack fit in a room? The interviewer wants a yes or no answer (with explanation, of course). That should be a clue, as should the fact that the question doesn't say how big the room is. Rooms come in all sizes. Intuition might suggest that the stack wouldn't fit in a phone booth but would fit easily in the Hall of Mirrors at Versailles. The answer is roughly this: The Empire State Building is about 100 storeys tall [it's 102 exactly]. That's at least 100 times taller than an ordinary room, measured from the inside. I'd have to break the skyscraper-high column of pennies into about 100 floor-to-ceiling-high columns. The question then becomes, can I fit about 100 floor-to-ceiling penny columns in a room? Easily! That's only a 10 x 10 array of penny columns. As long as there's space to set 100 pennies flat on the floor, there's room. An old-style phone booth has room.

Swagger counts. The goal is not just to get the right answer, but to make it look easy. Great athletes do this naturally. Lately, job seekers are expected to do the same.

• This is an edited extract from Are You Smart Enough To Work At Google? by William Poundstone, published by Oneworld at £12.99. To order a copy for £10.39, including UK mainland p&p, go to guardian.co.uk/bookshop or call 0330 333 6846.