When I picked up my first fare in Covent Garden last month, I couldn't even open the passenger doors. I had two gentlemen fresh from dinner in Langley Street, and I was panicking, pressing all the buttons I could find, fumbling with keys. Nonetheless they were delighted when I told them they were my first, and that consequently the ride – as goes the cabbie tradition – was free, to wherever they wanted to go. Clapham Junction, as it happened, though they might as well have said Tuxedo Junction as far as my frayed nerves were concerned.

When I started learning the Knowledge of London in October 2008, the examiner told us it was the hardest thing we would ever do. He wasn't exaggerating. There are no dropout figures, but each year Transport for London (TFL)usually licenses between a quarter and a third of the number of applicants, so we can safely say that most who start the Knowledge never finish it. The average "Knowledge Boy" (or, occasionally, Girl) spends three or four years covering around 20,000 miles within a six-mile radius of Charing Cross, out on their moped come rain, freezing wind, or traffic chaos. Hundreds of hours are spent drawing lines on laminated maps of the city, working out the most direct route from hotel to station, restaurant to office, monument to square. We learn thousands of "points of interest", taking in around 25,000 streets. And we don't just sit an exam, we have a potentially endless series of "appearances", in which we recite the perfect route between any two points in the city, until the examiners think we're good enough. I had 19 of them over a period of 18 months – many candidates have more.

The classic black FX4 Fairway taxi cab. Photograph: Martin Argles for the Guardian

The terror of the appearance is legendary – ask any cabbie and watch him wince. In the "olden days", John Mason – head of taxi and private hire at TfL – tells me, the examiners would play games such as putting the chair in the examination room facing the wrong direction, and give an automatic fail to the student who dared turn it around. "You can't do that in the modern age," he says. "I don't think it was acceptable in years gone by, to be frank." Dean Warrington runs the WizAnn Knowledge school that I attended (there are several schools around London, unaffiliated with TfL) and got his green badge – the licence that allows drivers to operate across Greater London – in 1996. He remembers one examiner who would choose the start and end points of his questions by throwing two darts into a map, and if the student felt this was unfair he would offer to let them throw the darts instead. "He was a crazy bloke." Nowadays the Knowledge seeks to be "as efficient, fair and transparent as possible," says Mason. Well, you can't deny the industry needs to modernise.

And it is doing so. You may not have noticed, but the classic cab is being retired. With a career broadly parallel to the much-mourned Routemaster bus, the Austin FX4 was introduced in 1958, updated and rebranded the Fairway in 1987, and – owing to air-quality targets set by the mayor of London's office meaning that vehicles more than 15 years old will no longer be licensed – is set to all but disappear from the streets of the capital in the new year. A handful have been granted extensions and will remain as a rare sight for a few months, but the Fairway has reached the end of the road. Meanwhile, hundreds of its successor – the bubblier TX (drivers often refer to it as the "Noddy cab") – have been recalled with mechanical problems and, as you may have read, its manufacturer Manganese Bronze has gone into administration. Mercedes (already licensed) and Nissan (probably due in 2013) are offering fuel-efficient people carrier style alternatives that appeal to the cost-conscious driver. For those of us who care what the London taxi looks like, these are worrying times, although Mason assures me TfL is consulting with other manufacturers with the hope of finding a distinctive London equivalent.

But it's not just the vehicles that are changing. In one of the most diverse cities in the world, 93% of black-cab drivers are currently white men. This won't be the case for much longer – the numbers of Bangladeshi and north African Knowledge Boys are rising fast. The take-up was 35% non-white last year, according to TfL's figures, so you can expect to see a real change in the face of the London cabbie in the next few years.

The new Mercedez-Benz people carrier-style London taxi cab. Photograph: Ray Tang/Rex Features

And then there's technology. Why do we need the Knowledge anyway when we have machines? Mason is emphatic: "London taxi service means that you can hail a cab, on the street, and before even the driver's really understood exactly where the end point is, he's set off and he's on his way in the right direction." Crucially for TfL, this means no sitting at the side of the road typing postcodes into computers while traffic builds up behind you: "I've no doubt whatsoever that if there was no Knowledge requirement in London today there'd be a massive impact on bus reliability, on journey times, the whole lot."

Technology can be our friend too, in the form of new mobile apps such as Hailo that allow a driver to be hired unseen by anyone with a smartphone. James Grant is the epitome of the 21st-century cabbie. Smartly dressed, he's also a jobbing actor, and a Hailo enthusiast. Like too few cabbies, he also takes credit cards, and like many – occasionally and unashamedly – uses an AtoZ Cabbie's Mate satnav. Designed specifically for the black-cab driver, "it doesn't say left here, right there", but simply puts a line on a map to show you the general direction if you get stuck. "If you hadn't spent three years studying it wouldn't help you," he says. And no cabbie, ever, can expect to know absolutely everything. Fairway driver Lionel May tells me, "You never stop learning", and he turns 80 in January. Will he be sad to hand his Fairway back to the garage? "Most definitely. It goes like a bomb." Everybody I speak to about the Fairway raves about its super-reliable engine that can do well over 500,000 miles without any problems.

Not that that's of much concern to the average passenger who only sits in the back for a few minutes at a time. Aesthetics might be though, and it's a gorgeous car. Warrington practically goes misty-eyed when he reminisces about his "little Fairway … I adored the way it looked, felt … they were lovely to see." This was a big part of the attraction for some of us Knowledge Boys – not just the car, but the whole beautiful, archaic industry, first licensed by Oliver Cromwell, idiosyncrasies and all. Fellow new driver (we're known as "butter boys" in the trade, because we take the bread and butter from the mouths of established drivers' families) Andrew Baker eulogises this: "I've always recognised the job as being something a little bit special, a little bit unusual, a little bit out of the ordinary." He tells me that when he moved to London to do his degree it was really an excuse to get to the city he'd always wanted to live in, and that it was "romance" that attracted him to the Knowledge, leaving the degree behind. I too was drawn to it in my mid-30s. In love with London and in a career rut, I saw an opportunity to become a working part of this magical town.

A Hansom cab, in Buckingham Palace Road, London, in 1933. Photograph: Popperfoto/Getty Images

I rent my cab from a garage run by the deputy master of the Worshipful Company of Hackney Carriage Drivers, Eddie Crossley (he did his one-year term as master last year). Crossley's great-great-grandfather was a horse-drawn-bus driver before progressing to the Hansom cab, and the business has been in his family ever since. Nearing retirement, but with his son in the office and on the Knowledge, Crossley is cautiously optimistic. "There are three things that have kept us distinct from private hire," he says: "A purpose-built vehicle, the turning circle [London cabs, as legend has it, turn on a sixpence – or 25ft] and the Knowledge." Well, the purpose-built vehicle is in peril, and 65,000 private hire vehicles have shown that you can get by without the turning circle if you really must, but it is "the Knowledge that allows us to ply for hire on the road". Eddie's garage has been in its current location in Hackney for around a century, and he hopes it will remain there for "a good few years yet".

So do I. The classic cab's 54 years as an icon may seem a good innings, but taxis have been in London in one form or another for well over 300. And TfL is still optimistic that a new purpose-built vehicle might be found.

As my old teacher Dean puts it: "Let's stay the pride of the world's taxi drivers – why not?"