To cycle in London, where I live, is to test the ability of the brain to process multiple and dynamic sources of information. It requires continuous assessments of risk and calculations of intersecting vectors. You have to assess the likelihood that a bus or truck might turn left across your path, which is the one of the most common causes of death for cyclists, and compute the varying speeds of cars, taxis, mobility scooters, motorbikes, rickshaws and fellow bike riders.

Among the latter, you judge whether someone is an unstable, first-time Boris-bike user. You assess whether each is a hard-shell or a soft-shell cyclist, a head-down racer carapaced in high-visibility and low wind-resistance, or someone more interested in an amiable amble. You note pedestrians and the chance that they might be tourists who have forgotten about the British travelling on the left and could therefore step off the pavement unannounced. You consider the possibility of doors opening on parked cars. You try to decipher the inscrutable instructions of cycle lanes.

In recent years, there has been a palpable change in the balance of power. Cycling has more than doubled in London in 10 years and through sheer weight of numbers bikes oblige cars to take more notice of them, to proceed gingerly, to give more room, to adjust their pace. The uncertain skills of some cyclists are also a factor, as wobblers make drivers more wary. The cumulative effect, on a summer evening's rush hour, is a swarming of two-wheeled traffic. There is some tension and aggression, as there always is in the competition for road space, but the atmosphere can be almost festive and mildly exhilarating. There is a variety of behaviour too, from fast, long-distance journeys from outer suburbs to the intricate negotiations of different users in small central streets.

The London swarms are one symptom of the way that, patchily but distinctively, British cities are growing a new culture of bikes. In Brighton, Bristol, Manchester, Newcastle and Sheffield, rates increased by between 80% and 109% in the first decade of the century (even though, in less metropolitan areas, rates stood still or fell). This culture is individualistic and entrepreneurial, occupying many niches. It includes cafes that embrace both cycling and some more-or-less well-defined social purpose, esoteric magazines, specialised crafts and industries, festivals, tours, clubs and rallies. Its manifestations include such things as the Dunwich Dynamo, in which hundreds turn up, on the Saturday in July nearest to a full moon, to ride through the night a 120-mile route from east London to the Suffolk coast. There is a society for watching films with bicycle-related themes. There is the still unreal fact that Britain cannot only scoop armfuls of Olympic cycling medals, but also provide two different winners, in successive years, of the Tour de France. This year the race will start in the Yorkshire Dales.

Cycling supports manufacturing industry of a kind often thought to have disappeared in Britain. In Brentford, Middlesex, there is a factory reached by a ride under a concrete flyover, scented with diesel and spring flowers, in which 230 skilled people make Bromptons, casting parts, brazing the elements together, smoothing their surfaces with rumbling machines and checking their alignment with measuring devices. The frames then go to Wales to be painted and back again. Then, passed around an informal-looking loop of standing people, the elements become in the space of 38 minutes bikes, each worker adding gears, brakes, wheels and other parts according to an individual specification requested by each purchaser. Brompton's sales are going up 25% a year; it currently makes 120 a day. It once tried to shift production to Taiwan, but it didn't work.

The Brompton bicycle factory in west London. Photograph: Luke MacGregor/Reuters

British bike culture runs a social gamut. There is, for example, the Bristol Bike Project, a charity that aims to help disadvantaged people, including refugees, the homeless and recovering substance abusers, by acquiring unwanted bikes: the beneficiaries learn how to recondition them, and then get to keep them, meaning they can acquire new skills and increase their mobility. It is hard to think of a more admirable set of objectives.

There is also Rapha, the brand that sells high-performance jerseys for up to £195, and embrocations and lotions packaged like beauty products. Its stylish website artfully combines nostalgic images of the weatherbeaten faces of old racers with new technology and claims of "superb functionality". Rapha's shop in Soho (and it has others in Osaka, Sydney, New York and San Francisco) presents its products like objects in an art gallery, or at least a fashion store. It is not just a place to buy things, but also a club that organises festivals and tours. The shop includes a cafe of calibrated chillness, where you can watch non-stop cycling television over your yimu oolong tea and cardamom bun.

Bikes come with calibrations of status, captured by the BBC's comedy series about itself, W1A, in which the haplessness of Hugh Bonneville's head of values is signalled by the fact that he can't fold his Brompton and that his high-visibility jacket is in traffic warden's yellow-green. The effortless superiority of his nemesis, the director of strategic governance, is apparent in his classier red jacket, the deft flicks with which he manipulates his folding bike and the fact that it is a raw lacquer version of Bonneville's: it has a clear finish rather than paint, the better to reveal the beauty of its brazed joints.

As cars get more like one another, and more boring, bikes are acquiring nuances and variations, of which the ultimate example is the personalised, made-to-measure, handmade frame. Finding this concept mysterious but intriguing – how much difference can it really make to have a machine's dimensions exactly tailored to yours? – I take myself to a light industrial zone in Camberwell, south-east London, the home of Saffron Frameworks. Here, in a small workshop, with tools calmly arrayed on the walls, Matthew Sowter makes frames for prices starting at £1,250 each, plus £300 for paint, plus further sums for the wheels, pedals, chain, brakes and all the rest. A total cost of more than £10,000 is likely.

I am taken into a world of craft, of precise choices of stiffness, of the diameters of tubes and the thickness of steel, of material added or taken away to match the degree of stress, of specially treated stainless steel, invented a few years ago, which overcomes that material's usual lack of tensile strength. Silver is used in soldering because, melting at a low temperature, it flows better and does a neater job. Polishing is done in seven stages, using different grades of sandpaper. Everything is directed to matching your measurements, taken by visiting a fitment specialist, and your desires, established in preliminary consultations. Sowter was formerly a chef, as well as a graphic designer and a racing cyclist, and you can see something of all these vocations in the job he does now.

A cycle superhighway in south London. Photograph: Linda Nylind for the Guardian

Saffron's clients are not all people with money to burn, but also those who have yearned for something handbuilt for years and now want "a bike for life". (And "the really nice thing" about these steel frames, says Sowter, is that they last for 60 or 70 years). He says the attraction is not only acquiring a machine that fits you and suits you, but also "getting involved in the process", that your life and the life of your bike intertwine from the latter's conception. "I never get anyone saying, 'I want to impress my friends'," he says, even if this motivation is rarely completely lacking, "but people put it across in subtle terms."

The Saffron workshop offers a glimpse into the attractions of the cults of cycling to its devotees. It is about a close relationship between the human and the physical, or material, of a kind that in a world of disposable products is rare. It is about rhythms – if riding a bike involves speed, and often the more the better, making one by hand is slow. Saffron produces about four a month and has a six-month waiting list. There seems to be a peacefulness that comes from this close attention to a specific range of skills, and seclusion in this esoteric knowledge. At the same time, these skills and knowledge form the basis of a community with others of the same mind. Sowter knows the very few others in the same business and they are supportive of one another, sharing knowledge, not rivalrous. He is working on establishing a frame-builders' guild.

Cyclophilia offers a particular way of connecting with the world and I encounter a similar range when I meet Mike White, a co-founder (with James Lucas, who also created the Bristol Bike Project) of the quarterly Boneshaker magazine. A man with a pointed beard and inseparable beanie, it is important to him that his perfect-bound, matt-paper publication is printed rather than online. "We all like the tactile, physical side," he says of cyclists, "not neutered by glass between you and what you're experiencing. In a car, you're compartmentalised. On a bike, you feel it all – smell, humidity, temperature."

Boneshaker started because Lucas "kept hearing great stories that no cycling magazine would take". It prides itself on having "no training tips, race diets or adverts". It is, says White, "very collaborative. We do it for the love. We get stories from people all over the world. We might publish some crazy, overenthusiastic workshop guy from Berlin or Bratislava just pouring out the words." It is lightly edited, such that the imperfect but authentic language of the contributors remains on the page.

He is telling me this in Roll for the Soul, a not-for-profit cafe in Bristol that is also a cycle shop and workshop (which, having collected a puncture on my way to Paddington, and then pushed my bike from Bristol Temple Meads to the cafe, I get the opportunity to use.) It is a non-elitist Rapha. "We think of it as a village hall," goes the blurb, "slap-bang in the middle of Bristol", where the aim is to "make things happen and reclaim a bit of the city centre from corporate shiny stuff". The idea is that cycling is not just a means of transport, a leisure pursuit, or even a social activity, but a force for change, a micro-utopia.

If much of cycling culture is individual, bottom-up, localised, entrepreneurial or insurrectionary, that is partly because government attempts to promote cycling have been erratic. The intention has been there for years, and there have been some achievements, but the task of retrofitting British cities for cycling, with too little money and too much opposition, has led to laughable cycle lanes becoming the stuff of websites and subjects for slots in Metro otherwise occupied by crazy pets. There are the lanes that start and then stop within a few feet, or point cyclists at people queuing for a bus, or tip them into the path of a truck, or guide them into the trunk of a tree that, in the abstract universe inhabited by whoever planned the lane, presumably isn't there. They have become emblems of you-couldn't-make-it-up official absurdity, environmental correctness gone mad, box-ticking bureaucracy.

Weird cycle lanes: the please-hit-tree, Grimsby.

See, for example, the pictures at anorak.co.uk or weirdcyclelanes.co.uk, which celebrates the bike infrastructure of Brighton and Hove.

To experience the ambitions and limitations of public cycling policy, you can sample CS2, one of Boris Johnson's trumpeted cycle superhighways. This is also the A11/A118, or Whitechapel Road/Mile End Road/Bow Road/Stratford High Street. It was called for a while "High Street 2012" in reference to the fact that it was the main route from the centre of London to the Olympic site and that part of the marathon was going to be run there, until it was rerouted to more photogenic locations.

It includes two of the more lethal spots in the capital, an area in Aldgate and Bow roundabout, where three people have been killed since 2011. Last November, in a sudden spate of five deaths in London in nine days, three were on CS2. Inquests have found that the infrastructure was not the sole culprit, but one coroner has criticised the blue strips of paint that pass for cycle lanes while giving no legal or physical protection and called the superhighway "an accident waiting to happen".

And, indeed, it is not a comfortable ride. At the end nearest the City, there are only the "ghost lanes", also called "virtual lanes" or "blue surface treatment", the notional strips that as often as not are occupied by the nearside wheels of double-decker buses. In jams, cyclists take to the pavements to get around them. The immobile white-painted bikes and bunches of flowers that commemorate fatalities don't add to peace of mind and, approaching the Bow roundabout, it is hard not to dwell on the casualties there. Eventually, there is a new lane, properly separated, but which, as at least one blogger has pointed out, has a number of odd and dangerous design quirks. The lane then expires with a convoluted sign that asks cyclists to invade pedestrians' space, then project themselves into bus lanes. This is now Stratford, home of the Olympics, but if you want to get to the velodrome, cathedral of the new bike nation, you have to weave unprotected along what are in effect the service roads of the Westfield shopping centre.

CS2 is in truth an abysmal realisation of Boris Johnson's pledge "to turn London into a cyclised city – a civilised city where people can ride their bikes safely and easily in a pleasant environment". It is a long way from being Denmark or the Netherlands, the two countries repeatedly cited as models for cycling infrastructure. In Groningen, they have multistorey bicycle parks. In Copenhagen, about 30% of journeys to work are made by bike, compared with the average in England and Wales of 2.9%, along an immaculate network of cycle paths. The cycling style is serene, mostly helmetless, confident that both traffic engineers and vehicles will take care of your wellbeing. It is nothing like the tense, defensive-aggressive London cyclist, armoured with high-visibility accoutrements. Maybe this is why the UN has decided that Denmark is the happiest country on Earth and Monocle magazine has called Copenhagen the most livable city.

Nor is Britain much like France, nor even the United States. Peter Murray, chairman of the independent organisation New London Architecture, knows a lot about cycling in both countries: just turned 70, he leads a ride of architects and property developers from London to the MIPIM property fair in Cannes every March, and last year cycled in a group from Portland, Oregon, to Portland Place, London, with the help of an aircraft to cross the Atlantic.

In France, he says, "there is an inbred respect". When his peloton of property people goes by, "lorry drivers go to the other side of the road, and people clap and go 'allez, allez'". In the US, perhaps more surprisingly in a petrolhead society, there "was amazing consideration". Truck drivers are "professional – they act as if they know they're in control of a deadly object". He portrays Britain as a depressingly aggressive place where, when drivers see a group of cyclists, "they just put their hands on the horn". (Sowter, however, is from South Africa and says that attitudes are worse there.)

According to Murray, cities such as Portland and Minneapolis have exemplary provision. New York, thanks to Janette Sadik-Khan, who was transportation commissioner under former mayor Michael Bloomberg, has more sophisticated and successful arrangements than London. It has, in Transport Alternatives, a 40-year-old lobbying group that works co-operatively with government. The politics are more confrontational in London, says Murray. Britain, he says, should try less to imitate Denmark and the Netherlands, where "the culture is just so different", and instead try to follow the US.

In Britain, the most bike-friendly large city (that is, not counting places such as Cambridge) is Bristol. It was, at least, nominated as such by a magazine called Cycling Plus. It has its bottom-up culture, of the Bristol Bike Project, Boneshaker, and Roll for the Soul, and some undramatic but effective alterations to the roads have made the centre much more relaxed than London. In the 1970s, it pioneered cycle paths when the engineer John Grimshaw put together a group that converted a disused railway track to Bath, which remains delightful and popular. Grimshaw's group became Sustrans, which is now a national charity campaigning for and achieving what it calls "healthier, cleaner and cheaper journeys" and "better, safer places to live in".

Since last year, Bristol also has a directly elected mayor, George Ferguson, an architect of no political affiliation, who was one of the founders of Sustrans, and whose preferred transport is now an electrically assisted bike. He says he "wants to change the culture where we have this constant fight between different modes of transport". He has brought in residents' parking schemes to deter the commuters who drive into the city and park in its streets and he is introducing a 20mph limit. He is pushing forward an east-to-west waterside cycle route across the city. "Every £10m you spend on cycling is so much better value than in any other form of transport," he says. "It's about safe routes to schools, environment and health, and air quality, all for a fraction of what you would spend on an equivalent."

Ferguson's plans are not universally popular. A traders' association drove a tank into the city to protest at the damage, as they saw it, which residents' parking schemes would do to their business. "There will always be people," says Ferguson, "who see restrictions on the use of a car as an attack on their liberty, but any civilised European city has them." Such things "are seen as infringements on freedom until they happen and then they are accepted".

He knows that Bristol is some way off matching other countries; if its cycling rates in the city are quite high, about three times the national average, that is still not very high compared with other European cities. "If we are the most bike-friendly British city," he says, "then British cities are not very bike-friendly." He currently has £11m to spend on his plans, but "I could spend another £100m very easily". But he says that Bristol "is a brilliant city to be a testbed, because it's not easy". It has hills and rain. It lacks the straight continental boulevards that lend themselves to separated lanes. "If you can make it work in Bristol," says Ferguson, "you can make it work anywhere."

George Ferguson, Bristol's mayor, is trying to improve an already bicycle-friendly city. Photograph: Chris Bahn

Like other British cities, it has some catching up to do. Jon Usher, of Sustrans, says that it is still suffering from the effects of a motorway that, along with dual carriageways and roundabouts, was pushed into the city in the 60s and 70s, at a time when those fabled Dutch and Danes were already installing cycling infrastructure. In the Netherlands, he says, they have been spending £16 per person per year, for 30 years, on improvements. Britain has more than a half-century to make up; not only has the country not invested in the same way, and created some hostile traffic engineering instead, but it has also developed a mentality about cars "which is really difficult to unpick". The communities secretary, Eric Pickles, for example, has called on town halls to ditch "their anti-car dogma", make town centres more accessible for motor vehicles, and undo their traffic calming measures.

Cycling, in fact, embodies the weaknesses and strengths of British civic life. It is a palpably good thing if more people cycle, for health, for the environment, for the safety of pedestrians. It is a cheap, reliable and effective form of transport and one of the quickest ways to get about cities. Bikes use less space than cars. Cycling encourages interaction, as it is easier to stop a bike and chat to someone than it is a car. Despite the fears of traders, the evidence of New York and other cities is that bicycle provision is good for business: a cyclist is likely to visit a series of small shops, but a driver is more likely to go to a supermarket car park and load up with a week's worth of stuff. Yet neither the funding nor the organisation is there to make these obviously good things happen with any great speed.

The debate is also conducted in adversarial terms, by both sides. It is seen as a zero-sum game in which what is good for cyclists is seen as bad for drivers, despite the fact that many people are both and despite the obvious truth that, each time someone switches from a car to a bike, they cause less congestion. Perhaps Pickles has a point that shops in market towns might benefit from easier car parking, but his vision doesn't seem to include the role that other forms of transport might play. It sounds more like a Faragist attempt to win cheap cheers. There is almost no political will to take on the essential task of balancing the legitimate concerns of all.

With government haphazard, change comes more from private actions. In London, mayoral policies have contributed, in particular the cycle hire scheme that was initiated by Ken Livingstone and enacted under Boris Johnson, but if the city's streets feel very different from the way they did a decade ago, that is largely because many people simply decided to start cycling. Murray observes that a big change happened after the tube bombings in 2005: "Suddenly, there were a whole lot more people on the road, some of them not very experienced. Cars were obliged to take notice."

Initiatives come from all over the place, from an untypical politician such as Ferguson, and from pressure groups. The architectural practice of Norman Foster, himself a fanatical cyclist, has proposed a 135-mile network of cycling highways built above commuter railway lines. The Danish expert on urban design Jan Gehl tweeted that Foster's vision was not proper cycling, on the basis that it focused on the function of transport and nothing else. In measured and consistent Denmark, it isn't, but as one among London's multiple modes it possibly makes more sense.

Myself, I am a functional cyclist. I use a bike to get around, as well as a car, public transport and taxis, and choose whichever seems best at a given moment. Sometimes, especially in bad weather, I have to persuade myself into the saddle, but more often regret it when my driving self wins the argument. I enjoy the freedoms and the exercise, but not the underlying tension that comes from knowing that every wrong decision has the potential, however slight, to be fatal. I have suffered one theft and maybe five accidents, one requiring attention in outpatients, none hospitalisation.

I care enough to have a bike a notch above the basic, but the devotion of true enthusiasts is mysterious to me. I ask Sowter, who tells me that "what gives me joy is the object that I am riding. It is the exercise and the endorphins, and the fact that every pedal stroke that comes from you equates into a movement. There's a lot of self-satisfaction in that." It is also about contact with nature: "It's great way to be out, to be in the countryside, to go down country lanes."

For some, it is about extremes of personal physical achievement, such as the Cent Cols Challenge, the "ultimate endurance test", in which 100 summits are tackled in 10 days. Philip Deeker, who organises the challenges with the support of Rapha, has spoken of the "art of suffering". He also said that "to discover that preconceived notions about one's physical limits are mere perceptions that can be banished by unexpected achievement is an extremely liberating experience'. This, he believes, is "the real 'soul' of cycling".

Cycling is both solipsistic and social. It offers a chance to inhabit a private universe, in which the machine becomes a flattering extension of your body and mind, but it is also the means to connect with like-minded others and interact more fully with the life of a city. At its worst, it creates a toxic mixture of aggression, entitlement and self-righteousness. Cyclists can be as competitive and status-conscious as any Porsche driver with, thanks to their green credentials, added sanctimony. Cycling culture can be overly male. It can be narcissistic and humourless. It can come with an understandable but overdeveloped sense of victimhood. Every cycling death is sad and wasteful, but it is striking by how much the greater numbers of pedestrian deaths get less attention.

At its best, cycling encourages a gentle, benign, idealistic frame of mind. It heightens awareness of your surroundings and other people. It can be co-operative and unmaterialistic. It encourages alertness and attention. You are less likely to be a binge drinker or a couch potato if you ride a bike. For these reasons, as well as the benefits to health, environment and so on, it can make cities richer, subtler, more open and generous places in which to live. Who could possibly not want more of this?