Back in the 1960s, when New York was the centre of street photography, the main practitioners of the form would sometimes cross paths. Lee Friedlander was friends with Garry Winogrand who often met Joel Meyerowitz as they crisscrossed Manhattan and beyond on the prowl for pictures that caught the city's tempo, its myriad everyday dramas, and its citizens at work and at play.

In terms of personality, Winogrand was easily the most aggressive. Friedlander later said of him, only half-joking, "He was a bull of a man and the world was his china shop." Meyerowitz later recalled how Winogrand "set a tempo on the street so strong that it was impossible not to follow it. It was like jazz. You just had to get in the same groove."

More than 40 years later, Winogrand, Friedlander and Meyerowitz are still setting the groove for street photography, as key influences on a generation that has rediscovered and is busy reinventing the form. Though street photography is almost as old as photography itself, and many of the great pioneers of photography – Eugène Atget, Brassai, André Kertéz, Jacques Henri Lartigue, Henri Cartier-Bresson, Walker Evans and Robert Frank – could all be considered street photographers of one kind or another, the term as it is now used denotes a genre – and an attitude – that the New York photographers of the 60s and 70s did much to define. The attitude might be best summed up by one of Winogrand's many singular descriptions his methodology.

"When I'm photographing, I see life," he once said. "That's what I deal with. I don't have pictures in my head… I don't worry about how the picture is going to look. I let that take care of itself… It's not about making a nice picture. That anyone can do."

Winogrand also said: "When things move, I get interested," which gets close to the instinct underpinning s treet photography: the desire to capture for a split-second the city's unending, ever-changing momentum in all its everyday oddness. Though he photographed in Los Angeles for a few years, Winogrand was essentially a New York photographer: frenetic, in-your-face, arty despite himself. In his book The Ongoing Moment, the novelist and critic Geoff Dyer sums up Winogrand's style. "The kinetic force of the city met his own '1200 ASA jitteriness' head on. The photos are jostled by what they depict. A kind of horizontal vertigo holds sway. The pictures are tilted, skewed, unsteady. There is nowhere for our gaze to rest because, in these pictures, nothing is at rest – least of all Winogrand himself. He is a still photographer only in the strictly technical sense of the word."

Winogrand photographed relentlessly. When he died in 1984, he left behind not just a wealth of images that are a testament to his impatient vision but also thousands of rolls of unprocessed film. In the end, his obsession had become a kind of mania. He was not searching, like Henri Cartier-Bresson before him, for the "decisive moment" when form and content, vision and composition merged into a transcendent whole. Instead, he was continuously chasing after the eternal nowness of life itself in all its raw, unmediated energy. That is what most street photographers hope to capture when they walk out into the city.

Winogrand remains a hugely influential figure, but it would be difficult to take pictures on the street now the way he did then. And, it would be a brave photographer indeed who would try and take photographs the way Bruce Gilden did in the 80s and early 90s, using confrontation as a kind of aesthetic. Gilden often used flash to surprise his subjects and to, as he put, it, "energise the frame". He was the epitome of the in-your-face street photographer. Today, on the more fearful and aggressive streets of London, these kinds of approaches would, before long, get you arrested or beaten up.

It would be even more difficult to take street photographs the way the more gentle practitioners of the form did. Both Britain's Roger Mayne, working in the 1950s and 1960s, and America's Helen Levitt, who famously began shooting in colour in New York in the early 60s, often photographed children at play in the streets and never thought twice about it. Neither did the children's parents or guardians. That is not the case any more. We live in an age of anxieties, both big and small, real and imagined.

Today, photography – and street photography in particular – is a contested sphere in which all our collective anxieties converge: terrorism, paedophilia, intrusion, surveillance. We insist on the right to privacy and, simultaneously, snap anything and everyone we see and everything we do – in public and in private – on mobile phones and digital cameras.

In one way, then, we are all street photographers now, but we are also the most-photographed and filmed global population ever. In Britain, our city centres and public buildings are monitored 24/7 by surveillance cameras, while security cameras track us in car parks, supermarkets, football stadiums, hotels and as we enter and leave our places of work. Last year, Google Street View arrived on the streets of Britain and photographed nearly 23,000 miles of road on roving panoramic cameras: mass observation on the kind of scale unimaginable a few decades ago. And, depending on where you stand, the most democratic or the most Orwellian kind of street photography yet invented.

Back in 2008, Home Office minister Tony McNulty, responding to a query from a photographer, wrote: "There is no legal restriction on photography in public places, and there is no presumption of privacy for individuals in a public place. It is for the Chief Constable to ensure that officers and police community support officers are acting appropriately with regards to photography in public places, and any queries regarding this should be addressed to the Chief Constable. However decisions may be made locally to restrict photography, for example to protect children. Any questions on such local decisions should also be addressed to the force concerned."

Of late, though, the police have been stopping and questioning, and, in some cases, detaining, photographers on the street with alarming regularity, using – some would say, misusing – the powers given to them under Section 44 of the Prevention of Terrorism Act. Nearly every photographer I spoke to for this article had tales of being stopped and questioned by the police, not just near government buildings, but all over Britain. Over the past year or so, there has been a constant stream of stories in the media about photographers being cautioned for photographing seemingly innocuous sites.

In January, an estimated 2,000 photographers gathered in Trafalgar Square for a protest against police harassment organised by an organisation called, I'm a Photographer, Not a Terrorist. It seems to have worked to a degree and many photographers I spoke to last week said they had noticed a decrease in the use of heavy-handed, over-vigilant tactics by the police.

To be a street photographer today, you need, as Martin Parr recently put it, "obsession, dedication and balls". And yet more people than ever seem drawn to street photography. On Flickr, there is a site called HCSP – Hardcore Street Photography – that has nearly 36,000 members. Nevertheless, as a genre it remains almost ignored by galleries and curators who are drawn to the more postmodern thrust of conceptually driven art photography. Paul Graham, one of the few whose work has made it from the street into the gallery – last year he won photography's prestigious Deutsche Börse prize – recently responded to a critic who dismissed photographers who specialised in "just snapping their surroundings". On Americansuburbx.com he wrote, "...there remains a sizeable part of the art world that simply does not get photography. They get artists who use photography to illustrate their ideas, installations, performances and concepts, who deploy the medium as one of a range of artistic strategies to complete their work. But photography for and of itself – photographs taken from the world as it is – are misunderstood as a collection of random observations and lucky moments, or muddled up with photojournalism, or tarred with a semi-derogatory 'documentary' tag."

Stephen McLaren, a London-based street photographer who has co-authored with Sophie Howarth a forthcoming book called Street Photography Now, agrees. "Street photography in Britain is going though a very vibrant period, but it has a problem with the fine art world. You won't often see it in the galleries. It's almost as if it's seen as too street-level, too authentic in some way."

Another sphotographer Matt Stuart tells me: "People say street photography is somehow old-fashioned and cliched, but, if that's the case, so is portraiture or sports photography; you might even say so is photography itself. Sure, we're recording the everyday world in much the same way that street photographers have always done, but times change and things move on, and street photography is a record of that at ground level. That is why it is so important to resist calls for it to be banned or controlled."

When I ask McLaren and Stuart to define street photography, they both tend towards the purist approach. "It's essentially a way of working wherein you have to be utterly open to what happens on the street," says McLaren, "So, no props, no models, no setting up of shots, and you always use available light. Then, it's down to a mixture of happenstance, luck and skill."

Stuart is even more rigorous . "I'd agree with all that but I'd also add that there should be no post-production like Photoshop or whatever and, just as importantly, no pre-production. In fact, I never make any form of communication with anyone I am photographing before I take a picture. You are looking in at something without being a part of it. That's very important. I try to be as invisible as possible, in fact, but I will explain what I'm doing after the event if people ask me."

Both of them agree that to be a good street photographer you need to be, as Stuart puts it, "incredibly patient and dedicated to the point of obsessive". Of the 10,000 or so photographs he has taken in the past two years, around 50 have made it onto his website.

"Meyerowitz and those guys set the bar incredibly high," he explains, "and I always feel they are looking over my shoulder when I am editing, as well as when I am shooting. You have to be ultra-selective. You have to be your own harshest critic."

For all the purity of vision that today's dominant strain of street photographers adhere to, there are those who straddle the street and the gallery. Alongside Paul Graham, there is the American Philip-Lorca diCorcia who actively disrupted the usual process of the genre when, in the 1990s, he photographed passers-by as they unknowingly walked under, and activated, a set of flashing lights he had set up beforehand on an overhead scaffold. The end result looks both intimate and unreal, as people are captured – and brightly illuminated – lost in their thoughts and daydreams. Here, the process is as important as the end result, which, intriguingly, is as close as street photography has come to studio portraiture.

Ironically, diCorcia was sued by an Orthodox Jewish man whom he had photographed in this way. The man claimed his privacy and religious rights had been violated when the photograph was exhibited in a gallery. The judge dismissed the case on the grounds that the photograph was art, not commerce.

The British photographer Stephen Gill operates in his characteristically understated way in the hinterland between street and conceptual photography. His 2005 book, Hackney Wick, comprises photographs of a vast, sprawling and busy flea market that sprang up a few years ago on the site of the old Hackney greyhound stadium. Gill's photographs were shot on a disposable plastic camera he bought from one of the stallholders and the art book was published on his own imprint, Nobody. Though he photographs on the street, and could be said to be pushing the boundaries of street photography, and for his Hackney pictures he "shot obsessively, three days a week, for six years", he is uneasy with the term.

"I'm not really drawn to a lot of the ideas around street photography, the notion of waiting around for something to happen, an accident or a man walking by an odd sign or things like that," he says. "What I do is always carried by the subject. In this instance, I was totally reacting to a place that I had stumbled on, and the place completely moulded and shaped the work. I was reacting, really, rather than going out looking."

In other ways, too, Gill is the antithesis of the traditional street photographer. "I do think photographers have a social responsibility, particularly when there is such a conflict between wanting our privacy and wanting to photograph everything. So, I try to be very clear about what I am doing, to myself, and the people I am photographing. I kind of mingle with the camera, I don't sneak around shooting without trying to be seen. I tell them what I am doing and that it might appear in a magazine or a book or whatever and people tend to be OK with that on the whole. It's a gamble but it usually pays off."

Gill, though, is the exception. Most great street photography is great precisely because it operates on the borderland between intrusion and observation. Even more problematic is the tradition of clandestine photography. The great Walker Evans took a whole series of provocative photographs with a concealed camera on the New York subway. Their power resides to a great degree in their voyeurism. As a street photographer, you walk the line in more ways than one.

"You are walking out into the world to see what happens. There's no agenda," says Matt Stuart. But with photography, there is always an agenda. The lure of street photography is more likely connected to its democratic thrust, and to the – usually mistaken – belief that it is the one kind of photography we can all do.

The evidence would suggest otherwise. "You are looking for that brilliant moment that 99% of the time you don't get," says Stuart, "That is exactly why it is both the most accessible and the most difficult kind of photography."