It is difficult to overestimate the influence of Robert Frank's classic The Americans on the generation of photographers that came of age after its publication in 1958. One of them was Joel Meyerowitz, who cites Frank as a huge figure in his development – but not in the way one might expect. "Robert Frank is the reason I make photographs," says the veteran street photographer, his streetwise Bronx accent still strong at 74, "but it was not seeing his work that blew me away, but seeing him work."

In 1962, having studied painting and art history, Meyerowitz landed a job as a magazine art director. Overseeing a studio shoot in downtown Manhattan, he had an epiphany. "This photographer was moving around the room, snapping – click, click, click! – and he was cajoling the subjects – these two young girls – to keep moving, too. I was mesmerised. I knew nothing about photography except that it had to be still, but there was this guy moving constantly as he was shooting movement. Robert Frank never said a word to me that day, but he affected me deeply. I walked out of there and I literally saw the world differently. Everywhere I looked, there was movement and there was colour."

Meyerowitz returned to his office, where his boss, a man called Harry Gordon, asked him how the shoot had gone. "I said, 'The shoot was great and, by the way, I'm quitting.'" In an attempt to change his mind, the astonished Gordon produced a copy of The Americans and explained that Frank was a great artist, not just a jobbing commercial photographer. "It didn't work," says Meyerowitz now, laughing. "In fact, it had the opposite effect. Twice, in a short time, Robert Frank had entered my life like a bolt from the blue." In fact, it was Harry Gordon who gave Meyerowitz his first camera. "That's when I went out on the streets, a shy kid with a borrowed Pentax who didn't know anything about photography."

Fifty years on, Phaidon has just published Taking My Time, a sumptuous – and, at £500, expensive – two-volume retrospective of Meyerowitz's work, which also includes a limited-edition print. It is, he says, a kind of visual diary of his life as a photographer. It includes his early street photographs, his more reflective landscapes made on a large-format plate camera in the soft natural light of Cape Cod, his portraits of "redheads" in the same setting as well as his epic chronicle of Ground Zero in the months after the 9/11 attacks. It also includes "some brilliant mistakes and amazing accidents" and those blessed moments when he experienced what he calls "the gasp reflex".

"A lot of what I am looking for is a moment of astonishment," he says. "Those moments of pure consciousness when you involuntarily inhale and say 'Wow!' But I've also included a lot of muddling-along photographs, because they, too, are a part of the journey. It's an honest book, I think, and a democratic one. I wanted to show how I got here and the questions I asked along the way."

The first question he asked himself back in 1962 was, "How do I choose what to photograph?" It's a question that resounds in different, ever more complex ways throughout the book, as he shifts formats and styles, reinventing himself continuously. Back then, though, it was almost unanswerable. "I was overwhelmed. The streets, the intense flow of people, the light changing, the camera that I couldn't quite get to work quickly enough. It just paralysed me. I had to learn to identify what it was exactly I was responding to, and if my response was any good. The only way to do that is to take pictures, print them, look hard at them and discuss them with other people."

In this he was helped immeasurably by his friend and fellow street photographer, the late Tony Ray-Jones, and by John Szarkowski, then the visionary curator of photography at New York's Museum of Modern Art. Meyerowitz memorably describes Szarzowski as "a master of Socratic dialogue". "None of us knew how to talk about photography back then. John created the vocabulary. He invented the language for contemporary photography through his books, but mainly through his shows. And he was a photographer as well as a curator, so we all listened."

Szarkowski saw a glimmer of promise in Meyerowitz's earliest work and included a single photograph by him in a group show, The Photographer's Eye, in 1964. "Not only that," says Meyerowitz, still sounding amazed, "he hung my photograph beside a Robert Frank! Can you imagine what that was like for a kid from the Bronx just starting out?"

In a way, though, Meyerowitz's initial ignorance about photography, its techniques and its traditions, worked in his favour. He did not, he says, even think about whether to use colour film or black and white. "The world was in colour. It was just so obvious to me. I had no idea people were snobbish about colour. To me, black and white just seemed back there, historical."

He recalls a night in New York in 1968, when a young William Eggleston, now regarded as the pioneering master of colour, visited him in his apartment and they spent hours looking at "my hundreds of colour photographs and his little box of black and white prints".

Eggleston only acknowledged the impact of that evening recently and you suspect his long silence remains a kind of sore point for Meyerowitz, who, for all his affability, is still something of the tough Bronx kid at heart. Like many photographers, too, he has an ego the size of the Empire State. When I ask him if has experienced moments of doubt, of crisis about his calling, he simply says "no".

I ask Meyerowitz about the combative, confrontational style of street photography espoused by the likes of fellow New Yorker Bruce Gilden, and he grows visibly angry for the only time in our conversation. "He's a fucking bully. I despise the work, I despise the attitude, he's an aggressive bully and all the pictures look alike because he only has one idea – 'I'm gonna embarrass you, I'm going to humiliate you.' I'm sorry, but no."

We end, inevitably, with the body of work by which he may well be remembered in the public eye. Aftermath: World Trade Center Archive was published in book form in 2006, and is soon to be installed as a permanent exhibition in the museum at Ground Zero. Meyerowitz was the only photographer allowed on to the site immediately after the 9/11 attacks, a testament to his extraordinary willpower and self-belief as much as his skill as a photographer. He was initially granted a worker's pass to enter the site by the NYC parks commissioner, Adrian Benepe, and then, without the knowledge of mayor Giuliani or the police authorities, he was given an official NYPD badge by the detectives he had befriended on the site. "They got what I was doing. Not one of the art galleries or government officials I contacted for help in gaining access to the site got it, but the cops understood it completely."

He started photographing there on 23 September 2001, when the heat from under the ground melted the soles of his boots. "I photographed everything 14 hours a day: the demolition crews, the construction crews, the first-aid crews, the debris removal crews, the intelligence squad, even the security guys who initially tried to keep me off the site." The archive is a work of testimony that will enter not just the history of photography, but history itself.

For him, he says in conclusion, photography has essentially been a series of questions he has tried to find a way to answer. "It's me asking myself: 'How interesting is this medium? And how interesting can I make it for me? And, by the way, who the fuck am I?'" Has he found a definitive answer to that one? "No, not yet," he says, smiling, "and time is running out. But I'm getting there."

This article was edited on 12 November to remove a misconstrued quotation.

• This article was amended on 13 November 2012. The original said that Robert Frank was photographing three young girls rather than two; that the "gasp reflex" is an involuntary exhalation rather than an inhalation; and that William Eggleston visited Meyerowitz in 1962, which should have been 1968. These have all been corrected.