So there I was, squelching knee-deep in trash in Rio's scariest favela on a wet afternoon, my arms too sore from vaccinations to move, my whole body wrapped mummy-like in multiple layers of noisy plastic protective clothing fit for a moon-landing. The Brazilian production manager was telling me how many security guards with machine guns we'd need. Next item on the checklist was bulletproof vehicles. The garbage smells were mugging our noses.

I was in Jardim Gramacho, the largest landfill in the world, location scouting for my new documentary about Vik Muniz, Brazil's most famous contemporary artist. Muniz had come to Gramacho to create art with and about the catadores, the people who pick through the trash mountain gathering recyclable materials for a living.

As I stood among the rubbish, I realised I didn't need a new documentary, I needed a new head. My old head was thinking about a line from Joseph Conrad's Lord Jim that is something of a mantra for me: "A man that is born falls into a dream like a man who falls into the sea. If he tries to climb out into the air as inexperienced people endeavour to do, he drowns..."

I looked around. I was afraid. Laurel and Hardy might have been a more appropriate quote, this being nothing if not another fine mess I'd gotten myself into.

At that moment I heard a squeaky horn honk and looked up. Cycling towards me, grinning and tooting on a novelty eagle-horn, was a catador in his 60s. I'd been expecting these pickers to be the scariest of scavengers, murderers and mental-health hardnuts, garbage-vultures in human form. We'd heard this place described as "where everything that's not good goes, including the people". Yet this catador was the most charming, funkily dressed man I'd ever seen. This was Valter Dos Santos, and after meeting him my life would never be the same again.

Muniz, whose idea was to try to transform the pickers' lives using art, says that "the moment when one thing turns into another is the most beautiful moment. A combination of sounds turns into music. And that applies to everything." This was that moment for me, when my vision for the movie transformed as I saw how outrageously charming the pickers might be. What if Shakespeare had gotten here first and rewritten The Wombles as the most dignified and inspiring – not to mention chic – of environmental heroes?

A life had to have taken some dramatic turns to have brought you over the narrow bridge of land into the swampy tumour that is the Jardim Gramacho landfill. I knew that. And I knew that the catadores here would be transformative to meet. But I hadn't realised how life-changing it would be.

I ripped off the soggy layers of protective clothing, grabbed a mobile phone, snapped a photo of the "mongo" on Valter's bike ("mongo" is the word used by the NYC Sanitation Department as a verb or a noun to describe the treasures salvaged from trash for personal use) and called up our producer, hard at work on Ipanema beach. I told him to order some caipirinhas, because the film was going to be even better than we could have hoped.

Across the bay I could see the iconic Christ the Redeemer statue with his back to us, his arms reaching towards the wealthy in Rio's south zone, Copacabana and Ipanema. They say that even the Christ turns his back on the north side of Rio, where the landfill lies.

MUNIZ describes Rio as St Tropez surrounded by Mogadishu. The garbage is the only place in Rio where the social extremes get mixed together. The posh rubbish from the south zone mixed with the cheap trash from the favelas. Garbage is the negative of consumer culture, it's everything that nobody wants, and when it disappears from everyone's lives, rich or poor, it doesn't disappear at all, it appears here, like a conjuring trick gone wrong.

I spent the next few days meeting the catadores. Garbage is a matter of opinion, they said. Tread carefully, because you are treading on money. They can make twice the minimum wage salvaging cans, bottles, plastics, paper. After carnival they pick out the discarded costumes and samba as they work. When the airline Varig discarded their old cabin crew uniforms everyone dressed up in them and goofed around serving each other recycled drink bottles.

That's the most striking thing: the good humour, the sheer fun. When we film The Governor, a grinning old-timer with a boombox strapped to his belly, he calls out, "I'm gonna be on TV." "Yeah, the animal channel," someone quips back.

And they are honest. They don't touch each other's piles of pickings. Many catadores had limited career choices: sex work, drug trafficking or recycling and they chose recycling, where the only person you hurt is yourself. There are a lot of accidents. And there is a huge amount of pride. They may be ostracised, but they know they play a crucial role as environmental stewards.

Zumbi is the resident intellectual. We hear about him before we see him – that when he sees a book, he doesn't see just recycling paper. He has kept every book he's ever found on the landfill, and he has a lending library in his shack. He's handsome, like a young Samuel L Jackson, with a white towel tied around his head and a paperback bulging in his shorts.

Irma cooks up the past-sell-by-date produce in pots and pans rescued from the garbage over a hot burner flame of methane outgassed by the rotting trash underfoot.

And Valter, who cycled past me that first day, was the bard of the landfill, the Yoda of the place, never without an encouraging rhyme for his colleagues. "The fight is long, but victory is certain," he said. He'd worked in the landfill for 26 years and his favourite catchphrase of all was "99 is not 100" by which he meant every single can recycled, every single art project, every single documentary film, will make all the difference. He died shortly after we first met, before the portraits were complete, but not before we would think about him every day for the rest of our lives. I hope that as many people as possible will get to meet him by watching our film. Because he does not disappoint. And the people in it, the catadores of Jardim Gramacho, recycle all of our spirits.

The documentary Waste Land is released on 25 February

Lucy Walker was born in London. She is an award-winning film and documentary maker