A few months ago I gave a talk to a group of primary-school students in Clifton Hill, Melbourne, about the fact that I am a volunteer gardener at Alcatraz, the famous island and former federal prison in San Francisco Bay. A forbidding rock, it sits high out of the sea, capped by a lighthouse and the imposing prison buildings. Although the macabre reputation lingers, Alcatraz is not only a major tourist attraction for prison geeks but also a reservation for seabirds, an important Native American site and a home to heritage gardens. Nine hectares of rock has come to mean a lot of things to a lot of people.

The gardens were established by the military in the 1860s, when the island was used to hold Civil War prisoners. Once the federal government took over Alcatraz in the 1930s, inmates continued to work the soil. I wanted to tell the kids about Elliott Michener, the counterfeiter-turned-gardener who planted the spectacular succulent gardens on the western side of the island in the 1940s. He had no horticultural background, but he studied books and the seed catalogues that guards gave him, and ended up working in the home and gardens of the warden, Edwin Swope. When Michener was transferred to Leavenworth prison in Kansas, he was not happy: “I believe that my best and only practical course is to get back to Alcatraz … I could at least grow Bell roses and delphiniums seven days a week and enjoy considerable freedom and trust, and in general make the best of things.”

As I shuffle slowly down the steep western slope in the cold, bright autumn sun, in order to replant the ‘Persian Carpet’, I certainly feel like I’m making the best of things. That life is good. First planted in the 1920s, the carpet is formed from ice plants (Drosanthemum floribundum) matted together. They help stop erosion, bloom pink and purple for several months of the year, and are known as survivor plants.

The word “survivor” means all kinds of things on Alcatraz, but in the context of the gardens it’s a reference to the 200 or so plant species on the island that continued to grow through a 40-year hiatus in care. Alcatraz stopped being a federal prison in 1963. A Native American occupation began in 1969, in protest of the federal government’s policies regarding the indigenous population, but that ended 19 months later in June 1971. Alcatraz then became a national park, partly to justify the eviction of the occupiers. Salvage work on the gardens began in 2003.

Another day we plant survivor irises – long, gnarled rhizomes – in soil hardened by the recent Californian drought. It’s always been dry, however, and the gardens have always been planted with species tough enough to cope.

In the 1930s, after the military handed Alcatraz over to the federal government, the warden’s secretary, Fred Reichel, asked the California Horticultural Society which seedlings might grow well on the island. Many of the recommended plants came from the Mediterranean, and were among those that flourished through the years of neglect. Fifteen rose species also survived, including the ‘Bardou Job’, which originally came from Wales but hadn’t been seen there for years. There are also gnarly fig trees, ancient fuchsias and agave plants that have grown to 4 metres.

Trees and flowers are still being found under ivy so thick that it holds up some of the fences and foundation walls, so old that its stems are the size of tree trunks. We enjoy a bit of what we like to call “extreme gardening”. Once, while we were on the rocky banks on the island’s east side attempting to pull out ivy that hadn’t been touched since 1963, I realised that if I succeeded in getting rid of it all, I’d have nothing to stand on. The banks of the island are unpredictable; some are built from the rubble of former buildings. In the old garden beds we find bones, bits of metal, broken plates. The other day a gardener found a stone arrowhead, which was sent off for analysis.

Bullet casings turn up, too. The 1946 Battle of Alcatraz, a gun battle that lasted, on and off, for three days, was triggered by an escape attempt involving Clarence Carnes, a Native American and the youngest ever inmate of Alcatraz. The “Rock” has imprisoned many Native Americans over the years, including more than a dozen “non-compliant” members of the Hopi tribe in the late 1800s.

Before Europeans arrived, Native Americans visited the island to hunt seabirds, but never lived there permanently. Like many of the more recent prisoners and their guards, many felt that bad spirits haunted the island.

I never did talk to the schoolkids about the reputed hauntings, though in retrospect I should have. I had plenty of other material and gave an enthusiastic spiel. Their teacher had already told them about the escape attempt of 1962, during which Frank Morris and the Anglin brothers left papier-mâché heads on the pillows of their beds, climbed into ventilation ducts through openings they’d dug with modified spoons, and slipped off the island on a raft made of raincoats, never to be seen again. My audience listened patiently on the topics of geography and gardening until the teacher called for questions. One little boy politely put up his hand.

“How do you make a papier-mâché head?”

“Balloons,” I told him. “Fill them up with air and slather them with newspaper and glue. Use hair taken from the floor of your local barber.”

A few weeks later I passed the message back to these miniature escape enthusiasts that one of the present-day gardeners had taken a video of a great white shark not far from the landing dock. It leapt out and grabbed a seal just as tourists were boarding the ferry. There was a lot of blood. “Believe me,” I said, “those guys did not get away.”

I didn’t manage to crush the students’ dreams, though. They drew treasure maps, and researched escape possibilities. And they certainly aren’t alone in their fascination with some of the island’s more gruesome happenings. One and a half million visitors head out to Alcatraz each year to learn about the gun battles, the various bids for freedom. (Thirty-six prisoners made 14 escape attempts. Twenty-three were caught alive, six were killed, two drowned, and five, including Morris and the Anglin brothers, were listed as missing and presumed drowned.) Stories about Al Capone and the Birdman of Alcatraz are also standard fare.

Things are a lot more welcoming on the Rock these days. In a first for the Golden Gate National Parks Conservancy, the graffiti on the dock and the water tower was recently repainted. It reads, “Peace and Freedom / Welcome / Home of the Free Indian Land”, in honour of the 1969–71 Native American occupation. One of the rangers greets tourists to the island by singing Alcatraz facts to the tune of Adele’s ‘Rolling in the Deep’. A couple of times a week, former prisoner Bill Baker spruiks his memoir. He’s in his 80s now, has spent more than half his life in prison (and about three of those years in Alcatraz) and has done, he tells us, “some bad things”. He was married not so long ago in the particularly beautiful garden known as Officers’ Row.

Last year, Alcatraz hosted some of Chinese artist Ai Weiwei’s installations in the old laundry quarters: massive silk dragons festooned with quotes on freedom, and Lego portraits of people incarcerated around the globe. That exhibition was replaced by one of photos and interviews with some of the millions of children who’ve lost parents to the American incarceration industry. And in turn came another exhibition, presenting massive portraits of present-day elderly inmates. Some of the more striking portraits were of Black Panther members, locked up 40 years ago. It’s important to us, one of the rangers tells me, that people don’t think we’re here to celebrate prison.

It’s a challenge to preserve such history without giving over to some people’s desire to glorify it. While the prison years are the focus of tourist interest – and Alcatraz is one of the most visited national parks in the United States – supervisor Kathryn Daskal is keen to organise more art exhibitions, to focus on the Native American occupation, and to promote the gardens, to broaden an outsider’s sense of the island’s history and meaning. Conservation of Alcatraz trumps accessibility to it, which is why the number of visitors is capped.

Of course, what is being conserved is complicated on an island where almost everything, including the topsoil, has come from somewhere else. Some sides of the island are man-made, making it bigger now than it was back in 1846. Nowadays the soil is self-sustaining: the gardens produce so much compost, and a rainwater catchment system was installed in 2009. (Water problems, and associated expenses, were one of the reasons the prison was shut down in 1963.)

Before humans arrived, Alcatraz was home only to seabirds. With the exception of the pelicans, after which the island is named, birds have returned in their thousands. Snowy egrets, Brandt’s cormorants and black-crowned night-herons jostle for space beside aggressive western gulls. The gulls have built nests all over the island, through the prison’s ruins, and stand guard around its perimeter.

The birds, the gardens, the Native American occupation – these all seem essential to an understanding of what the Rock has become and the effect it has on people. Richard Oakes, one of the Native American occupation leaders, once said, “Alcatraz is not an island. It’s an idea. It’s the idea that you can recapture and be in control of your life and your destiny.” In the 19 months that he and others lived on Alcatraz, ceremonies were held and connections between tribes rebuilt, as people found a way back to their heritage. The inmate gardener Elliott Michener intimated something similar when he said, “If we are all our own jailers, and prisoners of our traits, then I am grateful for my introduction to the spade and trowel, the seed and the spray can.”

Recently, a tourist approached us as we were fighting ivy back from the tree it was strangling. He asked if the purpose of the gardens was a form of military defence. To my mind it was just the opposite. These gardens, planted more than 150 years ago, were a way of asserting possibilities other than the defensive, the violent.

“No,” we said. “The people who lived out here just liked gardens.”