It was March 2000 and I was with my friend Paul in the Darién Gap on the Panamanian-Colombian border looking for rare plant species. We had just entered a clearing with our two Colombian guides. For me as a gardener, it was the most amazing place. Suddenly several teenagers appeared brandishing guns – they threw us to the ground and tied us up. Our guides were dragged off into the bushes; we never heard from them again.

Our captors interrogated us for several hours, accusing us of being variously CIA, drug-runners and British spies. I kept telling them that I was a gardener looking for orchids and Paul was a mountaineer, but they wouldn’t have it. That was the beginning. For the next nine months we were held hostage in a series of camps on the border. In many of them I built a garden, just to keep myself sane; I’d plant a vegetable patch or grow some corn. They often threatened us, giving us the odd kick or punch. They couldn’t speak English and conversation was restricted to us thanking them for the armadillo we had for supper.

They made us write ransom notes to our parents – they wanted $5m. But they had no means of communication with the outside world so the notes were never sent.

Every so often they threatened to kill us, which is why we were given up for dead. It was pretty traumatic, but it’s how I came up with the idea for the World Garden. One day this kid – we called him Scarface because of a huge scar he had on his face – told us we were going to be killed that day. As soon as he left, I started scribbling an idea in my diary. It was for a garden I would create in the grounds of my home back in England – Lullingstone Castle. My idea was to show where all the plants and flowers in Britain come from.

I still have the diary – the drawings are barely legible as my hand was trembling so much.

I designed a garden with several large flower beds – each one in the shape of the continent the plants and flowers had come from. For example, there would be a bed representing South America that would contain all the plants and flowers growing in the UK that originally came from there. People don’t realise that about 80 per cent of our plants come from abroad. It stems from the Victorian plant-hunters who travelled the world bringing back plant cuttings.

They didn’t kill us that day. And in the end we drove them mad. We didn’t do anything in particular – I just talked about orchids a lot. And Paul’s continual complaining didn’t go down well either. The day they released us was surreal; they couldn’t take it any more. They just told us to leave. They gave us everything they had taken from us nine months earlier: passports, rucksacks and $3,500 worth of travellers’ cheques.

The problem was, we got lost in swampland and had to go back. They got a guide to lead us out: they kept saying, “We hate you; please go!”

Eventually we found civilisation and contacted the British Embassy. At first they refused to believe us – they thought we were dead – but in the end we were flown home. When I got back, I rested. Then I started work on the World Garden. Lullingstone Castle has been in our family since 1361 and its upkeep was crippling. We opened the garden to the public and now have 10,000 visitors a year, so it keeps the estate going. And for that, if nothing else, I thank Scarface.

The earliest recorded plant-hunting expedition was in 1495BC when Egypt’s Queen Hatshepsup sent botanists to Somalia in search of incense trees. The Victorian botanist Robert Fortune disguised himself as a Chinese peasant to smuggle tea plant cuttings to India, giving Britain dominance in the tea trade. Orchids came to Britain in 1818 when botanist William Swainson sent home a box from Brazil, using the plants simply as packing material. The orchids flowered and became hugely fashionable.