"Hello Mr Welch, Visa Card Services here." That was line with which my nightmare started one Sunday morning, hungover, sitting on the sofa trying to piece together the night before.

The person on the other end of the phone, Mark, told me there had been a number of fraudulent transactions on my bank account since midnight, adding up to about £1,100. I'd never heard of Visa Card Services before, but then I'd never had money stolen like this before.

He then confirmed the last genuine withdrawal I'd made – at the Barclays opposite Highbury & Islington station – gave me a reference number and told me to ring the number on the back of my bank card. I did just that, quoted the reference number and spoke to someone who knew all about the supposed fraud.

Some cunning tricksters had apparently cloned my card at an ATM I had used and then treated themselves to a few things in an Apple store. Something didn't ring true about the whole thing – why would someone with a stolen bank card only spend £400 in the Apple store, for starters? But I watch enough consumer TV to know that these things happen.

The person apparently helping me, Rajesh Khan in HSBC's card protection department, had all my details: full name, date of birth and, crucially, my address. When he said a courier was on the way to collect my bank card for further examination, I didn't need to tell him where I lived.

I initially flinched at the idea, but when he explained it was needed to properly analyse the chip it seemed to make sense. After all, I had called the bank myself, this was no cold call, and he had all my details already. That's probably why I also typed my pin into the keypad of my phone.

"It's OK, Mr Welch, we can't see it, but we need to perform a pin block."

"I've never heard of that," I said, "but fair enough."

I packaged the card up as requested and waited for the courier to arrive. Rajesh called back twice, once to say the car was five minutes away, and again to say it was outside, quoting the car's number plate and describing the driver. He called again later that afternoon to say they had received the card and that I would have my money back in a few days.

Sucked in by the efficiency, I went through exactly the same process the following day with my credit card. The same fraudsters had somehow hacked into my online account and maxed it out.

But then a few days went by and Rajesh stopped calling. Worried – by this point I was, to my estimations, about £5,000 out of pocket – I called the bank, this time from my mobile.

After explaining the situation to two or three people, I heard the most chilling phrase of all: "But Mr Welch, your cards haven't been reported stolen."

Realisations kept hitting me as I relayed the conversations, over and over and over. Why had I given my card to a stranger? Why had I typed my pin into the phone? How did they have my mother's maiden name? How did they have my address? And, most of all, why in the name of all things holy hadn't I checked my balance to see for myself what the damage was before I even called the bank that Sunday morning?

Well, to answer the last question first, I suppose I didn't want to see what was happening. When I did check, things were far worse than I'd expected, and my rent had bounced to cap it all off nicely.

The Apple store story was all a lie – they had in fact spent thousands in clothes shops and, best of all, treated themselves to a Dixie Fried Chicken each evening. Forget the fraud, who spends £95 over three days in a Kentish Town takeaway?

The rest of it comes down to good faith. Once you call the number on the back of a bank card and go through security stages, you enter into a world of trust where you are no longer the boss and the person on the other end takes over. "My national insurance number? Sure stranger I've never spoken to before, here you go."

By now, I was really panicking. I called the police who put me on to their dedicated fraud line. After explaining my idiocy once again they went through the likely series of events that led to this theft.

It all started, according to the police, on the Saturday night where one of this gang will have watched me take money from the cash point. That's the details of my last transaction taken care of. Sinister enough, the thought of being spied on while you're trying to enjoy yourself at a garage night at the Buffalo Bar, but not the worst of it.

The police then believe I was followed home, which is how they got my address.

As for the call: well, credit where it's due, it's pretty clever. If you call a landline it's up to you to end the call. If the other person, the person who receives the call, puts down the receiver, it doesn't hang up, meaning that when I attempted to hang up to go and find my bank card, the fraudster was still on the other end, waiting for me to pick up the phone and call "the bank". As I did this, he played a dial tone down the line, and then a ring tone, making me think it was a normal call.

Fortunately my real bank gave me all my money back within 10 days, although I did have to get new accounts and cards. It was a pretty lean spell, and by the time I got my money back I had spent my last 60p on a tin of beans. The feeling of total financial ruin, of utter helplessness, isn't one I will forget in a hurry.

Setting up all new direct debits was an unholy pain and, four months on, problems are still arising and my credit rating has taken a serious knock.

I have had to sign up to a number of other bank schemes and government services to add further layers of protection. I get a monthly statement of credit checks in my name, for example, so I know if these people are using the information they have on me again.

It took a few weeks to stop worrying about the same people coming back to my house, too, although spending hours online reading about the link between bank fraud and violent crime – virtually non-existent, it would seem – helped with that.

I like to think I'm a tech-savvy, culturally aware person. I read about internet security, I know about phishing and all that, yet the knowledge left me when it counted and I handed over all my money like some wet-behind-the-ears yokel buying magic beans at a county fair. I'm surprised I didn't offer to help them spend the cash as well, get the job done properly.

Bank fraud is a bigger problem than I had ever realised. Experts suggest one in four of us will be directly affected at one point or another, while millions and millions of pounds is pumped into funding departments such as the ones that sorted out my problem and for the insurance it took to cover the stolen money. That's our money, paid in extortionate overdraft arrangement fees and so on.

Financial fraud is often deemed a victimless crime because, ultimately, it's only huge companies footing the bill, not individuals. Having suffered myself I can say that the stress, upset and countless hours spent sorting it out tell me it's anything but.

• A longer version of this piece appears on Andy's blog