How can anyone possibly "starve on benefits"? "Something doesn't add up." Responses to my blog and comment pieces about life on the breadline are filled with these kinds of musings. After all, everyone has an anecdote about a mate or a neighbour who has all the latest gadgets and designer sportswear for their children, "and they've been on benefits since, like, for ever".

Yet you don't know to look at someone whether their trainers came from JD Sports or Oxfam, or if they wash their hair in Charles Worthington or Sainsbury's baby shampoo. You don't know whether the young man in the town centre with a takeaway in the middle of the day is a night-shift worker or a student or, heaven forbid, has the day off work. Not everyone works Monday to Friday, 9-5. Someone needs to staff the checkouts at night so the 9-5ers can pick up a bottle of Bolly after work. The 999 control room staff work day and night shifts across all of the emergency services. And someone needs to clean those supermarkets and those offices.

You don't know whether the young woman walking down the street with a black baby and a lanky blond nine-year-old on her arm is a foster carer or a "slut". I was that young woman taking the kids to the shops to give my parents, who were foster carers, five minutes' peace. I was always met with whispers and stares, because people judge and people assume. One woman told me that I "should have kept my legs shut". I told her that I was a virgin, and, funnily enough, she didn't have a comeback.

So, in an attempt to explain again and again that life is not as black and white, and the "free money" is not as easy to come by as certain newspapers would have you believe, I find myself a stuck record, plastering my personal circumstances across the media in an effort to make people understand.

When I first needed to claim housing benefit, the payments were delayed, leaving me in arrears. I had to involve my MP and the town clerk, but the rent arrears were already piling up. When payments are delayed, bills bounce, leaving you with hundreds of pounds in bank charges, on top of your rent arrears. When you go back to work, you need to pay for a month's fees in advance to secure your child's nursery place. The money allocated to benefits is for housing, food and bills. But when it is not paid in full, or not paid on time, when you have to wait 11 weeks for it due to "administrative backlog", the money also has to pay for late payment fees, bank charges and rent arrears.

This mess is not unique to me. According to the Trussell Trust, the most cited reason for referral to food banks is delayed and missed benefit payments. Meaning, yes, Mr Gove, you were half right that food bank use is down to financial mismanagement, but it is the financial mismanagement of the Department for Work and Pensions, and the financial mismanagement of local authorities that administer housing benefit. Not quite the fault of the "feckless poor".

While it is easy for people to pontificate on what I could have done differently (taken my former employer to a tribunal would be first on my list if I could relive the past two years) or berate my "absent" friends and family (to whom I was too embarrassed and humiliated to admit how bad things were) – it is much harder to put your head above the parapet and say to a nation that hates benefit claimants: "Your perception is wrong. The system is flawed. People are getting hurt. Something has to change."

• Jack Monroe blogs at A Girl Called Jack and is author of a forthcoming recipe book for people cooking on a tight budget