I would have been still in my teens when LS Lowry first gave me one of his pictures. It was the mid-50s, and I used to help my uncle out delivering milk from his farm, driving around the streets of Mottram in Cheshire each morning in his little Austin van. Lowry lived in a large, detached stone house on Stalybridge Road, and I always arrived there at 11 o'clock sharp; the house was full of clocks and they'd all start chiming as I arrived. He was already fairly famous by the time he moved in, and you'd see him around the village in his black raincoat and trilby. He worked in a glass conservatory at the side of the house and I'd take the milk in there – he had two pints every day. He was a shy man, not unfriendly, but not one for idle chitchat either.

One particular morning, he was working on a sketch – he did many in pencil or charcoal – and he took one of the bottles of milk from me and tried to open it. In those days, there'd be two inches of cream at the top of the bottle, and as Lowry pressed his thumb into the foil, cream spurted out all over the paper. He tried to mop it up, but the picture just got more smudged. In the end, he passed it to me and said, "Put that in the dustbin on your way out, please." So that's what I did – I never thought twice about it.

In the months after that, he gave me several other drawings. I never questioned him doing this. After the first incident, I just assumed he wanted them thrown away. I've always imagined they were either finished pieces he wasn't happy with or works in progress that weren't going too well – he certainly never gave the impression they were gifts or anything. I could have kept them, but they simply didn't appeal to me, so into the bin they went.

When I first visited Lowry's house, his pictures were scattered all over the place. I never thought much of them – it always seemed to me a child could have done better. I prefer a good landscape or portrait, something that looks real. He once said to me, "I might be rich when I'm dead, but it won't do me any good." He didn't seem very interested in money – he stayed in the same house for 30 years and nothing ever changed.

By the late 50s, his fame grew and he always seemed to have visitors. I no longer saw his pictures all over the place – they got packaged up as soon as they were finished, sold to collectors and galleries. That's when it occurred to me that maybe I'd been a bit hasty throwing away those sketches. My uncle once asked Lowry to paint something for him, but he never did. I know he presented one to Mrs Swindell, the lady who used to clean and cook for him, and it got stolen from her home. Someone else bought one from him for a large sum and he was burgled, too. I remember feeling glad that I didn't have any – they'd only have attracted thieves.

A few years ago, I visited the doctor, and while I was in the waiting room I noticed a Lowry print on the wall. It was one with matchstick men coming out of a factory, and the spitting image of the sketches he gave me. Perhaps it had been a work in progress that he'd handed to me. You do hear of drawings like the ones I was given fetching many thousands of pounds in auction; someone said if I'd hung on to them, I could have ended up a millionaire.

But I've never really regretted what I did – I don't believe in fretting about things that are gone. I'd barely thought about it until an artist came to our local community centre to paint a mural inspired by Lowry, and included me in it carrying a crate of milk. It was the first time I'd mentioned throwing away those sketches, but word soon spread. I went to the market the other day and an auctioneer there shouted out, "All right, Ben, got any Lowrys?" At first I took those sorts of comments as idle jokes, but it upsets me that people take the mickey. I feel they could easily have done the same.

Now, though, I can appreciate Lowry's paintings more – they're full of old mills and beautiful stone buildings that were knocked down long ago, or turned into flats, and they remind me of my youth. But that's just nostalgia. I've never believed in hoarding things in case they become valuable – I take each day as it comes. Lowry did, too, I think, and I'm sure he would have respected my decision.