Whenever a video game story gets featured on the front page of the Guardian website – as my Grand Theft Auto V review did – I'm always pleased, but also slightly apprehensive. I know it'll be read by people who don't usually play games and that can be interesting. I fantasise that maybe one of those people will read the article and think "I'm going to give this gaming lark a try" and that they will buy a console on their way home from work, and that it'll change their life for the better. At the same time, however, I also know to expect a series of variations on two classic negative comments: "You geeks need to get out more" and "Do adults really play these things?"

These points of view are, of course, knee-jerk hangovers from the early era of games, when they were generally thought of as something kids did. In my response to James Delingpole's comment piece on GTA 5 earlier this week, I argued that the moral panic surrounding games was akin to the worry caused by the advent of rock'n'roll or video nasties – the older generation fearing some new madness of youth. With games, though, we can also throw in technophobia. Games are dangerous and scary because they are about violence (sometimes) and they require you to plug something into your TV. So you know, best just to ask for them to be banned, because if you don't like them then they are probably causing teenagers to kill people.

I get that fear, it's fine. It's utterly wrong in my opinion, but … I get where that fear comes from.

When I am feeling less reasonable, the "do adults really play these things?" comment really gets to me. Partly this is because I am 42 and I review these things for a living. I've dealt with my insecurities about this before, but to have people questioning the mental faculties of mature gamers is something of a personal slight. But it's not just that. It's something else. Something more important.

The brilliant game designer Raph Koster says that we love video games because they teach us things, and we like to learn. What's more, they teach us things in the exact way that our brains prefer to take on new information: through systems and patterns. Even better, games teach us within a safe environment where it's OK to get stuff wrong. You may lose a life in a game when you make a mistake, but good games brilliantly balance the inconvenience of this with the provision of power-ups and health packs – little nuggets of grace in the learning system. If you don't think games teach us things, then you should play Tetris or Minecraft or Richard Hofmeier's excellent retail simulation, Cart Life. You now know more about geometry, architecture and the economics of poverty than you did a few hours ago – which is nice.

So games are partly about learning and we should never stop learning. They challenge us and we should always be challenged. Portal is so clever and funny and difficult, you have to think in new and unusual ways about space and representation on the 2D screen. The smartphone titles Drop 7 and Stickets provide so much complexity with such rudimentary materials: shapes and numbers and manipulation. Do you want to feel clever? Get good at Drop 7. We should always seek to feel more clever.

So, you see, part of why this "do adults really play games" thing makes me cross is that it's about intellectual snobbery, or even just intellectual ignorance (is that an oxymoron?). No one scoffs incredulously at commuters doing the crossword, or at Magnus Carlsen winning a chess tournament – but they're doing the same thing as gamers – they are playing. And we should never ever stop playing.

And that thought brings me round to the real annoyance. That there are people who seek to place a metric on fun. This type of fun is better than that type of fun – or maybe we're not supposed to have fun at all? I wonder if the "do adults really play games" people scour the Guardian site looking for forms of fun that they don't agree with, and place similar comments on all of them. The fun police. Are you having the right sort of fun, the sort of fun I like? No? Then I shall condemn you in the comments section. What do these people hope for? That someone who has fun playing games will think, "Oh crap, what am I doing?!" And that they will stop?

Maybe I can save you some time. They won't. Because the people who read game reviews and the people who play games know that games are wonderful and can be enjoyed as part of a healthy lifestyle. Also, increasingly, they engage with us emotionally. Gone Home is painfully moving and so damned right about how families sometimes let us down when we need them; The Last of Us says something simultaneously terrifying and hopeful about what we will do to protect loved ones. That's a lesson all adults need and can use.

Games are also becoming more serious. Iranian designer Navid Khonsari is currently writing a game called 1979 about the Islamic revolution in his home country. He can never go back, because the conservative newspapers in Iran accused him of spying and of distributing propaganda. He wants to tell people about the revolution and he knows that a game is a good way to portray different perspectives, and to encourage viewers/players to explore and understand those viewpoints. Navid Khonsari once worked on Grand Theft Auto so he really knows that adults play and enjoy this powerful narrative medium.

I don't think people go to the books section of the Guardian and write "why not go outside? Do adults really read books?" under every review. It's OK to stay in, curled up with a good novel, but not a game. Why? Because one is somehow better than the other? In what sense? If we're going to be mean, how many people actually read really good, really intellectually enriching books? Look at the bestseller list – a lot of people like escapism and there's nothing wrong with that. Adults between the ages of 25 and 35 spend more than three hours a day watching TV – you're not telling me that's all political documentaries and worthy dramas, right? Hey that's fine too. Why judge? Just, you know, accept that others feel differently.

Adults play video games. Heck, there are hundreds of video games designed just for adults! It's just like everything else. Adults fly kites, adults play bridge, adults make sandcastles and do jigsaw puzzles. Games like GTA worry people because they're violent and they have some questionable content – adults can handle it. And not all gamers like GTA. As my friend Simon Parkin mentions in his article on immorality in game worlds for the New Yorker, it's OK to love games but be worried about aspects of them. We're adults, go for it!

As I've written before, games have won – they are here to stay, and their importance is growing. We live in what game designer Eric Zimmerman calls a ludic century; everything is about interaction and play – from TV to smartphones to social networks. And games are the approachable face of that. TV execs are learning from games, banks are making their websites more game-like, the military is definitely learning from games. Some of this is good, some of it is troubling, but sorry everyone, video games are not going anywhere – they're going everywhere.

Adults play video games. We talk about them, argue about them, love them and share them with our kids and our families. I consider my life to have been enriched by video games and by meeting their designers. Richard Lemarchand (Uncharted series), Adam Saltsman (Canabalt, Hundreds), Robin Hunickie (Journey) and Peter Molyneux are among the most interesting and intelligent people I have ever met. I have wonderful memories of playing and sharing games like Resident Evil, Pro Evolution Soccer, Rock Band, Battlefield, Katamari Damacy... the list goes on. I store those memories with my others, I don't grade them lower because they involved graphics and joypads.

If you've ever written "do adults really play these things?" please try one. Go to an event like GameCity or Rezzed or the Eurogamer Expo and have a go. Nothing bad will happen. You might confirm your beliefs, that's fine, but at least you'll see that adults play games. You won't have to ask that question anymore. You'll know. You'll have tried and learned. That's a good thing. That's something all adults do.