On a Monday in January, I visited the front page of the social news site Reddit to learn that, very occasionally, lobsters are pulled from the sea that are both orange and blue, the colours dividing in a neat line along the spine. Bottles of 7Up are used to contain a mood drug called lithium citrate. Somebody pointed out that, to get drinking glasses properly clean, it was best to use Sellotape to remove smudges. An eight-year-old girl had been detained wearing a suicide vest in Afghanistan, and there was a photograph of a commuter reading a tantric sex guide on the tube.

Like many people, around 7m on this particular Monday, I look in on reddit.com often, to see what happens to be capturing the imagination of this vast online community at any given moment. The people who run Reddit call it "the front page of the internet", bravado they can back up because more than 100 million unique users visit every month. A link that appears on Reddit's front page, sparely rendered in blue Verdana on a white background, will send traffic stampeding wherever it points. Gawker, an influential online voice in its own right, has called Reddit "the single dominant force in internet culture today".

"It's unpredictable, and that's powerful," Reddit's general manager, Erik Martin, told me. Martin runs the site with three dozen colleagues out of a cluttered office in San Francisco: pot plants, postcards, blinds drawn against the sun. "When I rolled up this morning and checked on Reddit, I had no idea what was going to be on the front page. And I think that's rarer and rarer. If I go on Facebook, or Twitter, or Google News, I basically know what I'm going to see. But on Reddit…"

But on Reddit there might be an item about multicoloured lobsters. The site is an ever-revising digest of distraction, dealing in new news and old, intriguing facts; in pictures, videos, songs; in broad jokes, subtle jokes, in-jokes; in questions, answers, hypotheticals; in contradictions, the scandalous rubbing up against the mawkish, trifles muddled with real outrages; in photos of babies, sloths, baby sloths – all of it sloshes through, 70,000 daily posts in Reddit's 6,500 forums, sorted by the community so the best contributions rise to become most visible.

Users vote. They click on arrows to push (say) a picture of a giant gingerbread statue up or down. Enough "up votes" and the link will rise in prominence, perhaps to the front page, where visitors are confronted by the 25 most popular posts of the moment. It was here that I learned of the death of Nelson Mandela. I also read about a man who claimed to have two penises and wept with laughter at an animated gif of people very intently dancing to electro house. I lost a night's sleep to explicit footage of a Russian car crash.

Now it was January, a time for resolution, and I planned to stop lurking on the fringes of Reddit. Instead, I'd take part, get an account and post something. Would it be possible to get to the rarefied ground of the front page? I spoke to some of the site's more successful contributors for advice.

It is thought bad form on Reddit to reveal your real name, and a user going by the name Shitty_Watercolour would not, except to say that he was an undergraduate at the University of York. "After a while you get a feel for what does well," he said, suggesting I focus on "quick, easily digestible content". A user called THENYEHHH, this confusing name no barrier to her gathering thousands of votes for her contributions, told me to keep content light, lively: "For the most part, everyone is just trying to have a good time."

The front page, as Monday evening approached, featured a picture of a cat sitting upright on its owner's lap to eat. Beneath it was an explanation of the science behind goosebumps and a trailer for the new series of House Of Cards. The American comedian Jerry Seinfeld had offered himself up to be interviewed by Reddit users, as had a porn actor. While Seinfeld discussed coffee blends and models of car, the porn actor fielded questions on the importance of good microphone positioning. She revealed an out-of-hours liking for Harry Potter and a secret talent for the mandolin. I was so taken by this last revelation, I borrowed it for my username.

Logged in as TalentForMandolin, I copied and pasted a link to an old news story from the BBC's website: a men's prison in Mexico had been subjected to a surprise inspection in 2011, where guards had found plasma TVs and sacks of marijuana in the cells, also a couple of peacocks, 100 fighting hens and 25 women. I thought this was weird and punchy enough to get some attention. I wrote a new headline for the story and pressed submit.

Some people have earned a lot of money through Reddit. In 2011, a user mused as to what might happen if a Roman legion fought America's modern military, and he wound up with a six-figure sum to write a screenplay on the subject. Others have earned just enough, for instance the struggling manufacturer of stained glass who put out a plea for custom at the end of 2013. ("Your support saved my little business," SenatorMars told the community.) Reddit's two founders are millionaires.

'When I rolled up this morning and checked on Reddit, I had no idea what was going to be on the front page. That's rarer and rarer. If I go on Facebook, or Twitter, or Google News, I basically know what I'm going to see.' Illustration: Arthur Chiverton

Steve Huffman and Alexis Ohanian became friends the day they moved in to student halls at the University of Virginia. "Steve wasn't so excited to meet me," Ohanian recalled. "He'd seen my name on my door and thought I'd be a girl." Huffman studied computer science, Ohanian history and commerce. Before graduating in 2005, they decided to start a company. Ohanian says, "I wanted us to be able to live like college students for as long as we could."

They spent months on a piece of software that would help people avoid queues in fast-food restaurants. It bombed in a meeting with investors, at which point they rushed together a new idea, "like a top 40 for interesting content", as they pitched it. Huffman admits to being light-fingered in the early days. He and Ohanian pinched the voting concept from a technology news site called Slashdot. From Slashdot, too, they spun off the idea of users collecting "karma" – brownie points, essentially, and worthless – in return for making beneficial contributions. Reddit's pared-down aesthetic "was copied almost entirely" from a bookmarking site called Delicious. Over a boiling summer, the friends sat at facing computers, listened to Gwen Stefani and built Reddit.

"We were lean," said Ohanian, who recalled eating little but hummus that summer. It was a phrase echoed by Reddit's general manager, Erik Martin, when I spoke to him by phone from San Francisco. "We're lean," he said of Reddit's 35-person staff. In 2014, the site looks almost as stark and unattractive as it did when it launched in 2005. This, Martin explained, allowed Reddit to continue growing (5bn page views last month) while keeping maintenance costs low.

These days, Reddit is a part of Advance Publications, which owns Condé Nast. Huffman and Ohanian, now 30, sold their creation in 2006. "Life-changing wealth," Ohanian called it, or $5m by industry estimates. Roughly speaking, Reddit has doubled in size every year since. It is yet to make a profit, "but we're not that far off", Martin said.

Given its eight years under corporate ownership, it's near miraculous that Reddit's verve, the unruliness that makes it appealing to so many, has survived. "Condé Nast realised pretty early they're not a technology company," Ohanian said. (He still serves on the Reddit board.) "They gave Reddit a ton of autonomy." Martin called Advance Publications "extremely hands-off".

Huffman has a different stance. He was kept on to run Reddit after its 2006 acquisition and launched the company's San Francisco office. Huffman found Reddit's ownership stifling and told me that by the end of his time at the company he wasn't on speaking terms with the then-president of Condé Nast Digital. One day on Reddit, the joke of the moment – there's always one – had been to mock the retail giant Sears, a big advertiser with Condé Nast. Sears complained, according to Huffman, who says he was told to remove certain posts, and eventually gave in. "I've still never forgiven them." Huffman left the company in 2009 and now runs a travel website, Hipmunk.

'Reddit is a snapshot of humanity, with all the beautiful parts and all the ugly parts intact.' Illustration: Arthur Chiverton for the Guardian

Martin told me that content is very rarely taken off Reddit. "There's only a handful of rules. Most of the time the moderators of the community remove [rule-breaking posts] before we even know about it. There are sometimes grey areas, edge cases, that we have to consider. But most of the time it's clear cut. Anything illegal is removed pretty quickly."

A forum called Jailbait, where users were encouraged to post pictures of underage girls, was banned in 2011; another called Creepshots, for secretly taken photographs of women, went, too. There are, however, plenty of Reddit forums remaining that cause unease in the wider community. (One user I spoke to cited a forum called SexyAbortions.) On a site that has a male-female balance of around 65%-35%, many find there's too much breezy misogyny for comfort. "Genuinely disappointing," Shitty_Watercolour called it.

"There is certainly a dark side to Reddit," Huffman said. "You get it all." People sometimes confess to crimes on these boards – even murder, last year, which triggered a police investigation as well as mainstream media coverage. Unflattering headlines were made shortly after the Boston bombings, too, when an innocent man was identified and vilified on Reddit as the person responsible. "Reddit is a snapshot of humanity, with all the beautiful parts and all the ugly parts intact," Huffman says.

Actually, given that people exist here anonymously, the site can seem a surprisingly affectionate and judicious place. It's a benefit of the voting system, which influences comments as well, so that ill-natured or illiterate remarks tend to be voted away out of sight. The comments under an average YouTube video might have been written by a different species. Unlike Facebook or Twitter, Reddit isn't dominated by the most tireless contributors. It's quite witty, quite warm.

Users I interviewed counselled me against trying to personify Reddit, to reduce it to this or that way of thinking. The community, it was pointed out, consisted of "homeless people to rich people, fast-food employees to doctors, agoraphobes to adventurers". Even so, it's possible to single out conspicuous leanings. Reddit is leftwing, irreligious, extremely keen on the actors Jennifer Lawrence and Patrick Stewart. A spirit of contrarianism runs deep. In 2010, when a moderator put up a test post with the instruction "please ignore", it received more than 20,000 votes and became Reddit's most popular post of all time. The top comment underneath read: "Don't tell me what to do."

My post about the Mexican prison was rising.

Seinfeld's interview now topped the front page. Underneath him was the House Of Cards trailer and a question about what would happen if China "randomly" invaded New Zealand. There was a post about worthwhile novels written in the 1940s and a video of a zookeeper masturbating a monkey. When my Mexican prison story pushed on to the front page, just after 8pm, it nudged off a cartoon about the weather in Australia.

I hadn't expected this. Plunging into the comments, I had to ask: wasn't it supposed to be hard to get to the front page? "One in a million, my friend," I was told. Someone else responded: "Meh, I've been there four times. It's no big." A veteran user called Aftli, who'd signed up for Reddit seven years ago, said one in a million was an exaggeration, but "it's pretty hard to do". And by the way, he added, "prepare for soul-crushing disappointment with the next 30 links you submit". A user called Toilet_Crusher was conciliatory. "Your post's been seen worldwide for sure!"

Sir David Attenborough took questions about mass extinction and shampooing during his Reddit AskMeAnything interview. Illustration: Arthur Chiverton for the Guardian

Martin told me that one of Reddit's great strengths was that it was both intimate and enormous. "You put something out in the universe and get a response. That response might be one comment. Or that response might put you on talkshows the next day." Ohanian said: "I like measuring things in New York City. We had 100m unique visitors last month. That's more than 12 New York Cities' worth of people. Mind-blowing!"

It's why the famous are more and more likely to appear on Reddit. Not just Seinfeld; Madonna came on last summer, Arnold Schwarzenegger in early 2013, Barack Obama in 2012. In Reddit terminology, where forums are known as "subreddits", and "karma" points are discussed as though of real monetary value, these interviews are called "AskMeAnythings". Correctly so, because Reddit will ask anything.

Molly Ringwald, was it true that on the set of The Breakfast Club you were bullied by Judd Nelson? Woody Harrelson, did your dad used to be a contract killer? Molly answered (in the affirmative); Woody didn't. AskMeAnythings don't always work, but when they do, they're wonderful. Sir David Attenborough took questions about mass extinction as well as his shampooing regime. When Shitty_Watercolour, who chose that name for a reason, painted a shitty watercolour as a gift for Attenborough and put it online, there was talk of printing, framing… Schwarzenegger enjoyed his Reddit interview so much he still drops by to post. So does the actor Joseph Gordon-Levitt.

While my story about Mexican prisons was prospering – now at No5, No4 – Gordon-Levitt came online to put up a link. It quickly rose into the top 10 and together we made our assault on Seinfeld in the top spot. I found it absurdly exciting, Seinfeld in New York, Gordon-Levitt in California, me at my kitchen table, holding the collective attention of the internet for 15 minutes or so.

It fell apart. Our power alliance crumbled. A video about crab fishing became Reddit's new favourite item. Gordon-Levitt lodged at No7 and did not move further. Seinfeld dropped down a place. I fell off the front page altogether.

In the days that followed, the prediction by my new friend Aftli – that I would face disappointment with my next posts – proved true. I put up an item about collective nouns and a biographical fact about Bruce Willis, to no response. I tried being edgy, with a joke about Hitler, and being cute, with a picture of a baby reading a newspaper on a bus. Reddit was indifferent.

This community wouldn't be told what to think, what to click; their 20m daily votes couldn't be lobbied for. When I asked Martin if he was ever courted by companies who wanted to harness the site's awesome traffic, he said, "People who know Reddit know that we certainly don't, uh..." He paused, to chuckle. "Nobody tells Reddit what to pay attention to."

Huffman insisted that from its earliest days he'd programmed Reddit to be resistant to anyone trying to "game" their way up the rankings. "Half of Reddit's source code was dedicated to catching bad content and users cheating." I was able to confirm this first-hand when I was banned from posting, temporarily, my contributions by now so terribly unsuccessful I was marked as a potential spammer. The Mexican prison story, which had peaked at No2, generated enough fresh traffic for the BBC to make this two-year-old item its most popular Latin American news story of the day. It had been voted on by 16,246 Reddit users. My subsequent posts, added up, got a grand total of 140. Which hurt.

Tell me about it, consoled Aftli, who still felt bruised by the failure of a forum he'd once tried to found, about interesting cash registers. Aftli advised me to forget about the front page. "For me, and I think for most Redditors, the site is more about the discussion than the links that are posted." Huffman agreed – making the top 25 wasn't the point. "I guess there are individuals who feel like they're competing, and they've won something [if they get there], but in reality they're part of something much larger."

Martin advised me to try looking "at the part of the iceberg that's underwater". Ohanian also suggested I venture deeper. Why not start with his favourite forum, called BirdsWithArms? So I did, and spent a happy half-hour clicking through Photoshopped pictures of owls holding tobacco pipes, canaries brandishing swords. Then I dug about and found a fascinating debate about which inanimate object was the bigger "asshole", the printer or the automatic sink? On Aftli's advice I tried ForwardsFromGrandma. I watched a thread develop in which a woman admitted she was dreading her own wedding, mortified at the thought of her fiance's family seeing how few friends she had. Slowly, as commenters pieced together where and when the service was taking place, people offered to come along and fill seats.

"The community is where the real value is at," Huffman said. "Since very, very early on, Reddit has had a soul."

It was a Wednesday and Huffman, like me, had spent hours riveted to the on-site confessions of a former mob boss, who spoke of initiation rituals and his grandma's meatballs. Elsewhere, advice was sought about how to turn the pages of a 340-year-old Bible. Tongs or spatula? A Starbucks barista admitted intentionally misspelling teenagers' names, so they couldn't post pictures of their cups on Instagram. Later, a Reddit staffer put up a note to announce that the site would be taken offline, briefly, for maintenance. A user posted a two-word response that seemed to sum up the popular mood.

"Now what?"