Stack Overflow is not a large community

Oh, sure, we all refer to the membership as a community - even the new top-bar site-switcher doohickey calls it one - but that's just shorthand. It is a place, the nexus of many related communities, a venue in which they can share information. You would no more call it a community than you would Portland - it's too big, too diverse, and has been for many years. There is a certain broad culture that is pervasive... But to call it a community is to ignore both the limitations and the strengths of communities - as well as the factors that make sites like Stack Overflow (or cities like Portland) work in spite of being far too big for everyone to know and remember one another.

That assertion generated more confusion than I initially expected... But I should have known better. It's natural to generalize one's interactions with a place to the place as a whole; I still think of Portland as the gal who approached me on the street offering brownies, and no doubt anyone reading this perceives Stack Overflow according to their own experiences, whether nice or nasty.

But that ain't the elephant...

Grid communications: scaling by breaking the social network

Hans touched on this already - the first and most crucial step toward allowing a site of this size to function is to discourage the sorts of interpersonal connections that would tie it down. I talked about this before, in a different context:

Conversations not required. When a question is asked on a traditional forum, answering it often demands some amount of participation from at least a portion of the community. Details are fleshed out, the problem is clarified, solutions are proposed and debated, others with similar problems chime in with their experiences, tangential points are made, and eventually - anywhere from hours to months later - the conversation dies out. It's a very social, very natural way to interact. And it suffers mightily from the problem that Shirky talked about: all that back-and-forth and associated latency kills any hope of scale. On Stack Overflow, we close or delete questions that can't be answered straight away - it's not very sociable, but it scales wonderfully by effectively enabling a vast, human-powered computational grid.

This isn't easy: by default, people do not behave this way. Here's an email we received recently - I'm sure you'll recognize it immediately, as it comes up quite often:

Is it possible to directly contact (personal/private message) another user? I really want to follow up with some of these posts (especially similar projects to mine), but the rules shun asking questions on the post, so...?

There's a big fat "Ask Question" button at the top of every page on the site - but folks still default to wanting to ask questions directed at a specific person, in the context of an existing "discussion"...

Pushing against the tide: the cost of an unsocial network

The cost of scaling to this size has been a constant battle against human nature. We are social creatures, and when asked - forced - to forego these personal connections, we get irritated. Scanning the answers to the most popular discussion here finds the same two stories repeated over and over again:

I'm here to learn but Stack Overflow doesn't want to guide me - my questions get downvoted and closed with nary a helpful comment. Folks are upset that Stack Overflow can't take them by the hand, welcome them and show them how to improve their work. This quote from the accepted answer is telling: Tell me, please. I don't care - say something MEAN if you have to: "too long", "already asked" "google search this", "obvious homework problem". I would rather be embarrassed five times in a row and finally GET IT than annoying everyone forever. But of course, this sort of one-on-one back-and-forth interaction doesn't scale. Heck, if everyone who came here asked 5 questions, the site would already be dead - if each question required this sort of tedious commentary, it would be even worse. The opposite response to that discussion is just as revealing...

I keep trying to educate folks asking bad questions, but no matter how much I write they keep coming - so I get more terse, more mean as I lose patience. Shadow Wizard captures this succinctly in his answer: So those regular users do their best to preserve quality by attacking the bad posts and trying to educate the users, sometimes being harsh while doing so. They're trying to do exactly what the first guy asked for, what Hans pines for: leave honest, even mean feedback. And then realizing that it doesn't help, and even when it does help it doesn't scale. This is why close reasons have descriptions attached - even one comment is too much to ask for most unanswerable questions. It's also why trying to cram very specific reasons into custom off-topic reasons for on-topic but poor-quality questions failed miserably on Stack Overflow: we can't enumerate all the badness, and in trying we just created pointless busywork for folks who could have used their time more effectively elsewhere.

And this is where Mysticial's answer subtly misses the point: the problem isn't "caretakers vs. vampires" - vampires don't care. It's not even caretakers vs. "tiny handful of people who care about asking good questions", because even though they often end up at odds with each other the truth is they both want the same thing, a persistent rewarding connection with another human... and that isn't a thing that either one can have.

So we've been having the same discussion for five years, because there is no permanent solution - it is a constant battle, and will be as long as SO exists. You can't change human nature; all you can do is actively work to subvert it, daily, in pursuit of your goal. So that's what we do here, but even that has become too labor-intensive...

Intensive farming

This is why we need to get away from the "big city" metaphor. The vast majority of users don't live here - they're just passing through. And we're not really trying to get most of them to stay if they're not interested in helping us build - if users were our goal, we'd be doing some sort of dodgy content-hiding trick to get folks to create accounts.

What we're doing here is more akin to gardening: planting seeds, fertilizing them, getting rid of the weeds, gathering the harvest. Except, Stack Overflow ain't your little backyard hot house where you tenderly nurture every tender seedling - it's more like a 25,000 acre wheat field. You cannot carefully tread the rows pulling weeds - you break out the heavy machinery!

The communities that do call this site home are those planters and cultivators, small indeed compared to those being fed by the results. Therefore, it is important to make sure that they are well-equipped and well-fed while keeping in mind that the communities they most identify with may yet be distinct from the roles they find themselves in here.

Your suggestions are generally good ones, albeit too vague to be directly actionable. So here's what we've been working on, plan to work on, or should be working on: