At 4:30 A.M. on Sunday, September 20th, I was sitting in a massive industrial building on San Francisco’s Pier 70. It wasn’t obvious in the dark, but many of the windows were broken or missing panes. To get to my seat, I had to step over dozens of bodies wrapped tightly in their sleeping bags. Next to me was a young man, 22 or 23, smiling maniacally and chattering about his databases, the glow of a laptop screen lighting his face from below.

This was not the first time my job had sent me to a waterfront shell of bygone industry. In early May I spent two long days at the Brooklyn Navy Yard, huddled in a large tent with dozens of people, young and old alike. While the temperature swung from finger-numbing cold to sweltering heat, I hardly heard any complaints. What I did hear was excitement: a constant buzz about new ways to do things, a steady exchange of sometimes crazy but always interesting ideas.

For those used to attending conferences and trade conventions, this may seem like a strange way to network and collaborate. But anyone who has been to a hackathon will tell you the norm is often thrown out the window. The hackathon, a caffeine-fueled non-stop marathon of coding, is a unique mix of social event and technical competition. Imagine an all-nighter the week of finals with a hundred like-minded strangers, and you’ll start to get an idea of the environment hackathons create.

The benefits of hackathons to the participants are fairly obvious. For developers of all ages and expertise, a hackathon is the perfect place to strengthen problem-solving skills and learn about new technologies. At the 2015 TechCrunch Disrupt in San Francisco, two young developers, 9 and 10 years old, built an app to help parents and kids manage allowance. They didn’t just build the app; they pitched their idea to a crowd of hundreds. I think it’s safe to say this opportunity gave these kids an education above and beyond what they’ve learned in school.

For those in the tech industry, hackathons are an attractive venue. The close interaction fostered between job-seeking developers and talent-seeking companies helps with well-tailored recruitment. Corporate sponsorship and support increases brand recognition and excitement with far less overhead than traditional marketing campaigns. Some companies even send teams of developers to hackathons to get time with and access to some of the latest technologies.

But it can be difficult to see what value these events provide outside the confines of time and space where they are held. If you go to a hackathon demo session, you’ll see dozens of apps that barely function and many more that don’t live up to their promise. When you go to enough, you’ll begin to see the same ideas (the Uber for X, the Tinder for Y, the Yelp for Z), slightly modified and reconfigured. It’s hard to convince those outside the club that these events are hotbeds of innovation.

The trouble is, the innovation that comes out of hackathons has to do with process, not product. The concrete, tangible examples that are shown after 24 or 48 hours of hard work often fail to capture a deeper level of value that is being created. And it’s a pity, because the same things that make hackathons a unique driver of technical innovation could be applied in many other fields with great benefit.

In 1959, Isaac Asimov wrote an essay with thoughts on organizing people to do creative work. He believed creative work requires some amount of isolation. It is the individual who is obsessive enough to keep reworking and reforming the fragments of ideas that lead to innovative thought — the final effort of creation is hindered if too many people are involved. Anyone who has been part of a group project understands this sentiment.

Asimov goes on to describe the role of collaborative work: by bringing people together in the right environment, it is possible to increase the amount of knowledge available as inputs to the creative process. More importantly, working as a group can lead to useful combinations of ideas that can lead to entirely new knowledge.

This process of sharing and combining ideas is explored in detail in Steven Johnson’s book Where Good Ideas Come From. In the book, Johnson highlights a concept known as the adjacent possible. Originally coined by theoretical biologist Stuart Kauffman to describe the realm of possible life forms that could evolve from the ones already in existence, the adjacent possible can also be applied to the world of ideas. The adjacent possible contains everything that we don’t know yet, but we could know if we combined and expanded on our current knowledge.

It’s important to realize that the adjacent possible is a messy place, and it can be a scary place to work. The vast majority of the ideas living in this constantly growing and undefined space of possibility are not going to solve any problems. They may seem silly or trite, interesting but not useful, or any other flavor of inconsequential. And when results matter, anyone spending too much time in the adjacent possible can look like a fool.

This is a reality that Asimov addresses when he discusses the atmosphere that supports innovation. He makes it clear that any group whose aim is creative work and the exploration of the adjacent possible must allow its members to be comfortable goofing off. Fear of embarrassment and strict direction will cut off the vital force that allows collaboration to succeed.

Please don’t think this is another way of saying think outside the box. The mandate here is for creative thinkers to live outside the box entirely, at least for a while. It’s a philosophy that has long been at the root of hacker culture, even before the widespread use of modern computers. The book Nightwork catalogues the history of elaborate stunts and pranks that have been performed by students at MIT. These ‘hacks’, as they are known, have provided an outlet for unusual problem solving and alternative thinking since at least the 1950s. It’s no coincidence that this intense form of play is popular at one of the top research universities; the two modes of thinking fit hand in hand.

Hackathons do not have a sole claim on this approach to innovation, nor should they try to. What hackathons do provide is a duplicable model for creating enticing spaces in which we can expand the adjacent possible, a model that can be applied in any industry.

To illustrate what makes the hackathon model valuable and unique, I’ll start with a comparison to an academic conference. Conferences and hackathons share many features: networking, idea cross-pollination, like-minded professionals and students working together on similar problems. The big difference is that at a conference, the work being shared is nearly complete. All the false starts, all the irrelevant sidetracks, all the messy fuzziness that happens at the beginning of a project is mostly, if not entirely, hidden. A polished conference paper can be an important stepping stone to advance the field, but it does not allow for the spontaneous interaction of little half-ideas. A conference helps focus and clarify, whereas a hackathon creates a vibrant mess of possibility.

At the other extreme is the informal gathering of tinkerers and builders. This is where hacking has thrived for decades — the garage, the basement, the local computer store. Canzhi Yhe, organizer of Cal Hacks, has written an excellent article describing how their hackathon is being revamped to bring hacking back to its roots: a bunch of curious builders getting together to do something cool, just for the fun of it. One potential downside of these organic get-togethers, as highlighted by Raphael Palefsky-Smith, is the bias towards the traditional developer culture. It’s no secret that there is a diversity problem in tech, and the promise of spending time with your hacker buddies doing hacker things doesn’t go far enough to get new perspectives at the table.

So somewhere in the middle we have this wonderful hybrid, the corporate-sponsored hackathon. If done correctly, the support provided by sponsors works to strengthen the experience without diluting the creative potential of the participants. The space, the food, the prizes, and the exposure are all perks that might be enough to convince someone on the fence to spend the weekend in front of a computer with little sleep. The at-the-elbow support and exclusive access to services that many companies provide helps draw in more experienced developers, who can mentor and inspire those just starting out.

But the biggest draw must always be the dedication to bringing people together for collaboration, for sharing and combining ideas that wouldn’t be connected otherwise. It is in the best interest of participants and sponsors alike to eliminate anything that drives the event away from this goal.

And it is this core purpose that makes the hackathon such an intriguing model for creative thought outside the tech industry. Because hackathons aren’t about apps and startups; they aren’t about pitches and prizes. They are about a commitment to keeping possibilities alive that might otherwise be ignored. They are about getting all the options on the table for the problems we haven’t even seen yet.

In his role as Developer Evangelist for IBM’s Watson Developer Cloud, Zach works with developers to make the steps between great ideas and viable products as easy as possible. He hosts a weekly technical webinar series, “Building with Watson”, to show how the Watson APIs can be used to build smarter apps.

About AlchemyAPI, an IBM Company:

AlchemyAPI, an IBM Company, helps to enable developers and businesses to build smarter applications through a SaaS based platform of cognitive APIs. Available as part of the collection of services on Watson Developer Cloud, AlchemyAPI transforms vast amounts of content, images, and unstructured data into information that can digitally disrupt industries and help drive business decisions. AlchemyAPI leads innovation with natural language understanding and visual recognition, changing the way developers around the world process vast quantities of web-based documents and images.

To learn more, visit: alchemyapi.com

For information on IBM Watson, visit: ibmwatson.com