And so, misusing a machine built to do math, that was prematurely optimized, built without simplicity in mind, inspired by the textile industry, backed by no underlying fundamental laws, with no way for us to understand it, that could generate more complexity than we could possibly embrace and armed with zero experience and a completely backwards approach towards problem-solving, we started the business of dealing with accidental complexity in the name shipping features––known today as programming…

…and a shit-ton of wonderful things popped from the other end. Damn it.

Keep on stackin’

Since programming was so damn useful we caved and started dealing with the accidental complexity. Over the years, when we tried to solve actual problems with computers, we found some inconveniences. “Ugh, punched cards are such a hassle”. “Ugh, machine code is so error-prone”. So we came up with ad-hoc solutions to those problems. We invented assembly code. This way, instead of writing “100101 0100 0111”, that means “ 4 + 7” you’d write “ADD 4 7”. Hey, much nicer huh? Thanks to these types of languages, we were able to tackle more complex problems and create more complex programs.

But with that ability also came more sources of accidental complexity. To put it bluntly, we were giving ourselves more rope to hang ourselves with.

We encountered some other problems and we created some more ad-hoc solutions for these too. “Ugh, it is so hard to understand the flow a program written in assembly”. Not to worry my friend, here is FORTRAN. We invented a high-level language that was easier for humans to understand. Same deal, easier to understand means easier to create more complex programs, which means more trouble for our limited minds. All well intended, but by solving a problem we created others.

You’d be angry too

We kept stacking sloppy solutions on top of each other but that only created more complexity and problems. In fact, I picture programmers as little people standing on top of a stack of poker chips so tall that we can’t see the ground, clouds around us and all. These poker chips (ad-hoc solutions) are not perfectly shaped and the whole tower wobbles so much that a regular person would puke their guts out, we are kinda used to it, though. Every time we see an imperfection with the poker chip right below our feet we put a new chip on top of it trying to cover the problem and hoping that it will be the last poker chip we will ever add to the stack, even though we subconsciously know we are lying to ourselves. And while on the top of this wobbly stack of imperfect poker chips we try to solve some of the most complicated problems known to humanity.

One of the the most blatant examples I can think of is how we represent text using a computer. Letters don’t have a mathematical origin but we needed to manipulate text with computers so, ¯\_(ツ)_/¯ we shoehorned it in there. With our limited and simplistic minds, we went ahead and gave one number to each letter. Today the letter ‘a’ is represented by the number 97 and ‘z’ by 122 in the computer you are using right now. But, turns out that there are characters other than letters, digits and symbols. Other cultures have radically different writing systems where a character has multiple variations or different orders and all sort of things that do not fit the original one-letter- one-number model. This problem is far from solved, we are even inventing new characters today: 😩😡😂😭😭😭 or 🙄🤓🖕 or new variations: 💪🏿🤘🏼🖕🏽 (chances are you can’t see all those emoji). Something so fundamental to modern computing like text does not have a clear mapping to our computers. This mismatch is a huge source of accidental complexity[12] and you, the user, are the one paying for that. Image how bad it is for things that aren’t as basic as text.

And what do we do to mitigate these type of problems? We come up with (ever-expanding and ever-changing) standards! This is how it goes:

“Fortunately, the charging one has been solved now that we’ve all standardized on mini-USB. Or is it micro-USB? Shit” Source: xkcd

In programming we have multiple solutions for a given problem, backed by multiple standards. Now the problem is which one to pick. Sometimes for political reasons, different vendors tend to use different standards. We have to translate between them in order to make machines interoperate or to target multiple platforms, like building an app for iOS and Android. All this leads full circle to the conversation depicted above.

An example of this happened a couple of weeks ago right in front of my eyes. GraphQL is an open source query language released by Facebook looking to unify the many different sources of data in a backend with a simple query language (GraphQL is a good example of a poker chip looking to hide the chips below). The programmer in me is actually very excited about it. Well, a developer didn’t like some part of it so he forked the project and created his own version that’d satisfy his own needs:

And again, for a given problem we have multiple solutions. I want to be clear, we can’t really blame the author of the fork, he may be trying to ship a feature. Maybe he even tries to contribute his changes back and it’s all good. But maybe, and this happens, Facebook won’t accept those changes because they don’t consider them to be the right thing for their project. Chances are, that sooner rather than later, we will need another solution, a level above GraphQL, that encapsulates all the different dialects of GraphQL, just to have the problem repeat at that level as well.

You know where this is going. Either as a workaround for a leaky abstraction, because of politics or because the wind blows, we keep stacking more sloppy and ad-hoc solutions on top of each other without asking ourselves whether that is the right thing to do, we keep adding poker chips to the stack without asking ourselves where the stack should be taking us. “Ugh, sharing code with other developers is so tedious, let me create a dependency manager”. In the tiny context of “sharing code with other developers”, a dependency manager may be considered a great solution, but is it a good solution in the global context of programming, or even better, in the global context of solving problems? Are any of the ad-hoc solutions that we have been stacking for the past few decades good in the big scheme of things?

“Wait, do we have a big scheme of things?“

No end game

No, we don’t have a big scheme of things! And that is exactly our problem. Since the very beginning of computers we have been in this vicious circle, exclusively solving what was right in front of our noses, the poker chips right below our feet––problems that we created in the first place––without ever questioning if those were the right problems to solve to begin with. Without ever questioning the context in which the problem was presented. Without ever questioning whether the solutions we came up with were a step in the right direction, following the right path that would get us closer to the overall end goal.

Of course we have never questioned any of this. We don’t have a direction, a path or an overall goal for programming!

We are chasing our own tail while complaining about it while making our tails more difficult to chase while being unaware of this situation. This is what progress in programming looks like.

Let me ask you this: What would programming be to you in an ideal world?

We don’t know! We’ve never asked ourselves this question!

We’ve been so excited with the possibilities that we haven’t weighed the caveats. We manage them in an ad-hoc fashion, but don’t take them into consideration as a problem of their own right because we see programming as a solution — not a problem.

And this goes on today––the web is a great example. The web brings so many wonderful things to the table that we can’t see how horrible and ill-defined the technology stack powering it is. If something as important as the web were done right, we could be doing so much more! It could be so much better, so much cheaper and so much more accessible for everyone! Now, the programmers who have to deal with that mess need to go back and waste time fixing the web and, because evolvability was never take into account, we can only make very tiny, local and incremental steps since we cannot stop the system.

Truth is, defining a goal for the progress of programming would be liberating — every programmer out there would have something to shoot for and a frame of reference for whether his or her ad-hoc solutions, libraries, frameworks, languages or tools were inline with that end goal.

But without an end goal, of course you have millions of developers releasing Javascript frameworks every day––everyone is pulling in whatever direction they may consider good. Innovation in programming is a big playground, with shiny things to play with, puzzles to solve, no rules, no schedules, no purpose––just play. This results in many more problems, like the huge sea of code impossible to navigate formed by open source libraries, frameworks, bug reports, deployment tools… for God’s sake, there are even companies built around these things! And of course, this gives us more rope to hang ourselves with. “Which JSON parsing library should I pick? Forget it, I’ll make my own”.

This problem is aggravated by the following fact: developers secretly love all the mess they are in, we are like pigs in the mud. We won’t tell you, but all these complex, yet uninteresting puzzles where we spend most of our time? Those are a lot of fun to solve! Since the beginning of programming, the role of the programmer was relegated to fit the impossible into the slow and clunky hardware: “And in those days many a clever programmer derived an immense intellectual satisfaction from the cunning tricks by means of which he contrived to squeeze the impossible into the constraints of his equipment”[11]. That was Dijkstra in 1972––little has changed since then. We may not know this consciously but it is not in our best interest to solve the problem of programming, we want to play.

On top of it, to improve this situation, programmers would be the only ones able to participate in the solution. Well, let’s add electrical engineers for good measure. Never mind that users are the ones paying the consequences, there is nothing they can do. Users are going to have to wait until we decided we’ve had enough fun, if that ever happens.

All right, all right. I may make it sound like an evil plot, where programmers are capricious people who deliberately enjoy cracking puzzles while ruining your life. Not true, we don’t do it deliberately. Really, we are not even aware of the mess we’re in! For the most part, all we see are the local problems that exist right in front of our noses and nothing else. “XML sucks, let’s do JSON instead”. We don’t perceive them as part of a larger problem. All we see is the poker chip right below our feet and we are not aware of the zillions of chips right below that one. We solve each local problem in order to continue shipping features to you. As an industry we have never asked ourselves, “are data exchange formats necessary at all?” or “are any of the problems I deal on a daily basis essential to programing?” or “instead of solving this local problem, could I just get rid of the problem to begin with?”. We will never ask ourselves these questions because we are busy shipping features to our users. But the fact that none of this is done deliberately does not make it less of a problem, it just suggests that we are not evil.

In any case, we can’t really blame the developer trying to solve a piece of accidental complexity in an ad-hoc manner. Nobody hired her to “solve programming”, she was trying to do her job and release a feature to users. We can’t blame the company trying to make a profit, companies are created for that purpose and this seems to be a lucrative opportunity. We got to this point organically, over the decades, one little decision after another, and we never realized we were creating a big problem due to our limited perception and scope. Just like frogs being slowly boiled alive. Ribbit! 🐸[13]

From the perspective of a developer, one more programming language or one more javascript library or framework is a solution to a (somewhat local) problem, not a problem in and of itself, per se. A single developer would rarely perceive any negative consequences in his or her lifespan, at least consequences big enough to make him or her aware of the magnitude of the problem. You’d have to zoom out several decades to see a strong relationship between cause and consequence. Those consequences, like bugs or the ridiculous amount of time it takes to build software, are diluted in our day to day and have been engrained in our society little by little. Moreover, with those problems came huge gains in technological advancements, so we never paid attention to the bad side of things. Now, it’s been five decades and everyone has a smartphone in their pocket. The cause of our trouble is deeply engrained in our lives, there’s no way we could roll that back (not that we should, just that we couldn’t if we wanted to).

These perception problems draw parallels to other big-scale problems we face. Global warming, for instance. The consequences also happen so incrementally that a single person cannot perceive them. Also, the actions of a single individual are not enough to produce any visible consequence, so it is going to be difficult to convince each individual to change their behavior. (Not looking to compare the magnitude or consequences of programming and global warming, just the perception problems 🐄💨)

Is it a problem worth solving?

Now, if you were to accept programming as a problem, hence, something that we could attempt to solve, you’d have all the right to ask: is it worth solving? Solving a problem takes time and we could spend that time creating more software that would solve more problems. Fair point. I’m going to say yes, I think it is worth solving. Let’s see why.

Remember how we got into computers in the first place? We built the computer so we could tackle more complex problems. Oh the irony! I think we face the same problem all over again. We have gotten to a point where we can no longer solve bigger problems with software effectively. If we can’t manage to make a very simple iOS app without bugs, who is telling you we can manage to make car software without bugs? We want to fly rockets and go to Mars, all this, controlled by software. That software could easily be the most complex ever built, are we sure it will work? How many more airplane or car crashes caused by software bugs do we need to convince ourselves that programming is a problem worth solving?

Remember why we invented machines in the first place? We were acknowledging the limitations of our human condition. We had trouble counting, we invented the abacus, we had trouble keeping tabs with the time we invented the clock. Well get this, in modern programming, the human is, yet again, the weaker link. Building software is 90% a human problem. Why? A software program can be so complicated that a single person cannot handle it, requiring teams of people to build it. The bigger the problem, the bigger the team. Now we are in the business of managing the (exponentially growing) communication channels in between developers. We have to ensure that everyone stays in sync to create a software system. To make it worse, we cannot point, see or listen to whatever part of the program we want to discuss in order to facilitate this communication, we are working in the world of ideas, remember? All this person-to-person communication and its coordination is a huge source of mistakes, but even if done perfectly, it would still take a lot of time. Have you ever played telephone?

Today, people are the fallible part of building software. Just like we use to do in the past, we should acknowledge our limitations and find a solution.

Plus, we could be doing so much more with all that time. You were wondering if putting some time aside to “solve the problem of programming” would be worth it. Well, developers spend 50% of their time finding and fixing bugs. If we invented better tools to create bug-free software that were easy to use, you could do twice as much for the rest of time. That alone seems appealing to me, but that is just the tip of the iceberg. If we reduced the need for communication, you could spend all that time creating more software to solve other problems. Additionally, if we made programming so accessible that regular users could make modifications, you could spend less time adding and shipping features for only a small subset of users — they could do that by themselves. Furthermore, imagine a world where the 20 million developers on earth today are freed from creating mundane and repetitive applications and they are put to solving the problems at the edge of human knowledge––the possibilities are immense.

Computers have brought us tremendous benefits. All the things we have created thanks to computers and all the problems we have solved are truly incredible. But that’s exactly why we should solve this problem. There’s so much more we could accomplish but it’s unlikely we’ll get there with our current tools. For example, there are a lot of great expectations around Artificial Intelligence, like human-level AI and beyond. Imagine that we managed to make AI with the level of intelligence of a monkey––that would certainly be a huge technological breakthrough. Now imagine that we all went to play with the AI monkey and we completely stopped all development in the field of AI. You’d be pissed, right? You’d expect some people to keep working on improving AI because it could be much better, right? This is how I feel about computers and programming––there is an opportunity cost to not solve the problem of programming.

So, if you ask me whether programming is a problem worth solving, my answer is a straight yes. From a quantitative point of view, all the time saved in productivity in the years to come would outweigh the cost for finding a solution to programming. The returns will be much greater than the costs.

Now from a qualitative point of view. Back in the day math was slow and error-prone and we created the computer. Now, it’s software development that is slow and error-prone. Back in they day math powered innovation. Today it’s software that powers innovation. Solving the problem of programming is the next logical step.

The computer was created with practicality in mind. We must stick to this sentiment. We’ve used computers to create amazing things so far, but we shouldn’t stop pushing the envelope. I am not suggesting that we stop the presses, have all programmers stop writing code and figure this thing out. We should continue extracting value from computing, but we must align our initiatives if we want to keep harvesting that value at a scale.