Grand Rearchitectures, Interlocking Plans

I have come to identify a pattern that crops up in proposals for business models, social engineering, computer architectures, etc.

It is this: instead of paring things down to the minimum (Antoine_de_Saint_Exupery's "nothing left to take away" / Steve Blank's "minimum viable product"), people propose large steaming piles of things which are (a) incompatible with what came before, and (b) depend on every component working flawlessly.

This is, in general, a doomed strategy – which you will note if you have ever had the misfortune to liked your healthcare plan and chose to keep it.

Some crazy proposed business models operate this way ("we teach the natives to harvest rain forest fruit in a sustainable way, then float the goods down the river to market on carbon-fiber-and-help catamarans with help rigging built with micro-loans from our new website…").

Some crazy proposed social revolutions operate this way ("after we cut off the heads of anyone with royal blood, we cut off the heads of anyone who objects to cutting off heads, then we turn the Cathedrals into Temples of Rationality!").

etc.

So, when I saw a software architecture proposes much the same thing ("we build a new virtual machine called Nock VM that's entirely incompatible with the existing standards, then we create a new language to run in it (also called Nock), then we build a higher level language on top (called Hoon), then on top we layer an operating system (called Urbit), encryption, namespaces, and delegation of privileges ….based on neo-reactionary politics! Oh, and also, we have a customizable UI that not only gives error messages in phrases you like, but it lets you turn political enemies into unpersons. And, wait, wait, I'm almost done: also I've got a new way that you've got to pronounce combinations of characters…so the characters '|:' are pronounced 'bardeg'. ") I was fairly dubious.

I note that more Frenchmen vacation in July than do in Thermidor.

…and Yet

After 10 minutes of reading the Urbit documents I was sure that it was technically plausible but practically idiotic.

Why would anyone want to throw away the current technology stack (x86 CPUs running Linux running either C++ that compiles into native code or Java that runs inside a Java Virtual Machine that is implemented with C, all communicating with each other using reliable TCP/IP) in favor of a pile of not just unproven but as-yet unwritten technologies ( x86 CPUs running a new virtual machine who interprets a beyond-cryptic tree-based programming language called Nock which is used to implement an also-beyond-cryptic language called Hoon, which is used to implement a new operating system called Urbit, all of which talks to other instances using the unreliable UDP protocol?

It's madness.

If the author of this monstrosity, Curtis Yarvin, had any justification for this insane project he was silent.

And that sentence right there explains why I spent more than ten minutes on this danger-Will-Robinson-attractive-nuisance thing.

Curtis (aka Mencius Moldbug), is brilliant. Sadly, like a few brilliant people I know personally, he is also elliptical. I don't know if this is because his surplus brainpower finds word-games and allusions to be an irresistible attraction, or because he honestly doesn't realize that most folks can't see what he's getting at.

So, anyway, having read Yarvin's political stuff and knowing that he has a habit of throwing away conventional thought in service of reaching deep truths, I gave Urbit more time. I read the documents, thought deeply about them, and let it percolate.

I Think I Know What He's Up To

In general, there are two reasons why people want to throw away everything that has gone before.

The first, as covered in the examples above of redesigning the health care system, redesigning the calendar, redesigning our conception of who is and is not human, redesigning our transportation systems, is egoism, contempt for local knowledge, and a desire to have the sheer fun of redoing everything from scratch.

In my experience, this explains the human motivation behind the majority of all ab initio rewrites.

There is a second explanation though.

If you face dozens, if not hundreds, of different measures of length, volume, and area not just across national and cultural boundaries, but even as you move from village to village within France , then a slow convergence of Parisian elles and Avignone elles – all while keeping "the king's foot" constant – does not solve the problem so much as keep the old problem and add a whole class of new problems.

No, there are times when a man must spit on his hands, hoist the black flag, and hoist the non-compatibility flag that forever demarcates the Ancien regime from the new.

Yarvin may or may not be an egoist (I've never met him), but he's not an idiot. If he's proposing throwing away 50 years of software stack, he's got a good reason…or at least a very very interesting one.

As the perhaps-apocryphical army officer review goes: "His men would follow him anywhere…if only to see where he was going".

So what problems is Yarvin trying to solve? And how does he want to solve them? I'll follow him, at the very least for long enough to find out.

He doesn't crisply say, but I've got my thoughts.

Out of the crooked timber of legacy computation, no straight thing was ever made

(with apologies to Kant)

Subsumption architecture is a concept in robotics which – and I'm being sloppy here – designs systems by making logical building blocks. One block solves problem X, one solves problem Y, etc. Instead of making a hyper-intelligent robot software stack which thinks deep thoughts about about chess and about climbing stairs, you build a lizard hind-brain that knows how to climb stairs, and then you build a chess playing robot which wants to walk from here to the chessboard, and lets its lizard hind-brain deal with any stairs in the way.

This is a clever approach. There is a problem though: leaky abstractions. (By the way, note that the example Joel chooses to use is …TCP. Just like I've mentioned here twice so far. This is not a random coincidence.)

In theory, we can take a database, slap an API on top, and declare that it is a device for storing data, just as we can take a refrigerator, slap an API on it, and declare that it is a device for making things cold.

In practice, though, there's a difference between theory and practice. A refrigerator behaves markedly differently when you put a single gallon of already cold milk in it than it does when you load it with 50 pounds of steaming freshly-butchered venison. Despite what the API claims, in the latter case, it ceases to be a device that quickly makes things cold.

To deal with leaky abstractions one has to either (a) expand the abstraction with all sorts of implementation-relevant details, thus exposing all the messy crap that you hoped to hide (suddenly a chef can no longer think of their refrigerator as "a cold making device" and has to think of it as a huge pile of engineering specifications), or (b) come up with a better device that truly does have an implementation that matches its specification.

For half a century we have used the first approach: we've connected computers with inherently flaky communications channels, we've tried to paper that over by adding redundancy and retry-algorithms on top (turning UDP into TCP), designing programming languages that blow up in certain situations and then pretending that they don't, etc.

You can build a shoddy foundation and build a one bedroom house on top of it and no one will get hurt.

You can maybe build a second story on top of that first house.

You probably shouldn't build a third.

The further you build, the more time you're going to spend fixing, fixing, fixing the foundation.

This is not an acceptable solution in the ancient cities of Mars or in Paul Graham's own personal Clock of the Long Now.

At some point you really need to rip the entire legacy of 1950s hackery down, sink a hole down to bedrock, and pour a foundation of blast resistant cement.

So, seriously, what problems is Yarvin trying to solve?

I've not had time to read the Urbit message boards, so this is speculation – but I'd wager a federal reserve note or two that I'm right.

As we move from single computers sitting alone in machine rooms to lots of PCs on desks, up to the current emergent mesh of billions of machines of all types in everything from light switches to coffee makers to gaming consoles, the scale changes.

We are no longer trying to make one machine solve one problem so much as make billions of machines (and trillions of ad hoc, short-duration pairings and groupings) work well.

When you log into your bank account in a web browser you're engaging a dozen or more machines and asking them to cooperate to do a job.

When you go to Google to ego-surf your own name you're doing the same.

When you have a LAN party (as we old folks used to call them) or play a MMORPG, you're doing the same.

When you check a bitcoin chain – well, you get the point.

The point is: we know that in the future even the most trivial act of computation will be spread across multiple machines. We know this because it is already true.

Spreading the act of computation across multiple machines is tricky, because of (a) the assumption of network connectivity even in the fact of the reality of network downtime, (b) the less-than-fully-specified state of an ongoing computation on any given machine, (c) the inability to tolerate failures and pick up partially complete work.

How Does Urbit Address These Three Issues?

First, a digression into a taxonomy of programming languages.

For the lawyer, the plumber, the physics PhD, all programming languages are alike: they're a sequence of letters in a text buffer somewhere that tell the computer what to do, and details beyond that are boring.

Well, no – the details aren't boring, they're fascinating. (I say that not because I have some special affinity for programming languages, but because almost any complex system is fascinating once you understand the problems, constraints, and solutions that drive it. I could watch "How It's Made" videos for hours. Ahem. Correction. could do.

There are dozens of different types of programming languages. By "type" I mean something akin to "schools of thought". Keynesianism vs Free Market. East Coast Rap vs. West Coast. Etc.

One of the most common language types is imperative languages. "Imperative" from the Latin imperativus, of course: to dictate, to give a peremptory command. See also: emperor. Imperative languages micro-manage the CPU. They don't tell it that health care should be reorganized, they tell it – ahem. Excuse me. Anyway, imperative languages tend to look like this:

monthly_interest_rate = 1.001 define calculate_interest(loan_amount, months) running_total = loan_amount do_it months times: running_total = running_total * monthly_interest_rate end print "total due: %f", running_total end

There are other common types. SQL is a database query language that is declarative (these language "express the logic of a computation without describing its control flow.")

In a declarative language one might write:

loan_amount * (monthly_interest_rate ^ months)

and let the language implementation figure out the best way to actually do the math.

In a sense, all of these languages are equivalent: the concept of Turing completeness means "any problem you can solve in language X I can solve in language Y".

Of course, in a sense, spoons are the same as shovels are the same as bulldozers. They can all accomplish the same job, just some will get the job done before the heat death of the universe / the workers rise up / the pile of hack built upon hack built upon hack collapses under its own weight.

It is the last that we are concerned with: what type of programming language is conducive to solving huge distributed problems without collapsing under its own hacks?

We want a language which:

will operate the same way if its doing its fraction of the task on node #1 or node #16,180,339,887.

will operate the same way if it is run today or in five years

will operate the same way if you hand it inputs of 1,2,and 3 while you're also playing an mp3 of Mozart as it will if you hand it inputs of 1,2,and 3 while you're also playing an mp3 of death metal

can either do a sub computation on the same node or delegate that sub computation to some geographically foreign node

will operate the same way if it pauses mid-computation to get a piece of data from a companion machine that delivers the datum in seconds or after three weeks of retrying over a crappy network

In short, to touch again the conclusion of the previous section, we want a programming language which lets us split computation up across multiple machines, run those sub computations in repeatable and fully specified ways, and deal with the fact that networks go down.

Q:Is there a class of languages that makes this possible? Because if there is, not only can we build space elevators (metaphorically, certainly, but perhaps also literally), but we can do it without spending any time worrying about all the crap of the current computational infrastructure which always lets us down.

A: Yes, Functional Languages

Functional languages are weird. Mandarin level weird (a quick aside: one would almost expect a language named "Mandarin" to be imperative, but that's crossing a stream too far).

Functional languages are closer to imperative languages (slogan: "do this…this particular way") than they are to declarative languages (slogan: "get a result that does X"), but their true weirdness is the way in which they handcuff the CPU.

If languages were people, your typical imperative language is a ditzy teenager with a sloppy desk: there are lots of post-it notes around, lots of half-remembered facts, a cork-board at the front of the room with some notes, etc.. Ask a program in a typical imperative language to do a very specific task (e.g. compute total payment for a loan of principle X with interest rate Y after Z months) and the internal monologue would be something like:

"I know that we give special discounts to our best customers. Who is this customer? Ah, there is it, pinned to the corkboard: this is for a partner of the firm. OK, so he gets the discount rate. And the discount rate is…what? Let me look it up in this file cabinet. OK, now I need some coffee, so let me write the discount rate down on this post-it note, get coffee, and hope that no one changes my post-it note before I return…"

There is a lot of looseness and flexibility in typical imperative language that makes a programmer's job pretty easy. If the function to compute loan payoffs accepts just three variables (initial amount, rate, months), and someone later comes along and asks for the special friend-of-the-firm feature to be added, there is no need to rewrite the function and everything that uses it to now pass in four pieces of data.

Looseness and flexibility is absolutely wonderful when you're building a hobbit hole: "this branch curves oddly…let's make a porch out of it!"

Looseness and flexibility is not called for, or even tolerated, when we're building space elevators. For that task we need a language that is brutal, uncompromising, and hard as concrete. Like Le Corbusier's plan to change human nature via modernist architecture. Or the rhubarb pie I attempted to make that one time Mrs. Clark let me into the kitchen.

Or math.

If the flexibility and ease of use of typical imperative languages come from sloppy data passing, then what does rock-hard Vulcan data passing look like?

Like this:

define calculate_interest(loan_amount, months, monthly_interest_rate) // Q: what if it's a leap year? // A: too bad; we only get to use the three pieces of data passed in // Q: what if it's for a partner of the firm? // A: too bad; we only get to use the three pieces of data passed in // Q: what if the underlying asset burned down? // A: too bad; we only get to use the three pieces of data passed in // Q: what if the customer has a discount coupon? // A: too bad; we only get to use the three pieces of data passed in // Q: what if the customer declared bankruptcy? // A: too bad; we only get to use the three pieces of data passed in // Q: what if the bank is being sued for racial discrimination in loans? // A: too bad; we only get to use the three pieces of data passed in // Q: what if the bank clerk hits control-C to override? // A: too bad; we only get to use the three pieces of data passed in total = loan_amount * (monthly_interest_rate ^ months) // Q: what if we want to store this total in a database for later use? // A: too bad; we only get to return one piece of data and zero side effects return total end

Once we add all those comments in we start to realize that there are VAST numbers of things that this function probably wants to deal with but can not because it's constrained by some choices we made earlier.

We have two choices: we can either decide to eliminate those complications from our business / computation model, or – more likely – we can write code that simultaneously solves the actual business problem and crisply acknowledges all of its inputs.

If we rewrite the code to deal with leap years, partners in the firm, bankruptcy law, etc., the code is going to much more complicated.

Bad, right?

No! That's a good thing. The complexity was there all along, but we were hiding it from ourselves.

Why, though, is it a good thing to make the complexity explicit?

The answer is that when complexity is implicit we might be unaware of the tight couplings that different parts of a system have. To get back to the analogy of the worker with the sloppy desk, if we're using not just information that was explicitly passed to us, but information that is pinned to a corkboard at the front of a room, and information that we have written on post-it notes, all sorts of problems develop. In the analogy, we run into problems if the worker freezes up for an hour: the post it note might change, the corkboard might change. We run into problems if we give up on worker #1 and hand the task off to worker #2 who sits in a different room with a different cork board and a different stack of post-it notes. We run into problems if we want to double check the computation three years later and the room, the corkboard, and the post-it note no longer exist.

If we very crisply define input and output, and remove side effects and data leakage from our systems, we get all sorts of very nice higher order tools that we can use in a world of billions or trillions of computing nodes coupled together by crappy wires.

Yarvin has created Nock, a very ugly, very weird language that runs on absolutely no hardware at all…except inside his own interpreter / virtual machine. Nock is not nearly as human-readable as the examples I've put above; an actual simple Nock statement (from the docs!) reads

*[a 10 [b c] d] *[a 8 c 7 [0 3] d]

Hideous stuff.

Too Hideous?

Nock code is weird, no doubt.

Yet the code at the bottom of our modern software stack is hideous too. At the lowest level, the machine you're sitting in front of runs assembly language programs that look much like this:

IRQ LDA #$00 STA CLICK LDA $C5 CMP #$01 BNE CONCHECK STA CLICK CONCHECK CMP #$0C

Software engineers do not actually deal with ugly code like this – they use much higher level code like the examples you've seen above. If we can cover over machine code with assembly, and assembly with C and C with Python or Ruby or Go, then surely we can cover up the ugliness of Nock with something cleaner…and, in fact, Urbit proposes to do exactly that, with Hoon (although my own aesthetic sense does not yet actually see Hoon as being all that beautiful…but that's a different point).

So sheer ugliness alone doesn't make us reject Nock. We can slap it with a coat of whitewash – or better yet, like Huck Finn, get someone else to whitewash it for us, right?

…but Nock isn't just ugly. It's weird. So weird that the whitewash may or may not cover it.

I put a bit on the screen above in a linear format, but Nock is not really a sentence-like flow of words like C, COBOL, Russian, Linear-B, or even the boustrophedonic Rongorongo. No, Nock programs are tree structures.

This is not unprecedented – Lisp ("The greatest single programming language ever designed.") does too.

And here – suddenly – the conceptual Legos start clicking together.

Because a Nock program is functional, it operates without caring what machine its on, what time it is, what the phase of the moon is.

Every Nock program is a tree, or a pyramid. Every subsection of the tree is also a tree. Meaning that each subsection of a Nock program is a smaller Nock program that can operate on any machine in the world, at any time, without caring what the phase of the moon is. Meaning that a Nock program can be sliced up with a high carbon steel blade, tossed to the winds, and the partial results reassembled when they arrive back wafted on the wings of unreliable data transport.

Nock programs – and parts of programs – operate without side effects. You can calculate something a thousand times without changing the state of the world. Meaning that if you're unsure if you've got good network connectivity, you can delegate this chunk of your program not just to one other machine, but to a thousand other machines and wait for any one of them to succeed.

Nock supports and assumes full encryption of data channels, so not only can you spread computation across the three machines in your home office, you can spread it across three thousand machines across the world.

The list goes on and on.

Envisioning and defining Nock took a stroke of genius. Implementing it, and Hoon, and Urbit, will be a long road.

But once it's all done, it will function like an amazingly solid, square, and robust foundation. All sorts of things that are hard now, because we have built our modern computational civilization on a foundation of sand will become easy. We have vast industries based around doing really hard work fixing problems that modern computing has but a Nock infrastructure would not – Akamai, for example, pulls in $1.6 billion per year by solving the problem that modern URLs don't work like BitTorrent / Urbit URLs.

When an idea, properly implemented, can destroy multiple different ten-billion-dollar-a-year-industries as a side effect it is, I assert, worth thinking about.

Switching from Dominos to Legos

Back in the early days of the internet when Usenet was cutting edge, there was a gent by the name of Timothy C May who formed the cypherpunk mailing list.

His signature block at the time read

Timothy C. May, Crypto Anarchy: encryption, digital money, anonymous networks, digital pseudonyms, zero knowledge, reputations, information markets, black markets, collapse of government.

I bring up his sig block because in list form it functions like an avalanche. The first few nouns are obvious and unimportant – a few grains of snow sliding. The next few are derived from the first in a strict syllogism-like fashion, and then the train / avalanche / whatever gains speed, and finally we've got black markets, and soon after that we've got the collapse of government. And it all started with a single snowflake landing at the beginning of the sig block.

Timothy C May saw Bitcoin. He saw Tor. He didn't know the name that Anonymous would take, and he didn't know that the Dread Pirate Roberts would run Silkroad, and he didn't know that Chelsea Manning would release those documents. …but he knew that something like that would happen. And, make no mistake, we're still only seeing small patches of hillside snow give way. Despite the ominous slippages of snowbanks, Timothy C May's real avalanche hasn't even started.

I suggest that Urbit may very well have a similar trajectory. Functional programming language. Small core. Decentralization.

First someone will rewrite Tor in it – a trivial exercise. Then some silly toy-like web browser and maybe a matching web server. They won't get much traction. Then someone will write something cool – a decentralized jukebox that leverages Urbit's privileges, delegation and neo-feudalist access control lists to give permissions to one's own friends and family and uses the built in cryptography to hide the files from the MPAA. Or maybe someone will code a MMORPG that does amazingly detailed rendering of algorithmically created dungeons by using spare cycles on the machines of game players (actually delegating the gaming firms core servers out onto customer hardware).

Probably it will be something I haven't imagined.

And then, five, ten, or twenty years from now, the new architecture will really start catching on. More and more computation will slide into the black, beneath the waves. SilkRoad will have no central server – parts of it will be running our your machine, copper-top. Amazon will still be in the cloud computing business…and so will your wrist watch. All sorts of interesting problems that we don't even think about right now because they're so intractable will become easy.

…and that's even without taking into account the parts of the system that I haven't talked about, like version control built into resources, etc.

In short, if Urbit works as designed, the world will get weird.

At least, that's my take on it.

Last 5 posts by Clark