Last time, we went all the way from the Morse Code to the creation of Unicode. And I’ve left you there with a question. How can you fit millions of characters into an 8-bit encoding? Wasn’t it exactly how that wild ride began — with short code encodings?

As promised, we’ll now dive into the more technical part of the Unicode. I’ll try to keep it relatively simple, but it’s going to be more complex than the previous part.

Since we’re talking about programming stuff, here is a photo of some hands typing on a computer keyboard

Fitting more into less

It isn’t completely right (nor is it wrong) to say UTF-8 is an 8 bit encoding because it’s a variable-length code. As the name suggests there won’t be a single length the code uses. In the case of UTF-8, you get code length varying from 8 do 32 bits.

But wasn’t this the problem with the Morse Code? If one letter uses longer code than the other, how can you know which one is which? It’s actually not that hard.

Make it variable-length

First, we have to remember that we’re operating on full bytes (which are 8-bits each), so we can only use code lengths of 8, 16, 24 and 32 bits. With this knowledge, we can sacrifice some bits for the purpose of storing code length in them. Let’s first take a look at the reference table, and I’ll follow with an explanation.

If it’s all clear to you, feel free to scroll all the way to the next chapter

Let’s focus on “Byte 1” column. We can see that depending on a number of bytes we use, the code begins differently. 4 bytes code starts with 4 ones, 3 bytes with 3 ones, 2 bytes with 2 ones, and 1 byte is different.

Anyway, as you might have guessed, the first byte tells us how many bytes should be read as one character. If you see something beginning with “1110…” you know that you need to read two more bytes to get a whole character. And as you can see those “next” bytes will all begin with “10…”. If you start reading a character and it begins with “10” then you instantly know something is not right. But you already know how it should begin, so you can skip a few bytes until you find one starting with anything other than “10…”. And that’s a great feature of this encoding, as it allows easy and fast processing and error recovery (you know, data get lost in the Internet sometimes).

Sacrifice some bits

However, you may be worried, that we’re losing some precious space by sacrificing bits of information for such things. And we are indeed. The fixed-length 32-bit code could theoretically encode 4 billion characters. But as we see in the table above, UTF-8 only gives you 21 bits for code point. Anyway, it’s still over 2 million characters, enough to meet Unicode standard (which defines a maximum of 1112064 characters). The limiting factor here is backwards compatibility with UTF-16, which is the oldest of Unicode encodings.

But even those 1,1 million characters is more than enough. Unicode 9.0 uses just 11% of all available space (128237 characters to be exact), and we’ve already encoded pretty much all available languages and scripts including Egyptian Hieroglyphs, Domino and Mahjong tiles, and of course Emojis — the language of the 21st century 👌😂

Keep it compatible

But why is one byte code different from the others? Why didn’t they do it all nice and satisfying?

The answer is backwards compatibility.

People creating UTF-8 wanted it to be as easy to use as possible. If most encodings were based on ASCII, then it would be only reasonable to make it ASCII-compatible as well. The way they did it makes all ASCII encoded texts also UTF-8 encoded. This made switching to Unicode much easier and is one of the great things about this encoding.

Enough with the theory. Let’s see it in practice.

Let’s encode

The good start to lay base ground for required calculations is trying to encode something from ASCII, because it’s supposed to fit into one byte.

For example, let’s take a letter “D”. If we check the ASCII table I’ve posted in the previous post, we can check that it’s code is 68, which is ox44 in base-16 (hexadecimal codes are actually used more commonly, so we’ll stick with them). Now we should check Unicode table to find out that “D” is U+0044. “U+” is a prefix for Unicode codes, and the rest is a hexadecimal value. If you know how to convert from base-16 into binary, you know that 0x44 = 01000100. Otherwise just believe my words.

So, we already know the code for “D” in ASCII. And if we look at the table I’ve posted above, we can see that U+0044 is lower than U+007F, so this letter fits into one byte. So the code for “D” in Unicode is also 01000100. Just as expected.

We can now do the same for letters “o” and “g” to encode the word “Dog”. It will be the same in both ASCII and Unicode: 01000100 01101111 01100111. Notice how each letter starts with a zero — this is an easy way to tell we’re dealing with ASCII here (which is equivalent to 1-bit UTF-8).

Time for something harder

But let’s be serious. It’s 2017 and nobody writes “Dog” anymore. Nowadays we use a dog emoji “🐶” instead. Emojis didn’t exist in ASCII so we go straight to Unicode table to find it’s code: U+1F436.

Looking at the conversion table, we’ll need all 4 bits to encode it, because U+1F436 is bigger than U+10000. After converting to binary we get 000011111010000110110. There are 4 zeros in the beginning to make it have a required length of 21 bits.

We can now change “x” in the conversion table into our binary digits and get an UTF-8 code for “🐶”: 11110000 10011111 10010000 10110110. Numbers written in bold are those, we’ve calculated above.

As we’ve expected. No single byte starts with a zero, and all bytes except for the first begin with “10”.

What’s so hard about it?

If we look at what we’ve already learned it all looks relatively simple and well thought. After all, we’ve just manually encoded some data. Why would there be any problems with it? There are three reasons why.

Being compatible

The first one goes all the way back to 1960’s when the ASCII was created. Since it became the de-facto standard base for all future encodings, there was no reason for Unicode to break it. So it was created to be compatible with ASCII. And it was a nice little thing. But why limit compatibility to ASCII alone? We can try to make it overlap with as many encodings as possible. And it usually went quite well, but there were a few exceptions.

Do you know what character had a 0xB5 (decimal 181) code in standard Extended ASCII? It was µ — a symbol for micro. And do you know what is the 12th letter of the Greek alphabet? It is of course μ — the same character. Or is it?

This StackOverflow post is an explaination by itself

You see, the first character (we’ll call it “micro”) was kept as U+00B5 to make Unicode and older encodings work together nicely. It looks the same, and actually comes from the greek letter μ (which is called “mu”), but it has a different meaning. Since Unicode doesn’t group characters by their appearance, but rather divides them into groups of the same origin and context. This means that there is a letter μ again under U+03BC, where all the greek letters are.

This is of course well defined in Unicode standard, and the micro character is marked as “compatibility”. It is recommended to use small greek μ instead, but try and tell it to people using your application.

It looks like I’ve written all those posts around xkcd comics. But it’s not my fault there is always a relevant one.

Supporting human languages

The second one goes back about 8000 years back. Yes, it’s no mistake here. Human languages and writing systems themselves are probably one of the hardest things to standardise. There are thousands of languages, using hundreds of writing systems. Through history, many languages have diverged from others making them similar but not quite the same. Writing systems often evolved one from another making them share some symbols, but not all of them.

Time for an example. Let’s say you’re going to visit the Netherlands. You board a plane, and after a few hours, you arrive in Amsterdam. We all know its name begins with the letter “A” — the first letter of an alphabet. But when you look at the map you can find many other cities. For example, Astana which is the capital city of Kazakhstan. Or if you have a map which happens to be in Russian look for “Астана”. This first Cyrillic letter looks a lot like a letter “A”, doesn’t it? But we know it can’t be the same letter — after all, it’s from a language using a different alphabet. Or you may find yourself in sunny Athens, the capital of Greece. But if you’ve bought a Greek map, you’re looking for “Αθήνα”. And again, this first letter looks like “A”, but we know it’s a greek capital letter alpha.