Hey, it’s been a while since my last post, so I thought a quick reminder of what’s going on here would be helpful. We’re dabbling in reverse engineering, and visual novels are our target. In the first part of this article, we’ve learned to analyze container formats using marvelous Kaitai Struct tool, which makes it pretty easy for just about everyone who knows a little about computers to try their hand in proper “clean room” reverse engineering, without deep knowledge of CPUs, registers, disassembly, etc.

Our main target is still Koisuru Shimai no Rokujuso, a lesser known, but still nicely done visual novel from PeasSoft. Last time we’ve discovered that it uses an engine that’s probably called “Yuka”. We’ve successfully extracted game files from .ykc containers, got our hands on visuals, sounds and music, but the final challenge still awaits: the script.

Meditation on novel’s internals

Before diving into the vast depths of binary dumps, let’s take a moment to reflect on how most of the visual novel engines work. Visual novel consists of the text (dialogues, narratives), visuals and sounds. To make it all come together, some sort of a control program is needed. Technically it is possible to just stuff it all as machine code in the middle of .exe file, but in 99% of cases (ok, I’m lying — in 100% of the cases I’ve seen by my very own eyes) engine authors don’t do that. Instead, these control instructions are kept as a separate file in some sort of (improvised) domain-specific scripting language. That’s what they universally call “the script”. Let’s take a peek at how it may look:

$ tarot = 0

$ memory = 0 scene bg01_1 with dissolve play music "bgm/8.mp3" fadein (2.0)

play ambience "amb/forest.mp3" fadein (3.0)

"Morning."

"Not my favourite time of the day."

"The morning is when you’re not awake enough to do anything…"

This is a small fragment of script’s source code of one of the VNs using Ren’Py — that’s one of the most well-known free-as-in-beer & free-as-in-speech engines. Leaving aside the question of whether it’s a good or bad idea to use Ren’Py per se, let’s just investigate what’s typical for visual novel’s script (and that’s exactly what we’ll be looking for):

the text — it might be the narrative (i.e. not spoken by a character, but part of the story told by a narrator) or some dialogue lines (attributes to the certain character)

— it might be the narrative (i.e. not spoken by a character, but part of the story told by a narrator) or some dialogue lines (attributes to the certain character) commands to deal with visuals — backgrounds / sprites (`scene bg01_1`) / character graphics / event graphics, sometimes augumenting them with some visual effect (`with dissolve`)

— backgrounds / sprites (`scene bg01_1`) / character graphics / event graphics, sometimes augumenting them with some visual effect (`with dissolve`) commands to play back the music or sound effects (`play music`, `play ambience`), sometimes also coming up with some extra arguments, most frequently with durations of fade-ins and fade-outs (i.e. slow gradual increase or decrease in sound’s volume)

(`play music`, `play ambience`), sometimes also coming up with some extra arguments, most frequently with durations of fade-ins and fade-outs (i.e. slow gradual increase or decrease in sound’s volume) dealing with variables : setting (`$ tarot = 0`), getting, checking conditions and branching

: setting (`$ tarot = 0`), getting, checking conditions and branching some extra stuff, like playing back voice files pronounced by certain characters, flow control and various service features, like comments, labels, macros, etc

Of course, in the real world most of the time we won’t be able to access the script’s source code — that would be too easy. It’s been probably half-a-century already since people learned to create compilers (as opposed to interpreters), so nowadays most likely the source code is being compiled into the executable byte code, which is then executed by the VN engine using some sort simple (or not so simple) virtual machine. Sometimes you’ll get lucky — for some popular engines you’ll be able to get a ready-made toolset — compilers, decompilers, debuggers, script validators, etc. — but most of the time life’s not that convenient.

Let’s get back to our visual novel that we’ve started to dissect in our previous article — Koisuru Shimai no Rokujuso. After unpacking of its archives, we found inside the images, the sounds, the music files, and the most mysterious artifact so far — a handful of files with .yks extension. Presumably, it’s where the novel’s script is (remember, the engine is presumably called Yuka, so it’s YKS = YuKa Script). To be precise, it’s not just one script, but there’s a quite a few of them:

YKS/ScriptStart.yks

YKS/trial/Yoyaku.yks

YKS/trial/trial_00100.yks

YKS/trial/trial_00200.yks

YKS/all/all_00010.yks

…

YKS/all/all_02320.yks

That makes us 103 files total in YKS/all/. Let me remind you that we’ve downloaded the trial version — but, it looks like the developers were kind of lazy to rip the full game content properly and we’ve got trial scripts in trial/ and full game scripts in all/.

Judging from my experience, visual novel authors tend to walk one of two ways: either packing every script bit into one huge file, or making tons of files for every scene / event in the game. Looks like it’s the latter we’re facing here. By the way, note that there’s also a distinct “ScriptStart.yks” — but probably it won’t be as interesting to us as the rest of the files. You see, engine developers frequently want their engine to be as all-purpose as possible and choose to implement elements like UI, saving/loading, CG galleries, menus, options screens, etc using the engine itself. It’s definitely possible to diassemble all that stuff too, but (a) it’s certainly not the best place to start given it lacks any text or other visual hints, (b) it’s not that interesting for our purposes. So, I propose to make a head on start and begin with more plausible scripts.

Where’s my martini, shaken, not stirred?

As in our previous article, we start with basic intelligence first. So, get your Aston Martin ready and let’s scout the surroundings.

First of all, this is a Windows game, so it’s a quite viable idea to just find some Windows box, run it and see what happens. That’s what we get right after hitting the “Start new game” button in main menu:

The very first text that one encounters in the visual novel.

The start of the story greets us. We’ve got a background here (half-a-minute search reveals that it’s a “bg01_01.png” from the BG/ directory), and we’ve got some text. Seeking for text is usually a very good idea to start from, so let’s retype it from the screen:

恋する姉妹の六重奏「セクステット」体験版Ver2をダウンロード頂きありがとうございます。

If you have any problems typing Japanese (and I bet at least some of you have completely no idea even where to start), here’s a quick primer on how to do it with some effort in several easy steps.

Typing Japanese in your PC without knowing anything about it

As you can see, Japanese text consists of individual characters (usually fitting into a square), so we take them one by one and:

check if it’s a punctuation mark first using the following line:

「…」、（）。

If you’ve got lucky, just copy-n-paste it. Note that both Japanese “commas”, “periods” and “brackets” are special.

if that’s not a punctuation, look for the character in this line:

あいうえおかきくけこさしすせそたちつてとなにぬねのはひふへほまみむめもやゆよらりるれろわを

then this line:

アイウエオカキクケコサシスセソタチツテトナニヌネノハヒフヘホマミムメモヤユヨラリルレロワ

still no luck? for example, you’ve got 恋 — then we’re doomed, it’s kanji; zoom your font up to 300–500% to make the details clearer and got to jisho.org, “search by radical” section; once there, look closely at the table of the elements that form kanji characters (these are called “radicals”) and look for the parts that look like parts of the kanji; taking 恋 as an example — a quick mediation reveals that bottom part of this character is 心 — pressing down this radical shortens a list of possible kanji to browse dramatically — it’s no longer multiple thousands, but a few dozen; browsing through them, we’re bound to find fifth character in “10” section — that would be the 恋 we’re looking for;

Sure it’s might be time consuming, but it’s better than nothing. Another semi-cheating method is to go to Google Translate, turn Japanese input there, switch into “drawing” mode and try to draw what you see. In 80% of the cases, you’ll be lucky and you’ll get your kanji right away. In other 20%, looking up kanji in a dictionary like a jisho.org is an ultimately foolproof method.

Another note that I’d like to point out is that I’m not sure if that would be “Ver2” or “Ｖｅｒ２” — note that it’s not the different fonts I’m using here, but so called “full-width characters” — that’s completely distinct characters, to be found in Unicode somewhere around U+FF01..U+FF5E — obviously, if we’re comparing binary representations, we need to be exact.

The text we’ve copied will help us in two ways. First of all, we can translate it and find out what’s going on. Even if you don’t read Japanese, you can always stick it into Google Translate or some other automatic translator and get a rough idea. So, actually, it’s not the beginning of the story. Authors thank us for downloading the trial version, so it’s not the “first chapter”, but some kind of a preface, some words from the authors. Second, we can take this line, encode it as ShiftJIS (as we’ve figured out in the previous article, there are really high chances that this engine internals all use ShiftJIS) and seek the files for it. Let’s take a piece of it and do the encoding:

$ echo ‘ダウンロード頂きありがとうございます’ | iconv -t sjis | hd

00000000 83 5f 83 45 83 93 83 8d 81 5b 83 68 92 b8 82 ab

00000010 82 a0 82 e8 82 aa 82 c6 82 a4 82 b2 82 b4 82 a2

00000020 82 dc 82 b7

So, here’s the hex string we need to seek in all our files. Alas, we fail here. Life’s not gonna be that easy.

ShiftJIS: how does it work

Time to make yet another excursion into Japanese PC culture and get ourselves familiar with ShiftJIS encoding. They say that there are as many Japanese characters as stars in the sky. I dunno if that’s the case, but it’s hard to argue that at the very least there are much more characters in Japanese that characters in English. Thus, having 1 byte to encode them all (with 256 possible values) is barely feasible. Thus, ShiftJIS uses at least 1 byte and at most 2 bytes. As you might see from this table, byte values of 00..7F are equal to ASCII (thus making ShiftJIS compatible with ASCII), and byte values of 81..9F and E0..EA mean that it’s a 2-byte combo. Note that for, again, for sake of ASCII compatibility, second byte is not arbitrary, but can be somewhere between 40 and FF.

Shortest crash course in Japanese language ever possible

Looks like we won’t escape yet another dive into Japanese language basics. So, to put it blunt, Japanese uses 3 groups of symbols:

hiragana — look something like ありがとうございます — i.e. simple round cursive shapes; ~50 glyphs, but there are a few variations like “big i = い, small i = ぃ”; 1 syllable = 1 glyph.

— look something like ありがとうございます — i.e. simple round cursive shapes; ~50 glyphs, but there are a few variations like “big i = い, small i = ぃ”; 1 syllable = 1 glyph. katakana — look something like ダウンロード — i.e. square-like, straight lines, pretty useful for typesetting; it corresponds to the same sounds as hiragana, but is mostly used to write down the loanwords (ダウンロード = da-u-n-lo:-do = download).

— look something like ダウンロード — i.e. square-like, straight lines, pretty useful for typesetting; it corresponds to the same sounds as hiragana, but is mostly used to write down the loanwords (ダウンロード = da-u-n-lo:-do = download). kanji — look something like 体験版 — i.e. complex drawings with lots of elements, fit into square shape; most of the time it’s kanji that give the most pain for Japanese language learners, and it’s many thousands of them.

Also, there are a few other symbols, like punctuation. For our purposes (albeit proper Japanese scholar would like to kill me for such a hypocrisy), let’s say that they are mostly equal to those of European languages. There are:

a “full stop” — 。

a “period” — 、

ellipses — …

quotes — 「」

full-width question mark — ？

full-width exclamation mark — ！

and some other, less frequently used symbols

But there’s a catch, as you might have seen: there are no spaces in Japanese. The trick is simple: in Japanese text you have “significant” words (which are written with a mix of kanji and hiragana) and participles, which are always written in hiragana. This way, one can detect start of the words by changing of script. Let’s take the name of our game as an example: 恋する姉妹の六重奏