Omar









Level 0 Wonder Boy: The Dragon's Trap (working title) « on: October 13, 2015, 02:44:26 PM »



TheDragonsTrap.com











----















Character GIF















SCREENS



















ARTWORK











WHAT



In The Dragon’s Trap, the player, Hu-Man, gets cursed by a dragon and transformed into a fire breathing lizard character. The game follow our character freely wandering in monster land, looking for treasures and dragons. With each dragon defeated the character gets transformed into a new animal. Each of them comes with different abilities allowing the player to explore more of the world. Lizard-Man breath fire, Mouse-Man walk on walks, Piranha-Man can swim, Lion-Man can break certain blocks, Hawk-Man can fly, etc.



The game combine elements of exploration, adventure and action similar to the old Zelda games. Think swords, arrows, boomerangs. Think shield that plays a *tching* sound and pushes you five pixels behind when you receive an arrow. Think undocumented secret rooms. The game is “open-worldey” and put an emphasis on open exploration. It plays as a side-scrolling game. The music will be beautiful.



WAIT…. I KNOW THIS GAME, WHAT?



Ok ok. Some have heard of this game. It was created by a Japanese company called Westone and released in 1989 on the Sega Master System.



What are we doing then? This is a great game and we love it. While Nintendo have iterated (and milked) Zelda and Mario series to the max, some of the old high-quality Sega licenses completely went off the radar. We would like to make an gorgeous modern version of it. Hand drawn, hand animated, 60 fps, full hd, recording instruments and improving the game however we can. The original is quite “raw” and hard to gets your hands on as new player, we want to address that. It also involve securing licensing from old Japanese companies. Hey, no sweat.



Here’s the catch: the way I would like to do it. I want this remake to be very accurate in term of controls and gameplay mechanism. I don’t want to mess it up. I don’t want a pixel to be misplaced or physics or collision to be accidentally inaccurate. Think of Mario: it isn’t too hard to make a game that may look or feel like Mario from a distance, but it’s quite hard to make a game as polished, deep and with all the details of Super Mario World. And very few people are doing so! And The Dragon’s Trap has lots of subtleties in its use of equipments, physics, item drop, attack/defense/charm stats. So while I want to make changes to the game, I want to make them from a position or understanding exactly how it works and making informed changes. Rather than just trying to replicate it by guesswork and losing a lot in translation.

That process will also help me understand how some old games were designed. Plus, it's more fun and interesting the hard way.



HOW



So I am first going to reverse engineer the original assembled/compiled game code and its data, try to figure out how it works and how I can create an enhanced version of it. I will document some of this process, some of it has been done already because I spent quite some time tinkering with the game. Feel free to ask questions. Onward we will see how this can be turned into a real game and work with Ben to make something beautiful.



So probably blog will start off very technical and when the project takes off it will become very visual

I have been dabbling with this on my spare time for a while now, so trying to catch up with some of what I’ve been doing. We have progressed with Ben and already at the state where we are discussing with a small french company interested in retro games who would help us making this game a reality.



WHO



I'm Omar. I'll be doing the reverse-engineering, hacking, programming. Past games I've worked on include Soul Bubbles (NDS), PixelJunk Shooter series (Q-Games), Tearaway, Dreams (Media Molecule). Recently working on a free software library called ImGui to help game developers create tools. Been following TigSource as a reader and now finally I am free to post about a project while it is happening. Yeah! Web:



Ben is amazing artist and animator. He'll be doing the art and.. animation. See his stuff at

This is the main/placeholder post, we will add art / shots here.Character GIFSCREENSARTWORKIn The Dragon’s Trap, the player, Hu-Man, gets cursed by a dragon and transformed into a fire breathing lizard character. The game follow our character freely wandering in monster land, looking for treasures and dragons. With each dragon defeated the character gets transformed into a new animal. Each of them comes with different abilities allowing the player to explore more of the world. Lizard-Man breath fire, Mouse-Man walk on walks, Piranha-Man can swim, Lion-Man can break certain blocks, Hawk-Man can fly, etc.The game combine elements of exploration, adventure and action similar to the old Zelda games. Think swords, arrows, boomerangs. Think shield that plays a *tching* sound and pushes you five pixels behind when you receive an arrow. Think undocumented secret rooms. The game is “open-worldey” and put an emphasis on open exploration. It plays as a side-scrolling game. The music will be beautiful.WAIT…. I KNOW THIS GAME, WHAT?Ok ok. Some have heard of this game. It was created by a Japanese company called Westone and released in 1989 on the Sega Master System.What are we doing then? This is a great game and we love it. While Nintendo have iterated (and milked) Zelda and Mario series to the max, some of the old high-quality Sega licenses completely went off the radar. We would like to make an gorgeous modern version of it. Hand drawn, hand animated, 60 fps, full hd, recording instruments and improving the game however we can. The original is quite “raw” and hard to gets your hands on as new player, we want to address that. It also involve securing licensing from old Japanese companies. Hey, no sweat.Here’s the catch: the way I would like to do it. I want this remake to be very accurate in term of controls and gameplay mechanism. I don’t want to mess it up. I don’t want a pixel to be misplaced or physics or collision to be accidentally inaccurate. Think of Mario: it isn’t too hard to make a game that may look or feel like Mario from a distance, but it’s quite hard to make a game as polished, deep and with all the details of Super Mario World. And very few people are doing so! And The Dragon’s Trap has lots of subtleties in its use of equipments, physics, item drop, attack/defense/charm stats. So while I want to make changes to the game, I want to make them from a position or understanding exactly how it works and making informed changes. Rather than just trying to replicate it by guesswork and losing a lot in translation.That process will also help me understand how some old games were designed. Plus, it's more fun and interesting the hard way.So I am first going to reverse engineer the original assembled/compiled game code and its data, try to figure out how it works and how I can create an enhanced version of it. I will document some of this process, some of it has been done already because I spent quite some time tinkering with the game. Feel free to ask questions. Onward we will see how this can be turned into a real game and work with Ben to make something beautiful.So probably blog will start off very technical and when the project takes off it will become very visualI have been dabbling with this on my spare time for a while now, so trying to catch up with some of what I’ve been doing. We have progressed with Ben and already at the state where we are discussing with a small french company interested in retro games who would help us making this game a reality.I'm Omar. I'll be doing the reverse-engineering, hacking, programming. Past games I've worked on include Soul Bubbles (NDS), PixelJunk Shooter series (Q-Games), Tearaway, Dreams (Media Molecule). Recently working on a free software library called ImGui to help game developers create tools. Been following TigSource as a reader and now finally I am free to post about a project while it is happening. Yeah! Web: http://www.miracleworld.net , Twitter: @ocornut Ben is amazing artist and animator. He'll be doing the art and.. animation. See his stuff at http://www.benfiquet.com and twitter: @BenFiquet *EDIT* Game got released April 18th, 2017, for Switch, PS4, Xbox One. Working on PC port now.---- « Last Edit: April 22, 2017, 03:18:13 AM by Omar » Logged www.TheDragonsTrap.com/blog, www.miracleworld.net , twitter @ocornut

Omar









Level 0 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #1 on: October 13, 2015, 02:44:56 PM »

He is the shopkeeper, he also hands out passwords and knows everything.

We don’t have a name for him yet. Any suggestion?



The game has lots of arbitrary hidden secrets (think: invisible door in the middle of nowhere) that rely on player’s word of mouth, but the smoking pig could help here. My idea is instead of handing out passwords like in the original, we would make this guy hands out hints about where to go or secrets in the game. I can picture him talking in a riddle-y manner.



And this is the how the original 1989 game looked, for reference









The remake would need to look better and smoother! The game runs at 30 Hz and we’ll like to bump that up to 60. Ben made a concept artwork of a character from the game: the smoking pirate pig. You can see it above.He is the shopkeeper, he also hands out passwords and knows everything.We don’t have a name for him yet. Any suggestion?The game has lots of arbitrary hidden secrets (think: invisible door in the middle of nowhere) that rely on player’s word of mouth, but the smoking pig could help here. My idea is instead of handing out passwords like in the original, we would make this guy hands out hints about where to go or secrets in the game. I can picture him talking in a riddle-y manner.And this is the how the original 1989 game looked, for referenceThe remake would need to look better and smoother! The game runs at 30 Hz and we’ll like to bump that up to 60. Logged www.TheDragonsTrap.com/blog, www.miracleworld.net , twitter @ocornut

Omar









Level 0 Disassembly « Reply #2 on: October 13, 2015, 02:45:16 PM » Making sense of the mess



Extracting the game data



So where I am heading here with the reverse engineering is to start from a ROM dump. I’ve dumped the cartridge myself. Those who know me from some retro gaming forums may know I have an obsession for old Sega 8-bit games. That’s the sort of thing I do to my cartridges:





(picture added for the dramatic effect)



I have different setup for different use cases. Pictured above, there’s a prototype of R-Type, it comes on a bulky development board that the developers (Compile) used in 1987. I plugged the board into a homemade adapter. On the back of the console there’s an extension port that was meant to accessories but has never been used. We put a gender adapter here and then a cartridge modified to carry a simple software I wrote for the Master System, in Z80 assembly. The software run on the console, copies itself to RAM then execute from RAM. The software in RAM now reads the game cartridge data (here R-Type prototype, but same works with Wonder Boy III The Dragon’s Trap cartridge) and copies it back on the custom writable cartridge that’s plugged on the back. The writable cartridge is battery backed. I unplug everything and read can read the writable cartridge back on the PC using a custom made cart reader (I didn’t make the hardware myself, I would probably hurt myself with a soldering iron).



So I’ve extracted the data from the original game. It is your typical Master System cartridge. What’s we’ve got essentially is a 256 Kilobyte file. I’ll call it wb3.sms. You can find this file on the internet frankly, although for the sake of it I’m showing the full process here.



The file in an hex editor. A thousand pages of number. That’s our game!







Toward making this a little more readable..



Here are simplified specs of the Master System just to understand what we are talking about.



CPU: Z80 at 3.57 Mhz

RAM: 8 KB

VRAM: 16 KB

ROM: 256 KB (for this game)



You can find more detailed documentation





(A Zilog Z80 CPU)



The way is works is the cartridge contains both the code and the data interleaved. There’s no file-system as on a modern OS, but the game knows where its data are. There’s no easy way to tell which byte are data and which bytes are code. Being able to differentiate code from data would be useful though!



Code starts at offset 0000 which is the first byte of the ROM here. Using a Z80 disassembler or an emulator you can follow on this code. What a disassembler would typically do here: it will start from the first byte at 0000 and decode it into a readable instruction, then on to the next. Here’s a reference to the Z80 instruction set:



A disassembler knows about all the branching instructions and follows every possible branch to map all possible code paths it finds and decode them into a readable instruction format. The reason you have to do that and can’t just decode every byte of the ROM sequentially is that Z80 opcodes are of variable size. Some instructions are encoded in 1 byte, some in 2 bytes, some in 3 or 4 bytes. Based on the 1st byte you can tell if you need a second byte, etc. So you need to be at a correct position to decode the following bytes as code. If you start disassembling in the middle of a 3 bytes instruction, they will be interpreted as a completely different stream of instructions. On other architectures such as the ARM family of CPU widely used in mobile devices nowadays, all instructions are encoded in 4 bytes so this issue is made easier.



Here an example of disassembling a few instructions. Onward I will sometimes use the form FFh or $FF to denote an hexadecimal value because this is how they are typically expressed in old assemblers and disassemblers. Modern languages such as C or C++ would use 0xFF. Sometimes I will omit the prefix/suffix to make text more legible.



From ROM address 028Fh, the following 13 bytes values are DB BF CB 7F 28 FA 10 F8 3E FF D3 3F C9

This disassemble into:



Code: Address -- Bytes -------- Decoded instruction -- My comments

028F: DB BF ina (BFh) ; read from hardware input port BFh into A register

0291: CB 7F bit 7,a ; test if bit 7 of A register is set

0293: 28 FA jr z,-06h (028Fh) ; if bit of 0, jump back to 28F

The first instruction, ina (BFh) read from hardware input port BFh (which happen to be reading the status byte of the video chip, according to this documentation



Code: 0290: BF cp a ; compare A register with A register

0291: CB 7F bit 7,a ; test if bit 7 of A register is set

0293: 28 FA jr z,-06h (028Fh) ; if bit of 0, jump back to 28F

When taken in isolation, the second byte of what was our ina (BFh) instruction became a cp a instruction!

None are “incorrect”, but we can easy tell that the game is meant to execute from 028Fh and not from 0290h because:

There’s an jump back to 028Fh, so that’s pretty explicit!

Doing cp a followed by bit 7,a wouldn’t make much sense, since both are test instructions that affect the Flags register, and the result of cp a instead tested for.

In context it become usually easy to tell what is a correct / meaningful sequence of instruction, something which a disassembler may have a hard time to do in some situation.



Now let’s see that comes after… (don’t worry about understanding the details of this code)



Code: 029C: 21 AC 02 ld hl,02ACh ; set HL register (16 bits) to 02ACh

029F: 11 6C CF ld de,CF6Ch ; set DE register (16 bits) to CF6Ch

02A2: 06 14 ld b,14h ; set B register (8 bits) to 14h

02A4: 7E ld a,(hl) ; load from the address pointed to by HL into A

02A5: EF rst 28h ; call the function at 28H

02A6: 12 ld (de),a ; load A register into the address pointer to by DE

02A7: 23 inc hl ; increment HL

02A8: 13 inc de ; increment DE

02A9: 10 F9 djnz -07h (02A4h) ; decrement B, if B is not zero, jump back to 2A4H

02AB: C9 ret ; return;

02AC: 26 80 ld h,80h ; <-- from there the code doesn’t make much sense

02AE: A2 and d ; also, we are right after a “ret” instruction

02AF: 81 add c ; which is an unconditional “return from call” instruction

02B0: FF rst 38h ; so we can easily guess that those bytes are not code

02B1: 82 add d ; they are data!

02B2: FF rst 38h ;

Here this disassembler choose to continue disassembly after the RET instruction at 02ABh in spite of the instruction being an unconditional return. The code after RET that won’t be executed, unless some other parts of the program jumps directly into it. Now, by looking at the “instructions” coming afterward we can guess that they aren’t meant to be executed by the CPU, but are rather data bytes used for another purpose.



Now, most Z80 disassemblers have a problem. Some of the branch are dynamic, created by the game based on state variables manipulations, and the disassembler can’t really tell about them. A static branch would be “if this register is zero, then jump execution to this address”. here we know about the address in question. A dynamic branch would be “take this address and add 2 times to the content of register HL”, then jump to that address. Because the register content can be computed in any number of way and come from any source it’s hard for a disassembler to find and follow every path of code. The code that I’ve looked at for The Dragon’s Trap seemed to use lots of dynamic branches and jump tables and most disassemblers would miss a lot of the code.



So we have two solutions here: use an emulator and inspect the game as it is running, or find a better disassembler.



Using an emulator



I’ve made this Sega Master System emulator called MEKA a while ago. So i know how the hardware works which is saving me lots of time here. In fact the emulator was called MEKA in homage to this very game! (Meka being the name of the first dragon that curses you into lizard form). So my love for this game runs from long ago! The emulator is old clunky software but it provides a bunch of very useful debugging and hacking tools which are perfect to assist in reverse engineering this game.







Using the emulator it shows me a short disassembled view based on the current CPU instruction. I can step through the code and set breakpoints. It’s still extremely raw and abstract, there’s literally tens of thousands of instructions and it’s hard to make sense out of them out right now. However we will still use the emulator and its tools extensively and we’ll see how we can make this mess clearer later.



Finding a better disassembler



So I mentioned dynamic branching that most Z80 disassemblers can’t follow. There are a lot of disassembler available (a basic one is fairly easy to write) and it became clear that if I gave them the compiled code for The Dragon’s Trap they output a poor result (not separating code/data, or missing out on code).



As it turns out, someone going by the nickname of Calindro has been working on a project called Emulicious (



When I first tried Emulicious it didn’t really work out perfectly and missed a lot of the code, but going back and forth with Calindro he implemented a few fixes and improvements to his disassembler and it largely improved the situation. Even though, it doesn’t manage to find and isolate all bits of code vs data automatically. But through a configuration file it can be provided a few code “entry points” that we’ve discovered by studying the partial disassembly and the behavior in emulator. So e.g. I tell the disassembler that I know for a fact there is code at address 696Ah and it will disassemble from this location in addition to the other locations it found itself. Using this data Emulicious can provide me with a fairly accurate disassembly that separates code from data.



Here’s the same chunk of code output by Emulicious.



Code: -: ; <-- this is a local anonymous label

in a, ($BF)

bit 7, a

jr z, - ; ..so when you read the jump execution you know where it is going

djnz -

+++++:

ld a, $FF

out ($3F), a

ret



_LABEL_029C_: ; <-- it is a named label because it has detected that

ld hl, _DATA_2AC_ ; other code is “calling” into that address

ld de, _RAM_CF6C_ ; <-- every location in the RAM range also gets a label

ld b, $14

-:

ld a, (hl)

rst $28 ; _LABEL_0028_

ld (de), a

inc hl

inc de

djnz -

ret



; Data from 2AC to 2BF (20 bytes) ; <-- It has figured out that those bytes are data

_DATA_2AC_: ; because no code is ever executed here

.db $26 $80 $A2 $81 $FF $82 $FF $83 $FF $84 $FF $85 $FB $86 $00 $88

.db $00 $89 $00 $8A

Note that the syntax is a little different. Here our “ina (BFh)” read as “in a, ($BF)”. They are totally the same instructions but different assemblers/disassembler uses slightly different syntax.



It is much better! The reason the disassembler is turning addresses (such as 029C or CF6C) into a named label (such as _LABEL_029C_ or _RAM_CF6C_) is to give the user a chance to rename them easily in a text editor. In fact, a lot of the reverse engineering work will consists in giving legible names to all those obscure labels to lift some of all that fog. Once we learn that RAM address CF41h contains the amount of boomerang carried by the players, we can rename the _RAM_CF41_ label into something like NumberOfBoomerang all across the generated disassembly. Then all reference to this variable will make the code more readable.



We can even use a compatible assembler to turn back the disassembly listing into code. If we haven’t made modification to any of the instructions, the output will be identical to what we started with. The label themselves are just references for the programmers and the assembler to convert into addresses, but their name aren’t part of the compiled code. If the disassembler has managed to turn all addresses and references into named labels (which it seems to have done successfully), then we can also move or add instructions into the disassembly and rebuild the game with modification. The chances are high that will still run without bugs. If the game code does some funky address computations expecting known/hardcoded addresses that the disassembler couldn’t catch, moving a single byte of the code will likely break the game until we manage to patch the offending code.



Attached is the disassembly output by Emulicious. It is 28000 lines of instructions + short data blocks for the main file + remaining 181 KB of unknown binary data left untouched. We are going to study those instructions and the data, and figure out if we can turn it into a modern game!



(180 KB)



Let me know if there’s something I can get into details, if it’s unclear, or just insane!

So where I am heading here with the reverse engineering is to start from a ROM dump. I’ve dumped the cartridge myself. Those who know me from some retro gaming forums may know I have an obsession for old Sega 8-bit games. That’s the sort of thing I do to my cartridges:(picture added for the dramatic effect)I have different setup for different use cases. Pictured above, there’s a prototype of R-Type, it comes on a bulky development board that the developers (Compile) used in 1987. I plugged the board into a homemade adapter. On the back of the console there’s an extension port that was meant to accessories but has never been used. We put a gender adapter here and then a cartridge modified to carry a simple software I wrote for the Master System, in Z80 assembly. The software run on the console, copies itself to RAM then execute from RAM. The software in RAM now reads the game cartridge data (here R-Type prototype, but same works with Wonder Boy III The Dragon’s Trap cartridge) and copies it back on the custom writable cartridge that’s plugged on the back. The writable cartridge is battery backed. I unplug everything and read can read the writable cartridge back on the PC using a custom made cart reader (I didn’t make the hardware myself, I would probably hurt myself with a soldering iron).So I’ve extracted the data from the original game. It is your typical Master System cartridge. What’s we’ve got essentially is a 256 Kilobyte file. I’ll call it wb3.sms. You can find this file on the internet frankly, although for the sake of it I’m showing the full process here.The file in an hex editor. A thousand pages of number. That’s our game!Here are simplified specs of the Master System just to understand what we are talking about.CPU: Z80 at 3.57 MhzRAM: 8 KBVRAM: 16 KBROM: 256 KB (for this game)You can find more detailed documentation here but you don’t really need to for the purpose of following this blog. Unless you are curious or at ease with the hardware info. The Zilog Z80 was a popular CPU first made in 1976 but used through the eighties in variety of game and computer systems: many arcade games, Amstrad CPC, Game Boy, Master System etc.(A Zilog Z80 CPU)The way is works is the cartridge contains both the code and the data interleaved. There’s no file-system as on a modern OS, but the game knows where its data are. There’s no easy way to tell which byte are data and which bytes are code. Being able to differentiate code from data would be useful though!Code starts at offset 0000 which is the first byte of the ROM here. Using a Z80 disassembler or an emulator you can follow on this code. What a disassembler would typically do here: it will start from the first byte at 0000 and decode it into a readable instruction, then on to the next. Here’s a reference to the Z80 instruction set: http://clrhome.org/table A disassembler knows about all the branching instructions and follows every possible branch to map all possible code paths it finds and decode them into a readable instruction format. The reason you have to do that and can’t just decode every byte of the ROM sequentially is that Z80 opcodes are of variable size. Some instructions are encoded in 1 byte, some in 2 bytes, some in 3 or 4 bytes. Based on the 1st byte you can tell if you need a second byte, etc. So you need to be at a correct position to decode the following bytes as code. If you start disassembling in the middle of a 3 bytes instruction, they will be interpreted as a completely different stream of instructions. On other architectures such as the ARM family of CPU widely used in mobile devices nowadays, all instructions are encoded in 4 bytes so this issue is made easier.Here an example of disassembling a few instructions. Onward I will sometimes use the form FFh or $FF to denote an hexadecimal value because this is how they are typically expressed in old assemblers and disassemblers. Modern languages such as C or C++ would use 0xFF. Sometimes I will omit the prefix/suffix to make text more legible.From ROM address 028Fh, the following 13 bytes values are DB BF CB 7F 28 FA 10 F8 3E FF D3 3F C9This disassemble into:The first instruction, ina (BFh) read from hardware input port BFh (which happen to be reading the status byte of the video chip, according to this documentation http://www.smspower.org/uploads/Development/smstech-20021112.txt ). ina (BFh) is encoded in 2 bytes as DB BF. If instead we started decoding start from the following byte 0x290 (0x28F+1 = 0x290) we would get:When taken in isolation, the second byte of what was our ina (BFh) instruction became a cp a instruction!None are “incorrect”, but we can easy tell that the game is meant to execute from 028Fh and not from 0290h because:There’s an jump back to 028Fh, so that’s pretty explicit!Doing cp a followed by bit 7,a wouldn’t make much sense, since both are test instructions that affect the Flags register, and the result of cp a instead tested for.In context it become usually easy to tell what is a correct / meaningful sequence of instruction, something which a disassembler may have a hard time to do in some situation.Now let’s see that comes after… (don’t worry about understanding the details of this code)Here this disassembler choose to continue disassembly after the RET instruction at 02ABh in spite of the instruction being an unconditional return. The code after RET that won’t be executed, unless some other parts of the program jumps directly into it. Now, by looking at the “instructions” coming afterward we can guess that they aren’t meant to be executed by the CPU, but are rather data bytes used for another purpose.Now, most Z80 disassemblers have a problem. Some of the branch are dynamic, created by the game based on state variables manipulations, and the disassembler can’t really tell about them. A static branch would be “if this register is zero, then jump execution to this address”. here we know about the address in question. A dynamic branch would be “take this address and add 2 times to the content of register HL”, then jump to that address. Because the register content can be computed in any number of way and come from any source it’s hard for a disassembler to find and follow every path of code. The code that I’ve looked at for The Dragon’s Trap seemed to use lots of dynamic branches and jump tables and most disassemblers would miss a lot of the code.So we have two solutions here: use an emulator and inspect the game as it is running, or find a better disassembler.I’ve made this Sega Master System emulator called MEKA a while ago. So i know how the hardware works which is saving me lots of time here. In fact the emulator was called MEKA in homage to this very game! (Meka being the name of the first dragon that curses you into lizard form). So my love for this game runs from long ago! The emulator is old clunky software but it provides a bunch of very useful debugging and hacking tools which are perfect to assist in reverse engineering this game.Using the emulator it shows me a short disassembled view based on the current CPU instruction. I can step through the code and set breakpoints. It’s still extremely raw and abstract, there’s literally tens of thousands of instructions and it’s hard to make sense out of them out right now. However we will still use the emulator and its tools extensively and we’ll see how we can make this mess clearer later.So I mentioned dynamic branching that most Z80 disassemblers can’t follow. There are a lot of disassembler available (a basic one is fairly easy to write) and it became clear that if I gave them the compiled code for The Dragon’s Trap they output a poor result (not separating code/data, or missing out on code).As it turns out, someone going by the nickname of Calindro has been working on a project called Emulicious ( http://emulicious.net ). It is an emulator but it also aims to provide great debugging/hacking features and include a powerful disassembler. The idea is that his disassembler is leveraging emulation techniques to provide a more accurate disassembly of the game. It also knows about the machine (its memory map, its I/O port map) and uses lots of trick to be able to map where all the code is located.When I first tried Emulicious it didn’t really work out perfectly and missed a lot of the code, but going back and forth with Calindro he implemented a few fixes and improvements to his disassembler and it largely improved the situation. Even though, it doesn’t manage to find and isolate all bits of code vs data automatically. But through a configuration file it can be provided a few code “entry points” that we’ve discovered by studying the partial disassembly and the behavior in emulator. So e.g. I tell the disassembler that I know for a fact there is code at address 696Ah and it will disassemble from this location in addition to the other locations it found itself. Using this data Emulicious can provide me with a fairly accurate disassembly that separates code from data.Here’s the same chunk of code output by Emulicious.Note that the syntax is a little different. Here our “ina (BFh)” read as “in a, ($BF)”. They are totally the same instructions but different assemblers/disassembler uses slightly different syntax.It is much better! The reason the disassembler is turning addresses (such as 029C or CF6C) into a named label (such as _LABEL_029C_ or _RAM_CF6C_) is to give the user a chance to rename them easily in a text editor. In fact, a lot of the reverse engineering work will consists in giving legible names to all those obscure labels to lift some of all that fog. Once we learn that RAM address CF41h contains the amount of boomerang carried by the players, we can rename the _RAM_CF41_ label into something like NumberOfBoomerang all across the generated disassembly. Then all reference to this variable will make the code more readable.We can even use a compatible assembler to turn back the disassembly listing into code. If we haven’t made modification to any of the instructions, the output will be identical to what we started with. The label themselves are just references for the programmers and the assembler to convert into addresses, but their name aren’t part of the compiled code. If the disassembler has managed to turn all addresses and references into named labels (which it seems to have done successfully), then we can also move or add instructions into the disassembly and rebuild the game with modification. The chances are high that will still run without bugs. If the game code does some funky address computations expecting known/hardcoded addresses that the disassembler couldn’t catch, moving a single byte of the code will likely break the game until we manage to patch the offending code.Attached is the disassembly output by Emulicious. It is 28000 lines of instructions + short data blocks for the main file + remaining 181 KB of unknown binary data left untouched. We are going to study those instructions and the data, and figure out if we can turn it into a modern game! disassembly.zip (180 KB)Let me know if there’s something I can get into details, if it’s unclear, or just insane! Logged www.TheDragonsTrap.com/blog, www.miracleworld.net , twitter @ocornut

flipswitchx









Level 3 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #3 on: October 13, 2015, 06:19:07 PM »



Your quest is noble, goodluck and godspeed sir





Oh right I almost forgot. High above the mucky muck, castle made of clouds... I've never heard of this until now, but now that I have, I must say it looks rad, and that you sir, are a wizard.Your quest is noble, goodluck and godspeed sirOh right I almost forgot. « Last Edit: October 13, 2015, 06:27:11 PM by flipswitchx » Logged

breakin







TIGBaby Re: Wonder Boy: The Dragon's Trap (working title) « Reply #5 on: October 14, 2015, 01:50:31 AM »



I'd like to share some experiences from my attempts at reverse engineering Zelda for the SNES. I had nailed a wonderful process and only stopped when I found the full annotated source from someone who had started X years ahead of me Anyway.



I added the ability to an emulator (snes9x in my case) to save a binary file where it encoded program counter as well as some flags needed to detect where all opcodes started and how they were to be decoded. This was partially needed because the SNES-CPU has a legacy NES-compatible mode that it can switch to at any time. This changed the length of opcodes so getting the mode correct was very important. I also recorded all non-deterministic jumps (data-dependent jumps).



I then had a C++-program that I wrote myself that took the ROM-file, the binary dump given from emulation and a simple file where I named labels and generated full disassembly. The label file looked like this:



Some benefits:

* Once I named a label, that name would start appearing in code that I hadn't looked at yet. Once some key functions were recognized and named other code started making sense.

* I added comments to all jumps/branches about where they could go (in the cases where I couldn't simply insert a labelname into the source code). This made me see code flow really well, especially when jump-tables were used. Zelda executes a state machine for each frame and this was very easy to detect.

* I could play a small portion of the game (say boot it up and then save a binary dump). Then I could generate disassembly from that run alone. This gave me much less code that I could sift through.

* If I started the game I could play around for a while and save a binary snapshot. I could then do something unique that I hadn't done before. The generator could diff those two snap shots to only generate dissassembly for that new thing.

* There were a few common jump-table formats that I recognized and added the ability for the dissassembly to generate code for. That way they would end up nicely formatted in the source code as a table. Adding game-specific hooks like this was really nice.



I also added some prediction where the dissassembly could fill in code that was never visited during my playthrough session. It was kinda good but sometimes it discovered data instead of code. A dream I had was to detect common "macros" (patterns) that the game maker (probably) used and extract those into actual macros.



In all it was a very nice workflow to just sit with the generated assembly and the name label-file and read... think... name some labels.. regenerate...



My code can be found



I hope your project goes well and that you have a lot of fun! Wonderful project! I love reverse engineering of old games.I'd like to share some experiences from my attempts at reverse engineering Zelda for the SNES. I had nailed a wonderful process and only stopped when I found the full annotated source from someone who had started X years ahead of meAnyway.I added the ability to an emulator (snes9x in my case) to save a binary file where it encoded program counter as well as some flags needed to detect where all opcodes started and how they were to be decoded. This was partially needed because the SNES-CPU has a legacy NES-compatible mode that it can switch to at any time. This changed the length of opcodes so getting the mode correct was very important. I also recorded all non-deterministic jumps (data-dependent jumps).I then had a C++-program that I wrote myself that took the ROM-file, the binary dump given from emulation and a simple file where I named labels and generated full disassembly. The label file looked like this: https://github.com/breakin/snestistics/blob/master/examples/zelda3-ntsc/labels.def Some benefits:* Once I named a label, that name would start appearing in code that I hadn't looked at yet. Once some key functions were recognized and named other code started making sense.* I added comments to all jumps/branches about where they could go (in the cases where I couldn't simply insert a labelname into the source code). This made me see code flow really well, especially when jump-tables were used. Zelda executes a state machine for each frame and this was very easy to detect.* I could play a small portion of the game (say boot it up and then save a binary dump). Then I could generate disassembly from that run alone. This gave me much less code that I could sift through.* If I started the game I could play around for a while and save a binary snapshot. I could then do something unique that I hadn't done before. The generator could diff those two snap shots to only generate dissassembly for that new thing.* There were a few common jump-table formats that I recognized and added the ability for the dissassembly to generate code for. That way they would end up nicely formatted in the source code as a table. Adding game-specific hooks like this was really nice.I also added some prediction where the dissassembly could fill in code that was never visited during my playthrough session. It was kinda good but sometimes it discovered data instead of code. A dream I had was to detect common "macros" (patterns) that the game maker (probably) used and extract those into actual macros.In all it was a very nice workflow to just sit with the generated assembly and the name label-file and read... think... name some labels.. regenerate...My code can be found https://github.com/breakin/snestistics but I don't recommend using it. The true power came from writing exactly what I wanted for my game exactly when I needed it. But I want to inspire with the ideas I had. One downside is that it does not work if code is executed out of memory. Some platforms use SMC extensively for performance reasons but most nintendo bases SNES game does not use this feature.I hope your project goes well and that you have a lot of fun! Logged

Jasmine





Boop





Level 5Boop Re: Wonder Boy: The Dragon's Trap (working title) « Reply #6 on: October 14, 2015, 05:23:28 AM » Wow. You are really going through a lot of hurdles to revive this game. It looks like I should play the original! I have a question --- in working to reverse engineer this title, do you plan to simply gain the code and release the game for free? This isn't a endeavor where profit is an end goal, I assume.



Also, the passion you and your friend are putting into this is astounding! I can't wait to see more! *subbed* Logged Portfolio ~ Soundcloud

Arne

Level 6









The PantymasterLevel 6 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #7 on: October 14, 2015, 05:39:07 AM »



There's some tiny bit of info about the pig in the



It's good to go straight to the ROM files for data. There's no knowing what internet-people have done to sprite sheets and such (they are often badly ripped from screenshots, hand cleaned silhouettes, sneaky edits.)



MEKA is nice. I used it to probe Phantasy Star for a romhack project. It seems the SMS likes to stream in tiles to animate them. However, it somehow packs the static tiles (though not the streamable ones because of convenience)?



WB3 has some pretty good graphics. I like the simple light-shade thing it has going on. I did some Wonder Boy pixel stuff for my

Interesting!There's some tiny bit of info about the pig in the HCG101 article . You might have read it already though.It's good to go straight to the ROM files for data. There's no knowing what internet-people have done to sprite sheets and such (they are often badly ripped from screenshots, hand cleaned silhouettes, sneaky edits.)MEKA is nice. I used it to probe Phantasy Star for a romhack project. It seems the SMS likes to stream in tiles to animate them. However, it somehow packs the static tiles (though not the streamable ones because of convenience)?WB3 has some pretty good graphics. I like the simple light-shade thing it has going on. I did some Wonder Boy pixel stuff for my Famicube project. Never actually played WB3 though, only the first one, on an arcade machine sometime in the 80's (perhaps a bootleg). I was very impressed by it... the skateboarding and amazing graphics. It seemed truly adventurous to my imagination! « Last Edit: October 14, 2015, 05:48:27 AM by Arne » Logged

Omar









Level 0 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #8 on: October 14, 2015, 08:55:32 AM »



Quote from: breakin on October 14, 2015, 01:50:31 AM Wonderful project! I love reverse engineering of old games.

Some interesting ideas here. I don't know the SNES but I imagine the problem might be more complex there than for Master System+Z80. The Z80 has quite a limited instruction set, so e.g. there's only so many ways you can achieve a data driven jump table with that CPU.



My approach even before I got a correct disassembly what to create a list of symbol and start adding label to ROM and RAM locations I could identify. Disassemblers can use a symbol file as input so that gets written back into the disassembly. Rince and repeat. Essentially I don't edit the disassembly directly but only the symbol file.



Emulicious does some mumbo-jumbo to detect jump tables, in particular to detect their length which is never explicit. Maybe using a mixture of analysing prior opcodes (which register is used, how are they masked?) maybe infer it from locations already marked as code or data. I had to specify a few things manually but otherwise it looks like it's done a good job at disassembling now.

Thanks everyone!Some interesting ideas here. I don't know the SNES but I imagine the problem might be more complex there than for Master System+Z80. The Z80 has quite a limited instruction set, so e.g. there's only so many ways you can achieve a data driven jump table with that CPU.My approach even before I got a correct disassembly what to create a list of symbol and start adding label to ROM and RAM locations I could identify. Disassemblers can use a symbol file as input so that gets written back into the disassembly. Rince and repeat. Essentially I don't edit the disassembly directly but only the symbol file.Emulicious does some mumbo-jumbo to detect jump tables, in particular to detect their length which is never explicit. Maybe using a mixture of analysing prior opcodes (which register is used, how are they masked?) maybe infer it from locations already marked as code or data. I had to specify a few things manually but otherwise it looks like it's done a good job at disassembling now. Logged www.TheDragonsTrap.com/blog, www.miracleworld.net , twitter @ocornut

Omar









Level 0 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #9 on: October 14, 2015, 09:05:42 AM » Quote from: M4uesviecr on October 14, 2015, 05:23:28 AM Wow. You are really going through a lot of hurdles to revive this game. It looks like I should play the original! I have a question --- in working to reverse engineer this title, do you plan to simply gain the code and release the game for free? This isn't a endeavor where profit is an end goal, I assume.



Also, the passion you and your friend are putting into this is astounding! I can't wait to see more! *subbed*

This project has been in my mind for a decade so yeah it is quite a passion project (perhaps obsession is a more fitting word ).



Profit isn't an end goal, however with the quality we would like to aim for, we will probably need Ben to sit down for a year only to make the graphics and animation content. Not mentioning the tech that I haven't fully scoped yet. That probably means the game if it becomes a real game won't be a free release otherwise we can't afford to make it. So we still need to figure out how we can make this development happens, living in a cardboard box or getting funding or doing paid work on the side to pay the bills, etc. And it needs to be legal, because obviously we're working from Westone's work here, so if it becomes a real thing it'll need to be licensed. Another possibility is that it ends being curiosity for-fun research project and doesn't become a full fledged game, in which case it's more likely we can just release everything as-in for the community. I will discuss some of those topics in the following posts, we haven't figured out all that. This project has been in my mind for a decade so yeah it is quite a passion project (perhaps obsession is a more fitting word).Profit isn't an end goal, however with the quality we would like to aim for, we will probably need Ben to sit down for a year only to make the graphics and animation content. Not mentioning the tech that I haven't fully scoped yet. That probably means the game if it becomes a real game won't be a free release otherwise we can't afford to make it. So we still need to figure out how we can make this development happens, living in a cardboard box or getting funding or doing paid work on the side to pay the bills, etc. And it needs to be legal, because obviously we're working from Westone's work here, so if it becomes a real thing it'll need to be licensed. Another possibility is that it ends being curiosity for-fun research project and doesn't become a full fledged game, in which case it's more likely we can just release everything as-in for the community. I will discuss some of those topics in the following posts, we haven't figured out all that. Logged www.TheDragonsTrap.com/blog, www.miracleworld.net , twitter @ocornut

Logbomb









Level 0 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #11 on: October 14, 2015, 10:57:00 AM »



For those who are considering playing the original game, I would highly recommend it, as it truly is one of the best games of the 8-bit era.



The smoking pirate pig looks awesome, looking forward to seeing more of Ben's work This is a very exciting project indeedFor those who are considering playing the original game, I would highly recommend it, as it truly is one of the best games of the 8-bit era.The smoking pirate pig looks awesome, looking forward to seeing more of Ben's work Logged

mtarini







Level 1 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #12 on: October 14, 2015, 11:16:07 AM » On a more technical note, I'm a bit surprised that your plan of attack includes the work-intensive (although fun) task of reverse engineering the original code or assets.



Assets: but, you are doing your own, right?

What will be the use of, say, ripped sprites.



Code: frankly, I cannot imagine how this will help. Is there anything mysterious in the original program, some magic formula of sort, than only the naked code can reveal? This may sometimes (rarely) be the case, for, say, obscure damage computation, or I dunno, AI behavior, the way certain probabilities are computed ... But I can't see any of that applying here. I expect that in the code you will find lots of technical, outdated, awkward-to-be-traced solutions, but nothing hi-level that you couldn't get by observing the game itself.



A more natural road would seem to me to just capture the spirit of the game and imitate it in your own game.

In other words, to start by cloning it, rather than emulating it.



Emulation (which I also am a big fun of, BTW) is for when you want to recreate the exact same experience, pixel by pixel, quirk by quirk, sound effect by sound effect, 100% faithfully (bugs and frame-rate included).





I don't mean this as a criticism! I just don't think I understand. Logged Warballs! · spherical fierceness · 1P · free · arena fighter · challenging

Omar









Level 0 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #13 on: October 14, 2015, 11:57:04 AM » Quote from: mtarini on October 14, 2015, 11:16:07 AM On a more technical note, I'm a bit surprised that your plan of attack includes the work-intensive (although fun) task of reverse engineering the original code or assets.

Assets include the level layouts, the timing of movements. Yes we won't rip or use the original sprites (that'd be easy anyway). I actually got my hands into this project originally because I was curious to find if the game had any unknown secret locations. So the first thing I looked was looking for the door mechanism/code and trying to locate the corresponding data. This led into something bigger!



The code HAS lots of awkward and outdated things, but this is what define the game. I think people underestimate the values of all the weird details adding up. (



Also I don't think extracting the level layout data from the ROM is crazy thing to do. It may permit the creation of other game hacks, level editors. So it would have quite a few nice uses aside from making the remake start from the solid base. This blog is a bit behind with my current work so I have to catch up but I've sort of changed strategy mid-way and I'm going to get into that when I have time to write the next posts. Assets include the level layouts, the timing of movements. Yes we won't rip or use the original sprites (that'd be easy anyway). I actually got my hands into this project originally because I was curious to find if the game had any unknown secret locations. So the first thing I looked was looking for the door mechanism/code and trying to locate the corresponding data. This led into something bigger!The code HAS lots of awkward and outdated things, but this is what define the game. I think people underestimate the values of all the weird details adding up. ( The devil is in the detail ). Because it's just a pile of weird details I don't think you can recreate it by feel. To answer your question there are lot of subtle mystery about this game, how certain monsters in certain spot tends to drop certain items. How wearing a certain sword affect item drops, etc. As stated above we don't want to recreate the exact same game but I want differences to be intentional rather than accidental. For examples, the original game is running at 30 FPS and I want the new one to be 60 FPS, that's one difference that I want to make happen and there will be hundred of them.Also I don't think extracting the level layout data from the ROM is crazy thing to do. It may permit the creation of other game hacks, level editors. So it would have quite a few nice uses aside from making the remake start from the solid base. This blog is a bit behind with my current work so I have to catch up but I've sort of changed strategy mid-way and I'm going to get into that when I have time to write the next posts. Logged www.TheDragonsTrap.com/blog, www.miracleworld.net , twitter @ocornut

Omar









Level 0 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #14 on: October 14, 2015, 12:12:01 PM » Quote from: Arne on October 14, 2015, 05:39:07 AM MEKA is nice. I used it to probe Phantasy Star for a romhack project. It seems the SMS likes to stream in tiles to animate them. However, it somehow packs the static tiles (though not the streamable ones because of convenience)? All rendered tiles on the SMS have to be uploaded to VRAM so the CPU is free to manipulate and encode the tile data however they want. There's no Character ROM like the NES can use. Most games employ some sort or multiple compression scheme depending on how much time they have to upload the tiles to VRAM. Rarely changing pictures can use more intricate compression, while tiles meant to be reuploaded every frame are optimised for fast upload. Typically game tend to stream tile for characters that have few instances and many animations (e.g. player character) and keep all tiles in VRAM for characters that have many instances and few animations (enemies).



Quote WB3 has some pretty good graphics. I like the simple light-shade thing it has going on. I did some Wonder Boy pixel stuff for my Famicube project. Never actually played WB3 though, only the first one, on an arcade machine sometime in the 80's (perhaps a bootleg). I was very impressed by it... the skateboarding and amazing graphics. It seemed truly adventurous to my imagination!

Oh your Famicube page is amazing. Only skimmed through the text yet (will read), but the mockup artwork are amazing. Games like Psycho Fox are under-represented. Is that a Meka Octopus in your Wonder Boy III sprite sheet? The gameplay became quite different after the first. Wonder Boy in Monster Land evolved into an action-adventure game with a fast arcade pace, and Wonder Boy III followed on a formula but made it a connected open-world with a sense of exploration, and it isn't an always-running game with the second one. We should make a game together! All rendered tiles on the SMS have to be uploaded to VRAM so the CPU is free to manipulate and encode the tile data however they want. There's no Character ROM like the NES can use. Most games employ some sort or multiple compression scheme depending on how much time they have to upload the tiles to VRAM. Rarely changing pictures can use more intricate compression, while tiles meant to be reuploaded every frame are optimised for fast upload. Typically game tend to stream tile for characters that have few instances and many animations (e.g. player character) and keep all tiles in VRAM for characters that have many instances and few animations (enemies).Oh your Famicube page is amazing. Only skimmed through the text yet (will read), but the mockup artwork are amazing. Games like Psycho Fox are under-represented. Is that a Meka Octopus in your Wonder Boy III sprite sheet? The gameplay became quite different after the first. Wonder Boy in Monster Land evolved into an action-adventure game with a fast arcade pace, and Wonder Boy III followed on a formula but made it a connected open-world with a sense of exploration, and it isn't an always-running game with the second one. We should make a game together! Logged www.TheDragonsTrap.com/blog, www.miracleworld.net , twitter @ocornut

Logbomb









Level 0 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #16 on: October 15, 2015, 05:34:39 AM » One of the challenges of bringing this game to a modern audience, is the length of the original game. By modern standards it is quite short, especially for an action adventure game. I certainly like the idea of keeping the original level design, so an idea for extending the length of the game would be to extend the game past its traditional ending.



The idea would be that once you have regained your Hu-Man form, the game continues with new areas that you play as the Hu-Man, and ultimately there is a final boss fight that ties the game together, possibly with a new incarnation of the Meka Dragon, or even a prequel version of Bio Meka from Monster World III and IV (or even both).



I think this would really work in terms of the story of the game, as the existing final boss doesn't seem to tie back into the main story of the Meka invasion, from the other monster world titles. It would also give fans of the original game something exciting and new, while still maintaining the structure of the original game. Of course it would also make the game longer, which is something that is needed for a modern action adventure game.



Just some thoughts, I know it is early days yet, but I think this is worth some serious consideration when the time comes. Logged

Virtu









Level 0 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #17 on: October 15, 2015, 08:53:46 PM »





It was sold just as any other game in stores back then, it was just the original Wonder Boy III with a few sprite swaps and translated text, as an attempt to cater more to the brazilian audience, since this "Monica's Gang" is a popular franchise around here. I guess it kinda worked, otherwise I don't think my parents would've gotten me this game!



Where am I going with this, anyways? Well, I have very little experience in ROM disassembling and the likes, but I'm guessing this hack should be rather similar to the actual thing, internally (I might be wrong though!). Off the top of my head, the only features that differ from the original game (from the player's perspective) are the title screen, the main hero sprites, the shopkeeper sprites, the sprites for the final boss and the text. Everything else was kept unchanged. I don't know if comparing the two ROMs in some way would be of any use, but you should be able to find this oddity fairly easily on the internets, even if just for kicks.



Best of luck to you two on the project! Wow, full support to you guys! This game was a huge part of my childhood.. well, not this exact version of the game, per se. Being a brazilian guy I had access to the odd hack that was "Turma da Mônica em O Resgate" instead. (roughly translates to "Monica's Gang: The Rescue")It was sold just as any other game in stores back then, it was just the original Wonder Boy III with a few sprite swaps and translated text, as an attempt to cater more to the brazilian audience, since this "Monica's Gang" is a popular franchise around here. I guess it kinda worked, otherwise I don't think my parents would've gotten me this game!Where am I going with this, anyways? Well, I have very little experience in ROM disassembling and the likes, but I'm guessing this hack should be rather similar to the actual thing, internally (I might be wrong though!). Off the top of my head, the only features that differ from the original game (from the player's perspective) are the title screen, the main hero sprites, the shopkeeper sprites, the sprites for the final boss and the text. Everything else was kept unchanged. I don't know if comparing the two ROMs in some way would be of any use, but you should be able to find this oddity fairly easily on the internets, even if just for kicks.Best of luck to you two on the project! Logged games and stuff:

THE DAY OF REASSEMBLY (2014) puzzle platformer | SUBVERTIGO (2015) topdown shooter

EvilDingo







Level 1 Re: Wonder Boy: The Dragon's Trap (working title) « Reply #18 on: October 16, 2015, 02:27:00 PM »



id software once re-made the first level of Super Mario Bros and sent it to Nintendo asking to license their game on the PC. Nintendo declined stating they're not in the business of making PC games. I think id made Commander Keen from that engine.



The development world has changed quite a bit since then and you might get a favorable response if you make an amazing recreation of Wonder Boy (but probably not.)



Who knows? It's probably impossible for independent developers to do this (I haven't been successful) but why not try licensing Wonder Boy and doing it properly? Sega (or whoever owns the IP now) probably won't license it, but how great would it be to love a game this much and get approval to actually make a game in this universe?id software once re-made the first level of Super Mario Bros and sent it to Nintendo asking to license their game on the PC. Nintendo declined stating they're not in the business of making PC games. I think id made Commander Keen from that engine.The development world has changed quite a bit since then and you might get a favorable response if you make an amazing recreation of Wonder Boy (but probably not.)Who knows? Logged Cliff Diving | 3D Bowling