Hi, it’s pent0 (I changed my nickname, yey :D).

I went hiatus working on EKA2L1 for 4 months with only sharing the progress on Discord. The last progress report was on my website, which I don’t think is convenient for reading, so I switched to WordPress here.

Sadly, not much progress was made on graphical display in the last 4 months except for what you saw in April. I’m terribly sorry for pushing the source code public and for making a public announcement without considering that people would feel let down when the progress was slow. Some progress was made in the graphics display department. Some critical, some just like fun on the road, but overall, it makes this emulator progressing towards one-pixel display goal. So let’s get into the real progress report.

Last 4 months, I’m the only contributor with 113 commits to the master, 22,445 changes and 9,970 deletions. The reason why there are so many deletions is because there is a huge codebase cleanup that removes dead code and unused embed dependencies. But the changes are the exciting stuff, right?

On master

Cross-platform stability

EKA2L1 can now truly compile and run on Linux, which is a lot faster compared to Windows. EKA2L1 was mainly developed on Windows, on both my laptop and my desktop, but my laptop is my main development environment (Most of my spare time privately or during class is spent working on this). And it’s slow for the JIT to be there, so I switch my laptop to Ubuntu.

Working with libstdc++ and clang, a ton of issues have been discovered. These are some of the issues fixed:

Missing standard library headers: Some MSVC standard headers include other headers, which explains why it compiles fine on Windows (MSVC) but doesn’t work with libstdc++ (they keep their header clean from the others).

Some MSVC standard headers include other headers, which explains why it compiles fine on Windows (MSVC) but doesn’t work with libstdc++ (they keep their header clean from the others). Mismatch integer size: Symbian is a 32-bit system. EKA2L1 uses an open-source version of Symbian standard headers, where they defined a 32-bit integer (TUint32) as unsigned long. This, of course, is not a problem on 64-bit Windows, as unsigned long has the size of 4 bytes. But on Linux, unsigned long has the size of 8 bytes. This causes many tedious problems that I wasn’t aware of at the time of first running EKA2L1 on Linux. Only when I do a static assert, this would come up. The solution was forking the open source git repo and fixing TUint32 to unsigned int. This is not a good fix, and I will revise it in the future.

Along with those changes, which is pushed to master:

Travis CI for Linux is up: Allow developers to fix their PR on Linux without actually compiling on it.

Allow developers to fix their PR on Linux without actually compiling on it. Daily builds for PR and branches are available: You can download builds from the Google Drive repo here.

Scripting

It sounds really weird to implement scripting now, right? The emulator can’t even display graphics yet.

It has some use: Hook to some functions: The source code of Symbian from Symbian Foundation is public and stored. I need to keep track of where the emulating system is at (which line of code) to produce the best result with my work. This is what I usually do: see all called function s (yes, EKA2L1 can log all the API passed), and lurk the source code to find where I’m at. However, as the emulator progresses, all the apps branch into a separate branch. And appears of failure and exception. I need to know which functions return what or what arguments are passed to the API function to know what caused the error. A script is best for this. I just need to write a script like this: @symemu2.events.emulatorEpocFunctionInvoke(824932975) def leaveHook(): leaveCode = ctypes.c_long(symemu.Cpu.getReg(0)).value symemu.log('Leave code is: {}', leaveCode)

The 824932975 is the ID of the function User::Leave. I use that to get the exception code. There are also many other useful uses of the script like reading/writing memory, get a list of threads and processes running, get all the chunks, etc…

Scripting is done synchronous with the emulator, only in Python and it relies heavily on the decorator. I will try to add more languages in the future.

Debugging

The GDBStub is available! The code was ported from Citra into EKA2L1.

But the use of it was minimal. No ELF provided, and I can’t use IDA’s GDBStub backend without an ELF (I guess).

It’ll still be useful if people develop and use EKA2L1 to debug. Symbian SDK, when compiles, provides both ELF and the post-linker Symbian image. The ELF is used for debugging.

ON PERSONAL BRANCH

Dynarmic support is back

For a long time, the support of Dynarmic on the emulator was broken. The main reason is that I was using Unicorn fine, and I don’t want to touch it since it’s slow on Windows.

But as the progress advances, Unicorn is a bottleneck for me: When you stop and restart the JIT continuously after 20000 ticks for rescheduling and HLE IPC, Unicorn JIT cache seems to be going somewhere, because it takes Unicorn 36 minutes on Windows to load all plugins through an ARM loop, but only 1 minute for Dynarmic. Of course, no one wants to wait 36 minutes to run a GUI that just says Hello World, right? And (again), it may be because of my emulator doing things slowly, but this has nothing to do with that loop. The loop is just raw ARM code with 2 system calls repeated: GetThreadHeap and TrapHandler (for exception).

The fixes for Dynarmic are: Fix wrong memory mapping, That’s the cause of slowdowns and errors. It’s really dumb, but it lets Dynarmic runs flawlessly. Good work to MerryMage.

The video up above is the emulator running with Dynarmic as CPU backend.

Some work on save states

EKA2L1 doesn’t do a full bootup of the system, only the necessary segments that can’t be ignored. But it is slow. The simulator is slow already, and the system load time is also.

A basic save state works is in the process, but it should not process until the emulator is really stable and it doesn’t need to change to kernel objects and the timing system.

These objects should be saveable: Thread: We save thread context, name, id, stack chunk id, etc… Mutex: We save holding thread id, waiting for thread ids, name and its ID Semaphore: The holding threads and the semaphore count Chunk: Save all committed region compress with Deflate, the base address, the maximum size, and the committed top and bottom. Timer, Library, Etc… The library system: Saves all loaded binary with code chunk id, data chunk id, and parsed image infos. The timing system: Saves all future events.

The basic concepts of saving are really easy. Each object saveable should be based on class saveable_object , which has two methods: save_snapshot_to_file and load_snapshot_from_file.

, which has two methods: save_snapshot_to_file and load_snapshot_from_file. Note that the concept is already implemented in some kernel objects (chunk, thread, semaphore). It may be changed in the future.

ImGui debugger

In the last few days of October, I implemented a basic debugger to find out the root of annoying glitch from the guest (Symbian), this debugger will also have use in the future.

Including changes: Basic memory viewer for each process A disassembler for each thread. Currently, there is no way to jump freely since the disassembler can’t detect THUMB/ARM mode without CPSR register. Objects viewer for: Threads (id, name, stack pointer, state) Mutex (id and name) Chunks (base address, committed, max size, etc…) Breakpoints. Symbian image doesn’t provide anything about which section is ARM and which is Thumb, which makes setting breakpoints really hard (since we can’t detect ARM/THUMB mode). The way we do this in EKA2L1 (only implement in Dynarmic right now), is to watch the PC and pause the emulation thread when the PC hits one of the breakpoints. This hook is in the callback function MemoryReadCode of Dynarmic. Emulation window: The emulation window is no longer alone and black, now it’s together with all debugging windows.



To get this I had to: Move all rendering code to folder drivers. This folder contains all the OpenGL rendering code and all codes related to graphics, wrapped into a class called graphics_driver . The core interacts with other parts of the emulator (keyboard, graphics, …) through drivers. Each driver contains a command buffer queue (thread safe), and runs independently on a single thread and processes all of those command buffers when available.



Fixed wrong thread initialization process and double signaling a semaphore

Seems like each entry point of a process is used for both process and thread initialization. The entry point is named _E32Startup and written in pure ARM assembly, checking the register r4. If r4 = 1, the entry point will initialize the new thread. Also, some IPC calls signal the client thread twice, causing many undefined troubles, so I have to do something :P.

Migrating from HLE API implementation to system calls, More system calls implemented

I went from implementing all API calls to implementing all system calls and servers that are critical and can be HLE’d.

By now, 103/312 system calls have been implemented, whos number and name can be seen here (my personal branch has the latest implementation).

I think by the time the first GUI app displays, half of the system calls will be implemented. Most apps utilize the majority of them.

As I said that, many API functions should be HLE’d. For example, the GLES libraries.

New HLE IPC servers!

POSIX : POSIX server is spawned and associated with each process. The existence of this server is, of course, to maintain compatibility with the standard C library implementation so all the external third party libraries could work. Some EKA2L1 async POSIX IPC functions are not implemented right now, but most of them are: read, write, open, close, dup, dup2, etc…

: POSIX server is spawned and associated with each process. The existence of this server is, of course, to maintain compatibility with the standard C library implementation so all the external third party libraries could work. Some EKA2L1 async POSIX IPC functions are not implemented right now, but most of them are: read, write, open, close, dup, dup2, etc… Domain: Each process can be joined in a group called domain. A domain can be joined in a larger group called hierarchy. And each domain has a state, that is accessible to the joined processes. The use of the state is for a process to determine what should it do. For example, if the state is critical, the process will load and do critical things. That’s the use of the domain server, which is implemented in the link above.

Loader: The standard API of Symbian don’t like to call system calls to load process/library directly, but rather doing it through an IPC call to the Loader server. The loader server can do the following work: Create new process through the system call ProcessCreate Returns information of an executable/library. Create new library object through system call LibraryCreate Create new logical device by loading a driver (dll file). I won’t do that.



Either way, three of those first have been implemented, on the link above.

Window: The window server creates graphic context, sprite and drawing bitmap, which is basically framebuffer. It really doesn’t touch the GPU (that part is from the manufacturer, through a logical device loading call). That’s what I want to do, get it to draw a bitmap. The process for the window server hasn’t progressed much since I implement things when it needs to be done. But for now, it can create graphic device, graphic context and request a redraw.

FS: File server is nothing new. But since the last time I did a report, it has been updated a lot. 29/170 of them has been implemented, which is not much but it’s what the app uses the most. Also, the virtual file system has been improved and made faster. There are two file systems: ROM (in-memory) and physical. Searching for a file in ROM in a big folder (sys/bin with 3000 files), now only takes less than a second. This is all thanks to the quick sort all files name (when I first load the ROM info), and binary search! Hurrayyy.



Wiki documentation

Akawolf is kind enough to let me access and modify the Symbian Foundation archive wiki. I document my work there when I have time to spare.

It may interested people who want to do some learning with the OS or hacking if they like.

All of my document is based on what I found when emulating the system and reading the source code of the OS.

You can find the wiki link here.

Ending words

I’m sorry there is no graphical output from Symbian guest yet, which is basically what everyone desires.

The emulator now just needs a couple fixes for some weird bugs and to have all servers implemented that can, and should, be HLE’d.

With the new debugger, I hope to speed up the process and spend more time with the projects to not let anyone that supported me down.

Please hold on for more progress next month. If you want to ask more questions, you can join the Discord here. If you like my work, and you have enough trust and faith in me, I have a Paypal and Patreon.



Credits: Urosh Uchiha for helping me writing the report.