The difference between MEMFS, IDBFS and NODEFS

By default, when you are using Emscripten to transpile any C/C++ libraries with file system operations, Emscripten uses a simulated file system called MEMFS to make sure libraries works under browser and node.js environment.

Use MEMFS is convenient and fast but it comes with few disadvantages:

As Emscripten can only use up to 2 GB memory, MEMFS makes it easier to run out of memory There will be a data “pass through” behavior between your main process and Emscripten (see “Solve a real-world problem: ffmepg.js file size limit”)

To resolve these issues, one way is to use IDBFS and NODEFS to play as the real file system for your application.

IDBFS, used in browser (and web worker) environment is to use IndexedDB as a file system to store your files

As IndexedDB has some synchronization issues, you need to use FS.syncfs() after writing files

NODEFS, used in Node.js environment is to use fs API in Node.js to simulate a file system.

How to mount IDBFS and NODEFS

To mount IDBFS and NODEFS, you need to use --pre-js introduced in Part.5. This time we need to overwrite a function called preRun (Details HERE).

A sample usage below:

Depend on your application, it should be OK to use other functions like preInit , but here we use preRun to do the task.

Solve a real-world problem: ffmepg.js file size limit

One day there is an issue reporting that ffmpeg.wasm cannot handle big files. To solve this issue, we first revisit our design:

It looks alright when the media file is not so big, but when the media file is as big as 100 MB, it looks unreasonable to pass such a big media file through postMessage() or send() and thus causes ffmpeg.wasm to crash.

The bottleneck here is caused by the fact that Web Worker / Child Process are acted as a pass through component to send and receive the media file, to solve this issue, we need to use a file system that can be accessed by both Worker and Web Worker / Child Process, so we do a redesign and here is the optimized design.

The idea here is to enable both Worker and Web Worker / Child Process to write and read from the IDBFS / NODEFS, which releases the bottleneck we see in the initial design.

Although it looks more complicated, but it solves the issues of processing big files in ffmpeg.wasm. Now you can try it in the CodePen below. (You can download a 90 MB video file)