2. Building a compiler for a “web bundler”

This will be a massive oversimplification of how Webpack works, as there are many different ways to solve the problem, hopefully this way will offer some insight into the mechanisms involved.

The overview of a compiler is below, we will be breaking down each phase.

Our application:

Our application consists of 4 files. Its job is to get a datetime, then hand that to a logDate, whose job is to add text to the date and send it to a logger. It is very simple.

Our application tree is thus:

PHASE 1

Using a 3rd party tool for AST parsing we (see code below):

Determine files full path (very important so its clear if we are dealing with the same file again)

Grab files contents

Parse into AST

Store both contents and AST onto a “module” object.

Process the dependencies inside the contents (using the AST “ImportDeclaration” value), recursively calling this function with the value

Finally add that function to the depsArray, so we can build up our tree with the first file appearing last (this is important)

SO our tree now looks like the below right array:

PHASE 2

A compilers job is to “Execute code which will produce executable code”. This means we will have 2 levels of code so we will review them 1 at a time. First we will review what the compiler builds, then review the built/outputted code (run by the browser).

First the built code

Templates:

Module template: Its job is to convert a given module into a module our compiler can use.

We hand it the module code and an index (Webpack also does this with the index).

We want the code to be as compatible in as many environments as possible. ES6 modules support strict mode natively, but ES5 modules do not so we explicitly define strict mode in our module templates.

In NodeJS all ES modules are internally wrapped in a function attaching runtime details (i.e. exports), here we are using the same. Again Webpack does this.

Runtime template: Its job is to load our modules and give a id of the starting module.

We will review this more later, once we have the modules code inside it.

Custom import/export:

With our import statement we will be replacing the instance of “importing” with our own. It will look like the middle comment.

Our export will do something similar to the import, except replace any “exports” with our own. See bottom comment.

It is worth noting Webpack stores dependency IDs on the module earlier. It has its own “dependency template” which replaces the imports and exports usage with custom variables. Mine swaps just the import itself (theirs swaps the entire line and all usages of it). One of MANY things which aren’t exactly the same as the real Webpack.

Transform

Our transform function iterates through the dependencies. Replaces each import and export it finds with our own. Then turns the AST back into source code and builds a module string. Finally we join all the module strings together and hand them into the runtime template, and give the index location of the last item in the dependency array as this is our “entry point”.

Now the code outputted from the compiler:

The left hand side is our runtime, the right hand side shows all the “modules” which are loaded. You can see they are the modules we started with at the beginning.

What is going on?

The runtime template IIFE runs immediately handing the modules array as an argument. We define a cache (installedModules) and our import function (_our_require_). Its job is to execute the module runtime and return the exports for a given module id (the ID correlates to its location in the modules array). The exports are set on the parent module, utilising pass-by-ref, and the module is then stored in cache for easier re-use.. Finally we execute the import function for our entry point which will start the application as it does not require calling an export itself. All imports inside our modules will now utilise our custom method.