Rewrite Software Projects You Use (aka Make Your Own Versions of Them)

I believe that reading through software projects and libraries you use is helpful in better understanding how the whole application works, but I think the next step in that journey is to pick some small libraries and make really simple versions of them.

Continuing on with the Express example, you can make an Express clone that only implements app.use . Doing just that gives you an end product that can handle all of the routing and rendering outside of the framework.

In order to get to that point, you need to create an application object that can have middleware registered. You’ll also need to decide how you want to store middleware, how to chain each of the functions together, and how to pass in a next function that will eventually call the next middleware.

Then you’ll need to figure out how to run an HTTP server to get the Request and Response objects/streams that you'll pass to your registered middleware functions.

Finally, you might add some default middleware functions like Express does to handle query string parsing and body parsing. Do you want to build in functionality for automatically parsing JSON request bodies, or do you want to leave that up to the user? It's totally your call. Maybe you can parse the query string, and let the user access it as a SearchParams instance instead of a plain old object.

Something you can also aim for is replicating the library's interface and then dropping it into a project that uses the reference implementation. You can get a lot of insight from drop-in testing it ⁠ — learning which parts break or aren't implemented yet.

I've been trying this out with Rapid, an Express clone. Is it going to replace Express? Hell no. Am I going to use it instead of Express for all of my personal projects? Probably not. But it's been fun, so far and there's plenty of more functionality to add or recreate. Also if you feel like making a PR on Rapid, they seem more than welcome.

Don’t just rewrite the projects verbatim, though. Copy the interface, but try the implementation out for yourself ⁠ — maybe you’ll get the opportunity to arrive at the same decision crossroads the original implementors did. You could end up making a different choice or at least appreciating the different ways you can tackle the problem. You can always go check your work against the reference afterward and improve it if the reference has a better solution. It probably will if it’s a large project like Express. Big projects can be encumbered with complexity from maintaining backward compatibility for features you might not need or care about.

Another cool project to recreate is Data Loader. Data Loader is a tool to batch load and cache resources. You make an instance of it by constructing it with a batch function that takes some keys and returns a Promise that resolves to the values that those keys relate to. The batch function could run a SQL query or hit a REST API or some other data source.

The user interacts with the instance by calling a load function with a single key and that returns a Promise which will resolve to the key's value when the batch function successfully loads the data.

The cool part is that Data Loader schedules the batch function to run after the current frame of execution by using some Node asynchronous primitives and caches the results. If you ask for the same resource multiple times throughout your web request lifecycle, Data Loader will only load it once. Also, you can ask for individual resources throughout the request lifecycle and Data Loader will batch load them at the end of each frame of execution.

Data Loader’s implementation is only one file (excluding tests), and there’s even a YouTube video from one of the creators that covers the entire source.

Writing your own implementation of that means you’ll have to cover concepts like creating promises, caching, using things like process.nextTick for asynchronous operations, and deciding how to handle errors from invalid user entries like bad batch functions.

Hopefully, you have some projects in mind that you use often but don’t really understand how they work under the hood.