The Homepage for myDrive

My first Open-Source project was creating myDrive, a Node.js, React, Express, and Mongo.db based Cloud Storage Solution (it’s basically a Google Drive Clone). This project basically started off with me playing around with Express, and finding an interest in allowing users to upload, and download files with it. Over time I started adding more and more to this project, and it eventually became the full-fledge application it is today! I also am a big fan of the Open-source community, and wanted to give back somehow. This project actually got kinda popular on Github, so far receiving over 800 stars, and 20K views. It really meant a lot to me seeing so many people interested in my project, which is why I am writing this article about some of the process.

I decided to use Mongo.db for this application, because I favored its No-SQL style, plus the Mongo Driver for Node.js is excellent imo, and is an absolute pleasure to work with. This being said, my ever-loving lust for mongoDB may have hindered the application in certain aspects. Everything is stored in mongoDB, and I mean everything, Users, Metadata, Images, File-Chunks, name any data and chances are its in the mongoDB database. The nature of MongoDB only allows Documents to be 16mb in size, because of this limitation, I was forced to use the built in Grid.fs driver, in order to spread the file into much smaller chunks. At the time I thought this was pretty cool, but as it turns out this does have a CPU impact (although it is very small), and storing the files right on the file-system would have put less strain on the Servers CPU! This is why in future versions of myDrive, I would like to implement a way to choose the backend database, a user could choose between Mongo.db, the file-system, or something like Amazon’s S3.

Structuring such a big Application was the biggest, and probably the most fun challenge for me. I needed to learn how to break myDrive into small, coherent pieces, each that have their own purpose, and allowed for easy testing and scalability. I will break this section into 2 pieces, one focusing on the Frontends Structure, and the other focusing on the Backends Structure.

Front-End Structure

The Front-ends structure is much simpler than the Back-ends, the most important thing to do here is abstract the logic away from the view itself, let’s see an example of this inside of myDrive.

PhotoViewer of myDrive, link to Github code

Notice there is no logic here, all methods are done through props passed through, how is this done exactly? Well, let’s take a look at the “Container”.

Container of the Photoviewer, Github Link

This is the Container, or the Parent of our view, this is where all of our logic will live, here we will import the view, and use it as a React Component, passing in all the methods we need. You would then import this Container/Parent Component, whenever you would want to use them.

Personally I name this Container file “index.js”, which allows you to just point at the Components folder location, and then Javascript will automatically use index.js by default. Removing some redundancy and overall reducing code.

Back-end Structure

The Back-ends is a little more complicated, it is broken up into 4 pieces (for the most part, some express routes only need 2–3 layers, but this is usually do to complications or some methods being too simple, but you should strive to break it up into 4 pieces).

The structure is as follows Express Route -> Controller -> Service -> Database Utility.

Express Route

File Express Router, Github Link

Make the express route as boring and dumb as possible, it should only access the file controller, have no logic, and not access the file service, or the file database utility at all. Notice how I handle authorization here with some middleware.

Controller

File Controller, Github Link

This is where all Error handling is held, for the most part there is no logic here either, this should just handle getting the data of the request (like getting the user, and id for this example), calling the service, and then sending data, or an error, back to the client.

Service

File Service Example, Github Link

The File Service is where most, if not all the logic lives, this is also where we will throw errors if anything goes wrong, for example, if we cannot find this file, or the user is not authorized to access this file. These are custom errors I created, they just extend the built in Error class, and adds a specific error code to them.

Database Utility

File Database utility, Github Link

The Database utility, as one may guess, only retrieves, updates, removes, and adds items to the database. There is absolutely no logic here, and the methods are very small. Because I structured myDrive like this, it should actually make adding other backend database methods much, much easier, so good job past me.

This all might seem like a lot, and some of it might even seem unnecessary, but using this structure will make adding features in the future much easier, allow other users to read your code easier, and make testing a breeze.

Regrets, and Other Challenges

There were some other challenges I faced while creating this application, one was design. I barely knew any CSS before making this, the little I did know was from a Udemy course, so a lot of the CSS was me learning as I created myDrive, and I feel like this is sometimes evident in the code. This being said, if you do also struggle with CSS, do not feel discouraged, you need to learn as you go, if I can do it, you can too. Also learn Sass! It makes dealing with CSS so much better.

Streaming Videos, was another challenge, it took me a couple days to get this working correctly, then later when I added encryption it further complicated things, and being the over achiever I sometimes am, I thought it would be a cool idea to add transcoding with FFMPEG. Let me tell you, mixing Encryption with FFMPEG for video streaming, is like mixing Pills with Alcohol, while it might seem like a fun idea at first, but in practice it was an absolute nightmare. And I wish I didn’t waste so much time trying to get it to work (it still doesn’t work very well, which is why its an optional add-on, and not recommended for myDrive, but video streaming without the transcoding works fine).

Lets wrap this up

Overall, I loved building this project, like I really did LOVE it, I never had so much fun building something on my own before. And seeing this project slowly come to fruition in-front of my eyes, with my own hands, really proved to me this is what I want to do for the rest of my life. That, and the community response I received was just absolutely hard-throbbing and exceeded any exceptions I ever had. It truly feels like I am part of the Open-Source community now, and this whole experience will be engraved into my well-being.