Designing the right file architecture is not straightforward in Machine Learning. After struggling on that question for a few projects of my owns, I started to discover simple patterns that cover most of the use cases I stumbled upon when reading code or coding my own stuff.

This article is about sharing those discoveries with you.

Disclaimer: This article is more a proposal than a definitive guide, but it succeeds quite well for me. It is intended to provide a starting point for beginners and maybe bootstrap a conversation. Since I struggled to architecture my own works at first, I thought I could share this part of my work. Of course, if you have better files architecture principles, please contact me so we can speak!

What one usually needs to achieve?

Think about what you will probably have to do when you will be doing ML (coding perspective):

You will have to code a Model

This model will have (at least) two different stages: training stage and inferring stage (production)

You will have to feed a dataset to this model (training stage)

You might have to feed it single elements too (inferring state)

You will have to tune its hyper-parameters

To tune them you will have to make your model configurable and create a kind of “API”, at least an endpoint where you can push a configuration

You will need a nice folder architecture for training results (so you can skim through it and remember each experiment easily)

You will need to graph some metrics like the loss or accuracy (in training AND production phase)

You want those graphs to be easily searchable out of the box.

You want to be able to replicate any experiment you do

You might even want to be able to jump back in time in the training phase to check your model a posteriori.

That’s quite a lot and one can be easily lost in refactoring files and folders to make everything fit nicely, on top of that, there are probably other needs that I’m not even addressing in the list above so let’s try to find out some best practices.

The overall folder architecture

An image is worth a thousand words:

Files architecture

README file: Most of the people are probably using Github, so please take some time and write a good markdown with at least those sections: “ Installation ”, “ Usage” , “ Test” , “ Useful Links” for any too heavy files to put directly into your repository.

Most of the people are probably using Github, so please take some time and write a good markdown with at least those sections: “ ”, “ , “ , “ for any too heavy files to put directly into your repository. main.py file: Unique endpoint, simple. More information below. You can also do its variation with two files ( train.py/infer.py ) but I don’t really see the need and it usually ask for building two APIs for each file.

Unique endpoint, simple. More information below. You can also do its variation with two files ( ) but I don’t really see the need and it usually ask for building two APIs for each file. data folder: create a folder and put inside a script to download your datasets. Make your script prepare the folder nicely if needed, for example, the script can create the train/val/test sub-folders if they don’t exist.

create a folder and put inside a script to download your datasets. Make your script prepare the folder nicely if needed, for example, the script can create the train/val/test sub-folders if they don’t exist. models folder: This is where you put your model files. I don’t think there is only one way to deal with that folder, one can write a file per model or a filer per model type or even have sub-folders. Just try to keep consistency.

This is where you put your model files. I don’t think there is only one way to deal with that folder, one can write a file per model or a filer per model type or even have sub-folders. Just try to keep consistency. __init__ file: I dig more into this file below but It’s a python helper to make your models more accessible and simplify the potential complexity of the models’ folder.

I dig more into this file below but It’s a python helper to make your models more accessible and simplify the potential complexity of the basic_model.py file: I dig more this file below too. I think most of the models in TensorFlow can share a common architecture, I’ll explain what choices I made and why.

I dig more this file below too. I think most of the models in TensorFlow can share a common architecture, I’ll explain what choices I made and why. hpsearch folder: This folder will contain any helpers for doing custom hyper-parameters search. If you use a library to do it, you potentially don’t need it but a lot of time you need to customise something anyway. Keep those functions pure and separated so you can test them easily.

This folder will contain any helpers for doing custom hyper-parameters search. If you use a library to do it, you potentially don’t need it but a lot of time you need to customise something anyway. Keep those functions pure and separated so you can test them easily. tests folder: A test folder … for your tests … you do them, right?

A test folder … for your tests … you do them, right? results folder: Obviously, this one will contain results. More information in TF on how to provide a good sub-folder architecture for TensorBoard below.

Note: Add a “.gitkeep” file in the results folder and add the folder to your “.gitignore” file. You probably don’t want to push all your experiments to your Github and yet avoid your code to break from a fresh install because this folder is missing.

This is the very basics. Of course, one might need to add other folders but somehow it seems to boil down to this basic set.

With a good README and other bash scripts as helpers, anyone coming to your repository can simply follow the “Install” commands and “Usage” commands to start reproducing your work 👌

A man of culture

A basic model

As I said, I ended up recognising patterns in the way models are engineered in TF. This leads me to design a very simple class which would be extended by all my future model.

I’m not a fan of class inheritance but I’m neither a fan of rewriting the same piece of code forever and clearly, when you do ML, models share many similarities needed by the framework you use.

So I tried to find an implementation avoiding the known banana problem of inheritance while letting one inherits as deep as it wants.

To be completely clear, consider this class as the top parent of future models, leading to the construction of your model in one line with one argument: the configuration.

For the sake of clarity, let me show you the commented file directly:

Basic model file

Some notes:

Regarding the “best” attribute of my configurations: People usually ship their code without the model best hyper-parameters, does anyone know why (honest question)?

The random function is static because you don’t want to instantiate your model to have access to it and yet, one can wonder why adding it to the model itself? Because it is usually completely tied to the custom parameters of the model. Note that this function must be pure and can be made as complex as needed.

The example init function is the simplest version, it will always load the last saved models of an existing folder or (if the result_dir is empty) initialise randomly using the init_op

The __init__ script

The __init__ script you can see on the folder structure does not have much to do with ML. Yet it is a simple way to make your code more accessible for yourself and others.

It adds a few lines of codes to make any model class directly accessible from the namespace models : So you can type anywhere in your code: from models import MyModel , whatever how deep is your model folder architecture.

Here is an example of a script achieving this task:

Nothing fancy, but I found it useful so I decided to add it to this post. 🍺

The shell API

We have a global consistent folder structure, a nice basic class to build our models, a nice python script to load our class easily but designing the “shell API” and especially its default value is of the same importance.

Because the main endpoint to interact with your work in ML is definitely the shell whatever tool you are using. The shell is the cornerstone of your experiments.

The last thing you want is having to change hardcoded values in your code to iterate over those experiments, so you need to have access to all the Hyper-parameters directly from the shell. You also want to have access to all other parameters like the result directory or the stage (HP search/Training/inferring) etc.

Again for the sake of clarity, let me show you the commented file directly:

That’s it! Hope it will help newcomers in the field to bootstrap their own research and feel free to comment or ask questions.