Here at Dia&Co coffee is for everyone! And our workplace is perhaps as far as you can get from what was depicted in the legendary 1992 film Glengarry Glen Ross.

That being said there is still value in the ABC: Always Be benChmarking! Much of what we do right now at Dia is creating new models, with freedom to choose their implementation, so a decent amount of R&D is required.

A question I found myself asking was the seemingly nonsensical “what is good and how do I get there?”. We’re just about sensible enough to know that we won’t make a near-perfect model on the first iteration, but when do we determine we have our MDP: minimum deployable product?

A simple empirical way to get towards “good” is to study the delta-improvement in the model as you add increasing complexity. However you want to do this as consistently as possible, and in the face of perhaps somewhat unknown new directions. So here’s an overly enthusiastic guide to adhering to your own low-tech benchmarking ethos:

S pend a lot of time thinking about the data you will use, so you can design model independent and implementation independent ways of engineering your features. This will reduce the amount of times you need to go back and modify the definitions of your features, which makes each iteration more difficult to compare.

so you can design model independent and implementation independent ways of engineering your features. This will reduce the amount of times you need to go back and modify the definitions of your features, which makes each iteration more difficult to compare. Always plan out the next few implementations you intend to try and develop your code base with the extensions in mind. Make sure you always design simultaneously for at least the current step and the next step in the plan, and hopefully the extensibility of your model will follow on naturally from this.

and develop your code base with the extensions in mind. Make sure you always design simultaneously for at least the current step and the next step in the plan, and hopefully the extensibility of your model will follow on naturally from this. Keep the data extraction and feature engineering stages separate from the model building stage , and make sure they are well indexed, meaning that “model i was run on data j” is something that is recorded.

, and make sure they are well indexed, meaning that “model i was run on data j” is something that is recorded. Create basic infrastructure that stores all attributes associated with the model you ran. This can be as straightforward as saving the results in a pickle file along with the code commit-hash, python environment details, config parameters etc: