In Part 1 of this series I identified three areas I consider foundational skills for Data Scientists: SQL, Random Variables/Conditional Probability, and ML basics in R or Python. Knowing your way around these areas gets you a seat at the table and puts you in a position where you can start to add value. With these as a starting point we’re ready to move on to the next step, addressing how you level up. The first category I’ll tackle is developing systems thinking.

System: a set of connected things or parts forming a complex whole, in particular. a set of things working together as parts of a mechanism or an interconnecting network.

Why is it important to develop systems thinking? Because data is not produced in a vacuum. Rather it is generated by a system that is operating (or being operated) in the real world- a web application for searching the internet, a messaging service for connecting with your friends, an ecommerce site for buying whatever your heart desires. The data produced by each of these will be different because people are trying to fulfill different goals when using each one.



Understanding a system and its associated context guides you to useful questions, drives your experimentation to validate answers, and can help you design metrics that reveal whether you are truly succeeding at your goals (as well as revealing when you are kidding yourself). For instance, the call to action on each of the applications listed above is different. In the search app, it is the search button, in the messaging app, the send button, and in the ecommerce app, the buy button. Should the call to action be big or small? The answer in all likelihood depends on the system. For search or ecommerce we might want it to be large, since we want to make the action as easy as possible. For the messaging service though we might want it to be smaller, since otherwise it might take up too much screen space making writing messages annoying, which would decrease the usage of our product. So if we ran an experiment and generated data about button size vs. actions and didn’t take into account the context in which we were operating (Search/ecommerce vs. messaging) we don’t learn anything. Our data is useless without context, and the context comes from the system that generated it. Put another way, understanding the system that underlies your data helps you define what “good” means. Without it you have only numbers- values devoid of any meaning and your conclusions will be non-informative in the best case and nonsensical in the worst.



Given this, how does one develop systems thinking? There’s nothing like reps to get better at something, so I’m going to lay out some exercises below. The goal is to get you thinking about how data comes to exist, what you can learn from it, and (eventually) how you can leverage it for value. I advocate a 3 step process to get there:

Develop an understanding of the world around you through toy models. This will help you see systems. Develop product sense through building quantitative frameworks for valuing your models. This will help you identify the value layers in a system. Develop opinions, make predictions, and make yourself accountable to them. This will help you ensure that the intuition you develop doing (1) and (2) is spot on.

Step 1: Develop an understanding of the world around you through toy models

The first step to thinking in systems is to start noticing and trying to build an understanding of the systems in the world around you. A great tool for this are toy models. For those who are unfamiliar:

A toy model is a deliberately simplistic model with many details removed so that it can be used to explain a mechanism concisely.

They have heavy use in physics for building intuition around a problem. Through building a simplified version of the phenomena you’re interested in, you can learn a lot about how it works and even develop meaningful predictions. There’s a really neat example of this in radioactive decay, when unstable atoms spontaneously break apart into lighter elements, releasing energy in the form of radiation while doing so. To fully describe this process you need quantum mechanics and quantum field theory, if you want an exact calculation of the decay rate for a particular group of atoms and the amount of energy they release. However the first physicists to study this problem didn’t have those tools available and needed to come up with simpler approaches. To do so they built a toy model.

They reasoned that radioactive decay is a rare event, in the sense that the probability of a given atom decaying between two particular points in time is very low. If this were not so, every element we observe on earth would have already decayed into a stable state and we’d never measure any radiation.

This implied a process governed by poisson statistics, which contain a single parameter: the event rate or half life of the process. This parameter must encapsulate the entire high level description of the process, since there is no other information available.

With that they had something they could measure: Using a radiation detector such as a Geiger counter, repeatedly measure the number of hits over a defined short time period. From there it was relatively straightforward to verify that the counts did indeed follow a poisson distribution and extract the rate parameter and half life.

This might seem pretty trivial, but as it turned out this simple approach, once validated, ended up being the basis for methods like carbon dating. This toy model couldn’t explain all the details around the decay process, but was able to give some of the first accurate estimates of the age of the Earth!



There are many practical examples of toy models yielding valuable insights:

Sabermetrics in baseball, which models each interaction as an independent event with a given probability of a specific outcome (eg. hit, walk, out, run, etc.), and tying each outcome to the marginal increase in the probability of a win,. This yielded, among others, the insight that walks were equivalent to singles in that both resulted in a non-out event and a player on first base.

The efficient market hypothesis, which models participants in the stock market as having access to perfect information. This yielded the insight that market mostly moves together and thus price variations in stocks at any given moment are most likely random.

PageRank, which models browsing the web as a conditional random walk through web pages, where a surfer can only access a page if it was linked by the previous page they were on. Running time forward on this model, one finds that pages with the most links into them come dominate browsing time, implying they are the most important sources on a given topic.

None of these examples are strictly accurate with respect to the processes they seek to model, but all yield useful insights. Each of these were built by considering how the system that generated the data could have looked and working forward to understand the implications. So how do you learn to this? My suggested exercise is:

Find something you’re interested in. Hypothesize the fundamental principles behind that thing. Reason about implications of what you’ve discovered in (2). Verify in data. Repeat!

Some examples I like to get you started:

Do this enough times and you’ll find yourself breaking down every process you come across and gaining a deep understanding of how things work with much less information than others. But you’ll also find that it’s only part of the picture. Toy models will help you figure out how things work, but you’ll probably then start to wonder if some of those things should exist at all- whether they are producing outcomes that are valuable. This brings us to Step 2.

Step 2: Develop product sense through building quantitative frameworks for valuing your models

Once you get really good at building toy models, you’ll be able to look at any process and be able to construct a model that describes how that process generates data. However, it won’t tell you anything about the utility of said process. Does the process generate a useful outcome? How much is that outcome worth? Does the outcome justify the cost? Should that process even be happening at all?



To answer these questions you have to learn to reason about value produced, not just outcome generated. In fact, all of the toy models I listed above had a key value nugget that came out of them- a part of the insight that could be directly translated into a useful gain.

The sabermetrics result of walks being equivalent to singles implied that they should be priced the same by the market- that players who had a high walk rate should earn as much as those who had a high batting average but hit mostly singles.

The Efficient Market Hypothesis yielded “buy the market” strategies as opposed to stock picking or active investing.

PageRank as the basis of for web search took advantage of the fact that connections between web pages were far more informative than their content when determining web page importance, which vastly increased search relevance and decreased spam.

You can see pretty plainly that each of these toy models is pointed at a problem that was very much worth solving, impacting millions of people and generating hundreds of billions of real dollar value. Anyone who was working on these problems were clearly working on the right things. So how did they know? You’ll commonly find this skill of choosing the right things to work on labeled in technology circles as “Product Sense”, which is basically the art of identifying which problems are worth solving and which solutions to those problems are most valuable.



Excelling at this involves developing a valuation framework for any model, product, or process you’re exploring, the center of which is the core metric to optimize around.

For sabermetrics, the core metric is dollars per win, as the goal of a baseball team is to use their resources (payroll) to win games. It turned out that the market overvalued hits and undervalued walks, an inefficiency exploited by some small market teams to compete effectively with those with larger budgets. And eventually used by a large market team (the Boston Red Sox) to win a championship.

For the Efficient Market Hypothesis, the core metric is % growth of wealth- how much more money do you have a the end of a time period compared to when you started. This pointed investors to low load Mutual funds and ETFs, which have outperformed hedge funds as an asset class over the last 10 years (especially after accounting for the fees).

For PageRank, the core metric is # of searches per click (we want this to be low implying that the best result is found quickly). PageRank optimizes this by returning the most relevant results. This made the web a useful place to be and, of course, formed the backbone of one of the largest technology companies ever created.

I advocate the following process for learning to do this:

Pick a product or service you enjoy using and know well. Build a toy model describing how the product works- who its’ users are, how they find the product, how they convert and how they engage. Add in a framework that describes how the company generates costs and revenue- acquisition channels, operating costs, and monetization. Pick a feature you like and use your model and framework to figure out how much they’re making off of it. Have an opinion on whether they should have that feature or not, or what information you would need to have to make that decision. For bonus points, do the same with a new feature that they don’t already have. Repeat!

Here are a couple of examples of this that I really like that should get you started:

Running through this process a few times, especially if you couple steps (1) and (2) builds your product development muscles. It’s great prep work but doesn’t quite get you all the way there. There’s one critical piece missing: How do you know if you’re right? Or, put another way, how do you know that you’ve become good at this?

Step 3: Develop opinions, make predictions, and make yourself accountable to them

Which brings us to step (3), start making predictions, ideally with some skin in the game. Going through steps (1) and (2) a bunch of times is going to lead you to have opinions on everything. But the goal isn’t just to have opinions, it’s to have good opinions. The only way to get there is to add in a feedback mechanism. This means writing your opinions down, making and tracking predictions with them, and generating some accountability for yourself to them. The accountability piece is really important- without it you’ll tend to remember your successes and forget your failures and you won’t improve. Ultimately you’re going to use this process to make really difficult decisions for yourself (like where to work) and your company (like where to invest), so you really want it to generate good results.



Examples of opinions people have used to make a lot of money:

Investing in Amazon when it was unprofitable, but generating huge amounts of free cash flow.

Building another search engine when Yahoo and AltaVista had clearly won the market.

Buying real estate or stocks immediately after the 2008 crash.

Investing early in AirBNB when it was the crazy idea of allowing strangers to sleep in your home.

You want your opinions and associated predictions to be as good as these and accountability is the key to making this happen. Some low risk ways you can generate accountability:

Make your opinions and predictions public through blog posts, tweets, or newletters. (The sports analytics community typically goes this route.)

Start a side project based on one of your opinions, where you generate users and/or revenue.

Put some money behind your opinion by making investments.

I keep a log of all my opinions and the predictions I make with them, and try to put something behind them so that it will hurt (but not too much) if I’m wrong.



A few I was right about (at least up to now):

Amazon and Apple (circa 2004)

Bitcoin

Bay Area Real Estate post 2008

A few I missed on:

Microsoft and Google

Ethereum

Fitbit

I don’t always do the right thing with my opinions, but have gotten closer over time. But having some accountability and skin in the game has definitely made me better over time. And the confidence that results from that has made me more effective in decisions with higher stakes.

The key thing is that this last step closes the loop- by enforcing an accountability feedback step, you’ll learn whether you are successful at understanding systems and extracting value from them.

Wrapping up

If you run through these exercises you’ll know how to

Identify systems in the real world and analyze them through toy models. Apply product sense to discover which of those systems represent real opportunities by layering quantitative valuation frameworks on your models. And Hold yourself accountable to the opinions and predictions you come to in (1) and (2) to ensure you’re actually generating good ones.

Get here and you‘re starting to supercharge yourself, ready to contribute to the development of a business or a product at a very high level. And you’ll be ready for the next step, learning how to generate your own good ideas and getting those creative juices flowing. I refer to this as expanding your idea generation platform and we’ll cover it in Part 3.

(Looking to level up in your career? Build on your skills at WorldlyBayes)