I will soon reach the one-year mark of my fellowship at HMS, which seems like a fitting time to examine how effectively I have spent my time here so far. I have been a practitioner of self quantification long before the movement acquired its name, having tracked some aspect of my life since I was 16. Given the movement’s growing popularity, I thought it appropriate to share some of my life hacking experiments. My approach has cyclically peaked and waned in sophistication, something that I will expound upon later in the post, but I believe that the overall trajectory of my effort has been that of increasing usefulness. Any lifestyle change, particularly one that involves compulsive tracking of one’s behavior, ought to result in actionable information that is demonstrably useful and not merely be a quantitative exercise in vanity. In this post I hope to show that this can in fact be the case for self quantification.

There are two broad ways in which I use self tracking. The first and more important is as a tool to adjust my behavior in semi real-time, on the order of hours or days, to increase the efficiency and quality of my work. The second is more retrospective, where I analyze months or years worth of data, to reach conclusions about the long-term trajectory of my career and work habits. Below I will try to give examples of both.

Hours Worked

The first and simplest question to examine is the amount of time worked. Prior to the start of my tenure at HMS, and several times since, I set a target number of hours to work per day (varies depending on the day of the week). Below I plot the number of hours worked during each month of my job so far.

For a given month, the “filled up” part of the rectangle represents the number of hours worked. A white band at the top indicates the number of hours I fell short of my target, while a red band indicates the numbers of hours by which I exceeded my target. For real-time behavioral adjustment, I have similar indicators for hourly, daily, and weekly tallies, and so I am able to adjust my work ethic almost in real-time; to work harder if I’ve been slacking off, or take a break if I’ve been overworking.

Since I’m showing several months worth of data, let’s take a retrospective look. So far this year I have more or less hit my target. In fact, summing the white and red bands above I find that I am off only by 5 hours or 0.3%, which is quite remarkable. The only way that I could meet a target I set almost a year ago with such frightening accuracy is because of the constant adjustments I make thanks to quantitative tracking. What is also interesting about the above picture is that the time-span of my adjustments can take months. I started off lazy, probably because I was still adjusting to the new job and to a new city, but as time went on I managed to make up for lost time and increase my efficiency. If I were to dig within a given month, I would find a similar tendency but on the timeframe of weeks. For example months 6 and 8 above look very tranquil, as if I hit my targets without skipping a beat, but digging inside reveals a very different picture:

Month 8 looks reasonably calm. I started off a little bit under, but by the last two weeks I made up for it, similar to the months-long trajectory of my overall tenure so far. Month 6 is very different though. I overworked the first two weeks (week 1 is short because it’s just the weekend), crashed in week 3, and recovered in weeks 4 and 5. Lest someone laughs at the juxtaposition of “overworked” and “56 hours”, I should note that the above numbers represent hours of distilled work. Distractions like bathroom breaks, facebook/reddit, stretches, etc, are accounted for, and so in practice an “8” hour day corresponds to me being in the office for about 11 hours, and a “13” hour day corresponds to me working every waking hour of the day, including the daily bus ride. This in itself is interesting because it suggests that my efficiency, averaged over all types of work, is around 72%, something to which I will return in a bit.

A similar picture holds for hours within a day. Thus my behavioral adjustments are multi-scale, made in increments ranging from minutes to weeks, but in a basically subconscious way. I have a panel of indicators always open, and my brain internalizes what it sees and automatically adjusts.

Project-Based Analysis

Now that I know how many hours I spent working, the second obvious question is how those hours were spent. At any given day I am usually working on several projects, and I have set goals in the short-term (months) and medium-term (year+) for how my time should be distributed across these projects. Below I plot my intended targets (inside wheel) versus the actual distribution 10 months in (outside wheel). Numbers correspond to percentage of total time:

On the whole these numbers look quite good. The agreement, if you ignore the yellow and green slices for a moment, is quite notable. I must reiterate that on a day-to-day basis the actual distribution looks nothing like this. It is all very chaotic. I do set aside time on weekly and monthly bases to make sure I’m not veering too far off from track, but looking at the aggregate 10-month data for the first time, I am taken aback by how much quantitative tracking has helped me meet my targets. For reference, this is what things look like plotted as a function of time, where each time point is the weekly average for each project.

Nonetheless, what the summary distribution does show is that I blew it big time when it came to “Planning & Logistics”, on which I spent far more time than I had budgeted, and “Project 3” took the brunt of the hit. This information will be very useful to me in the coming months as I adjust my habits to correct this error. Coincidentally, when I split open the “Planning & Logistics” category, I see that a significant fraction went to one-time projects (daggered), an issue that I will come back to later:

Work Type Analysis

Another way to slice the data is by the type of work done. Below is a breakdown, averaged over the 10 months, of how my time was spent in terms of work type.

The categories are a little abstract and merit some explanation. “Thinking” involves the process of thinking/creating my own ideas, and includes actual thinking, coding, doing math, etc. “Reading” involves thinking too, but is restricted to the consumption of other people’s work, and includes, beyond reading papers, attending talks, watching online lectures, etc. “Writing” similarly involves thinking, but only of the type that is restricted to turning existing thoughts into the written word. The same is true for “strategizing”, and the rest are self-explanatory.

The above is a little grim or not too bad depending on your perspective. The core set of activities required for my scientific output are just “reading” and “thinking”, and constitute 48% of my time. Thus in a prefect world free from all obligations, including that of the dissemination of my work, I would be about twice as productive. Even if one concedes that dissemination is an integral part of the scientific mission, it would bring up the total to 59%. That leaves “logistics” (emails, meetings, …), on which a depressingly large amount of time is wasted, and strategizing, which while a fun mental activity, has consumed far too much of my time.

Instead of taking a summary view, one can also plot the number of hours spent as a function of time. I have a set of interactive gadgets that allow me to visualize all sorts of things. Below, on the left, I plot the number of hours spent per month on each work type category (color coding same as before), and on the right I plot the hours spent per week just strategizing.

What’s clear is that most of the strategizing was done early on, when I first began my job and had to plan a number of projects simultaneously. It was more or less an upfront, one-time investment. Nonetheless, the above picture tells me something very important that I did not know before. That despite strategizing being a one-time investment, it was such a hefty one that even 10 months later, it shows up as 14% of my total time. This is an issue that I repeatedly notice. Things that appear innocuous, for example spending a week here or there focused entirely on doing one thing and thinking “it’s just a week”, can in fact throw off the balance of my work distribution for months to come. My mind is preconditioned to think that a week is not a large amount of time, at least in academia. But this type of tracking quantifies what a serious hit it is to dedicate an entire week, let alone a month, to a single task.

On the left plot one also sees two dips in thinking activity, one corresponding with a peak in reading activity and the other with a peak in logistical work. The first corresponds to a significant investment I made in learning a new mathematical method, and the second corresponds to a lot of bureaucratic work that was needed to enable students to join my group. This effort begins to bear fruit as my mentoring activity begins to grow in June, coinciding with the time when a few students joined my lab. Of course, I did not need to look at these plots to know what happened, but they do allow me to quantify the amount of time spent in each case.

Work Type Targets and Distributions

I do not generally set targets for work type allocations like I do for the projects. The only hard targets I currently have is to spend >10% of time writing, and >15% of time reading, which I have hit. This was something I struggled with tremendously before, and I only succeeded when I started tracking work type specifically.

I do however have a number of soft targets. For one, I try to do intellectually heavy work in the morning when my brain is freshest (thinking and reading), and leave the less demanding work for the evening. Below is the distribution, as a function of time of day, for each type of work.

On the right I plot all aforementioned work types except reading, and on the left I zero in on thinking, strategizing, logistics, and writing. The first revelation is that my soft target is a complete fantasy. If anything, I spend more time on logistical work earlier in the day than I do on thinking. Writing and strategizing do have slightly higher peaks earlier in the day, which is news to me. These are all previously unknown facts about my work habits that I will now work to correct. In the right plot very different patterns emerge for mentoring, which is an activity that is still very much in flux, and socializing/networking, which as expected dominates in the afternoon. Just for fun, I radially plot the information on the left above on a 24-hour clock below. I do not find this visualization very useful, but I like to constantly experiment.

Another soft target is to maximize the length of uninterrupted time for any given activity. This too can be ascertained from the data, plotted below, where I show the distribution of every work type as a function of the length of uninterrupted time spent.

This is also unfortunately somewhat depressing, as the mode occurs somewhere around the half hour line. That means most of my activities run for about half an hour before being interrupted, most likely by my losing focus. On the positive side, more mentally intensive activities, thinking, reading, and writing, cluster in a group distinct from logistics and strategizing. They have a fatter tale that extends all the way up to 6+ hours, a state of true Zen that I achieve for only 0.79% of my tasks. I do however attain a more realistic state of 3 uninterrupted hours 5.4% of the time that I am reading.

The Technology

I will end with some remarks on the software that goes into making this possible. First off I want to qualify everything by noting that by far the hardest part of self quantification is not building the tools but the lifestyle changes, the nearly OCD-like tracking of one’s behavior, that I think presents the biggest obstacle to most people. Having said that, good tools make this process easier, by seamlessly integrating tracking within one’s daily minute-ly habits. The success and longevity of my various efforts over the years have correlated very strongly with how well the tool in question worked. I suspect that as wearable computing begins to make a serious dent in our habits, self quantification will begin to flourish even more.

When I first started seriously tracking things as a teenager I had a very elaborate and completely custom-built system, probably in the 10k+ lines of code regime. It told me way more than I needed to know about things I could do little to change, and consumed an inordinate amount of time to maintain. As I grew older and my responsibilities increased, I found it increasingly more difficult to dedicate so much time to this effort. For a while I dropped self quantification completely.

The system I’m discussing today is built in large part on existing tools. All the raw data is stored as events in an Outlook calendar, which acts as my database (previously I used a real database). The analysis is all done in Mathematica, using custom-built code that I developed. The only “real” software engineering that I had to do was a custom-built C# DLL that maintains an active link between Outlook and Mathematica, and that allows me to track changes in Outlook in near real-time using Mathematica’s Dynamic functionality. From the start, I wanted the system to be about practical utility instead of cute but useless statistics. As a result, I would only build a piece of functionality when I found myself asking the same question over and over again (e.g. how many hours did I spend on project X this week.) This has led to me having to refactor the code twice now, and probably many more times in the future, but I have found this organic approach to be far more effective than my previous (more structured) attempts.

Beyond the quantitative tracking system, I do have a large note taking and qualitative planning system, built on top of OneNote, Workflowy, and Visio. My experience here again has been of relying less and less on custom-code in favor of off-the-shelf tools. There is one feature that I would love to see built either into OneNote or Workflowy, but that is a post for a different time.

Concluding Remarks

I must admit to feeling some hesitation before writing this post, as I fear it may come across as self-indulgent. My motivation in writing it was to communicate how useful self quantification can be. It is a lifestyle change that I believe some can benefit from, and one that can reduce the chronic stress from which far too many of us suffer.

Update: Hello Hacker News!