Measuring karma was a heavily-debated topic for a while on HackerNews. The goal is to provide some measurement that both accurately measures overall contribution to the community and encourages consistent engagement. Several solutions were discussed and a few were even tried. For example, pg tried to replace the overall karma score with an average score. All three combinations (total only, average only, and both) were juggled around in the top right corner, until eventually the simple total was used.

While all these attempts were good ideas, I think there is an even better metric that should be used here: the Sharpe ratio.

Risk adjusted returns

When you look to invest in an asset, it's often important to consider the risk of an investment as well as the overall historical returns. Assuming you have no external knowledge of the asset, looking at the volatility of its returns is a pretty good estimate of risk. Ideally, you'd like to see super high returns with no volatility– just a steady stream of money rolling in day after day. The Sharpe ratio is a way to combine the returns of an asset and its historical risk into a single number.

Simply defined, the Sharpe ratio is the return of the asset minus the risk free rate, all divided by the standard deviation of returns. For instance, if we have an asset which has returned an average of 10% per year and an annualized volatility of 20%, and we assume that we could park our money in a savings account for a risk-free 1% per year, then the Sharpe ratio for this asset is (10 - 1) / 20 = 0.45.

Naive Sharpe karma

How does this translate to a social news site like HackerNews? First, let's consider each user to be an investable asset. Each time a comment or story submission is made, that user is generating a return. Since each comment starts with a default score of 1, let's go ahead and make that the risk free rate of return. So the karma metric I'm proposing would be:

where:

is the user

is the number of comments for user

is the karma score for the ith comment of user

is the standard deviation of the karma scores for user

In programmer-speak, the pseudo-code for this is :

function karmaSharpe(user) { if(user.comments.length == 0) return 0; var numerator = 0; foreach(var comment in user.comments) numerator += (comment.karma - 1) / user.comments.length; var denominator = stdev(x => user.comments.karma); if(denominator < 1) denominator = 1; return numerator / denominator; }

A better metric

The Sharpe ratio is certainly an improvement over the previous two metrics. Totals give a very low-fidelity view of the user's contribution and averages make no distinction between consistent, high-quality members and more skewed users who have a couple really highly-upvoted submissions. The Sharpe ratio addresses both of these shortcomings by including an additional dimension in the calculation: volatility.

The metric is still not perfect, however. A good metric would include time in the formula, since a high-quality user would be consistently contributing every day. An improved metric is below:

where:

is the user

is the number of days user has been registered

is the number of comments for user made on day

is the karma score for the comment of the day for user

is the standard deviation of the karma scores for user

And in pseudo-code:

function timeAdjustedKarmaSharpe(user) { if(user.comments.length == 0) return 0; var excessReturns = dailyAverageExcessReturns(user); return karmaSharpe(excessReturns); } function dailyAverageExcessReturns(user) { var groups = user.comments.groupBy(c => c.Date); var excessReturns = groups.select(g => g.sum(c => c.karma - 1) / g.count); return excessReturns; }

Conclusion

It may not look very simple, but it actually is straight-forward. We are effectively saying that each user should generate an average comment score of 1 point per day to break even. Anything you make beyond 1 point is considered an excess return for the day. We then simply take the Sharpe ratio of the average daily excess returns. The resulting metric ensures that users are incentivized to make consistent, high-quality submissions and punishes one hit wonders and those who take a spray-and-pray approach.

If you liked this article, please upvote it on HackerNews.

Footnotes