: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead inon line

Because volatility seems to cluster in real life as well as the markets, it has been a while since my last article. Sorry about that. Today we will be taking our first giant leap along A Non-Random Walk down Wall Street.

The Non-Random Walk Series

A Non-Random Walk Down Wall Street is the cheeky title of an academically challenging textbook written by Lo and MacKinlay in response to the best-selling Wall Street classic, A Random Walk Down Wall Street, written by Professor Burton Malkiel. A Non-Random Walk Down Wall Street is a collection of papers which challenge the prevailing random walk hypothesis. Despite containing only outdated results and being mathematically unforgiving, it's an impressive textbook which has inspired me to write a series of articles about it.

This series of articles has the following goals: Bolster or invalidate my original findings using the NIST test suite; Translate Lo and MacKinlay's papers and tests into more intuitive terms; Extend the results to the present day to determine if they are still relevant; Extend the results to emerging markets with a strong focus on South Africa; And bridge the theoretical and practical gap between machine learning and market randomness. Whew!

This series is also inspired by many of the thoughtful comments I received after I published my post about the random walk hypothesis, Hacking The Random Walk Hypothesis. So please keep the comments coming.

P.S. Simply because market randomness tests don't exactly make for great WordPress featured images, the featured images for this series of articles will be screenshots from all of my favourite Wall Street inspired films. This article's featured image is from the most recent Wall Street inspired film to hit the box office: The Big Short. If you haven't seen it yet, do yourself a big favour and go watch it. Afterwards, if you want more information you can always suffer through an earlier, non technical article of mine: A Recipe for the 2008 Financial Crisis.

Article Outline

This series will début with Lo and MacKinlay's first paper: Stock Markets Do Not Follow Random Walks: Evidence from a Simple Specification Test. In this paper Lo and MacKinlay exploited the fact that under a Geometric Brownian Motion model with Stochastic Volatility variance estimates are linear in the sampling interval, to devise a statistical test for the random walk hypothesis. This post covers the theory and application of this test.

This post is broken up into the following sections:

Should you have any criticisms or comments about this post or the random walk hypothesis in general, please let me know via the comments section at the end of this article. I always appreciate the input.

Efficiency, The Markov Property, and Random Walks

The random walk hypothesis is a popular theory which purports that stock market prices cannot be predicted and evolve according to a random walk. This hypothesis is a logical consequent of the weak form of the efficient market hypothesis which states that: future prices cannot be predicted by analyzing prices from the past ...

To a statistician the assertion that future prices cannot be predicted by analyzing prices from the past goes by a different name: the Markov property or, more intuitively, memorylessness. Any time series which satisfies the Markov property is called a Markov process and Random Walks are just a type of Markov process.

The idea that stock market prices may evolve according to a Markov process or, rather, random walk was proposed in 1900 by Louis Bachelier, a young scholar, in his seminal thesis entitled: The Theory of Speculation. In his paper he proposed using Brownian motion, a Markov (and Martingale) process, to model stock options. That said, it wasn't really until 1973, when the Black Scholes formula for derivatives pricing was published, that the idea gained traction.

Since then the use of stochastic processes for derivatives pricing has become industry standard. That having been said, the philosophical question regarding whether or not stock market prices really evolve according to a random walk or, at the very least, according to the popular stochastic processes used in industry today, remains. To paraphrase Queen, we are left wondering: is this [The Random Walk Hypothesis] real life? Is this just fantasy?

Is this the real life? Is this just fantasy?

Personally my mind rebels against the theory because it is too elegant; too simple. I like complexity; I like chaos. So I choose to spend my spare time learning more about the theory of randomness and I enjoy trying to find ways to test the conventional wisdom ... and hopefully someday learn to beat the market consistently ;-).

Back to the top

Variants of the Random Walk Hypothesis

A typical test of the random walk hypothesis involves three steps. First off you assume that asset prices do evolve according to a random walk and you select an appropriate stochastic model. Secondly, you define which statistical properties you would therefore expect to see in asset prices. And lastly, you test whether or not the asset prices exhibit the expected properties. If the asset prices don't exhibit the expected properties, then the assets don't evolve according to the model of the random walk hypothesis you assumed they did to begin with.

It's not good enough to simply state that market returns aren't random, you need to also specify what type of random they aren't.

Admittedly the fact that I didn't follow this process exactly in my previous article on Hacking The Random Walk Hypothesis was its biggest shortcoming. Luckily a supportive statistician on Reddit helped me see the light: it is not good enough to simply state that market returns aren't random, you need to also specify what type of random they aren't. In light of this below I have defined three popular forms of the random walk hypothesis.

RW1: The first and strongest form of the random walk hypothesis assumes that the random disturbance, , is independent and identically distributed (IID). This corresponds to the Geometric Brownian Motion Model wherein volatility of the random disturbance, , allows only for homoskedastic increments (constant ). Under this hypothesis, variance is a linear function of time (discussed in more detail in the next section).

RW2: The second, weaker form of the random walk hypothesis relaxes the identically distributed assumption and assumes that the random disturbance, , is independent and not identically distributed (INID). This corresponds to the Heston Model wherein the volatility of also allows for unconditional heteroskedastic increments. Under this hypothesis, variance is a non-linear function of time (discussed in the next section).

RW3: The third and weakest form of the random walk hypothesis relaxes the independence assumption meaning that it allows for conditional heteroskedastic increments in . This corresponds to some random walk process wherein the volatility either has some sort of non-linear structure (it is conditional on itself) or it is conditional on another random variable. Stochastic processes which employ ARCH (Autoregressive Conditional Heteroscedasticity) and GARCH (Generalized AutoRegressive Conditional Heteroscedasticity) models of volatility belong to this category.

In other words, any successful refutation of the random walk hypothesis must, ultimately, be model dependent. Furthermore, that model must clearly fall within the spectrum above. It just so happens that the weaker the form of the random walk hypothesis, the harder it is to disprove and the more powerful your statistical tests need to be.

Any successful refutation of the random walk hypothesis must, ultimately, be model dependent - paraphrased from Lo and MacKinlay's book.

To illustrate this point consider how easy it would be to show that some asset's prices don't evolve according to Brownian Motion but, on the other hand, how difficult it would be to show that the same asset's prices don't evolve according to some stochastic process without independent increments and with conditional heterskedasticity!

Back to the top

Stochastic Model Specification

The model which we will be basing our statistical test of the efficacy of the random walk hypothesis on is a stochastic log price process which is acted upon by a drift and stochastic volatility component. We also define a simple model of stochastic volatility which is used later on to produce the results for simulated asset prices. That having been said, the reader should please note that the test is robust to most forms of non-conditional heteroskedasticity. In other words, we will be testing the second variant of the random walk hypothesis, RW2.

Stochastic Log Price Process

Let denote the price of some asset at time and define as the log-price process. This log-price process satisfies the Markov property and is given by the following recurrence relation:

where represents the drift component and is a random disturbance samples from some distribution with an expected value of zero, . In the results on simulated data section we show that the results of the test on two versions of the above model: one with homoskedastic increments which is essentially Geometric Brownian Motion (this model relates to RW1); and another with unconditional heteroskedastic increments which is essentially Geometric Brownian Motion with Stochastic Volatility (this model relates to RW2).

The Homoskedasticity (RW1) and Heteroskedasticity (RW2) Versions

In the homoskedastic version of the model the random disturbances, , are sampled from a Gaussian distribution,

This corresponds directly to the Geometric Brownian Motion model. In the heteroskedastic version of the model the random disturbances are sampled from a Gaussian distribution with a stochastic ,

where

This is essentially Geometric Brownian Motion with stochastic volatility, however, I would like to stress that this is not the same as the Heston model which I have worked with before. It is a simplification.

Nevertheless, the desired effect of stochastic volatility namely, fatter tailed distributions and a higher volatility, is clearly present as can be seen from the two density plots below and the following two time series plots.

At this point you may be wondering what all the fuss is about. Well, we want a test for the random walk hypothesis which passes (it concludes the market is random) even if the returns demonstrate heteroskedastic increments and large drifts. Why? Because both of these properties are widely observed in most historical asset price data (just ask Nassim Taleb) and neither invalidate the fundamental principle underpinning the random walk hypothesis, namely the Markov property (unforecastibility of future asset prices given past asset prices).

In the next section we will show how the parameters and can be estimated. In the section following that Lo and MacKinlay's variance ratio test, which is robust in the presence of drift and heteroskedasticity in but is still sensitive to the autocorrelated increments in , is defined. Then results will be shown.

R Code for the Stochastic Model

Given values for and , the R functions below work together to produce log-price, price, and discrete returns processes of any length. These processes are later used to test the calibration and the variance ratio test.

View the code on Gist.

Using this code is quite simple. To chart fifteen asset price paths each five years long with an without stochastic volatility I would just need to type the following commands into the R command prompt.

View the code on Gist.

And I would get something like this out in the plots.

Stochastic Model Calibration

The key to understanding how the variance ratio test works, is to understand the different ways in which the parameters and can be calibrated using their maximum likelihood estimators.

Maximum Likelihood Estimator for

The parameter represents the component of daily returns which are attributable to drift. Given a log price process, , containing observations, the maximum likelihood estimate of is given by,

or rather,

This can be computed in R using the following function,

View the code on Gist.

And below is a graph illustrating how the estimates for get more accurate as the number of observations in the log price process gets larger (more data = more accurate). The dots indicate the true value of which was set of 0.1, and the lines indicate estimates of from generated log price processes with stochastic volatility of length 1 year up to 50 years. Ideally we want as much data as possible in our tests which is why the results at the end are limited to assets which have been trading or have been tracked for at least ten years.

Maximum Likelihood Estimator forwith discrete samples

There are a few ways to estimate . You could calculate the standard deviation of the log price process using every observation in . Or, you could sample every second point along and estimate :

Use every observation: Use a subset of the observations:

We can express this idea in terms of a sampling interval, , whereby every observation is used to estimate . When we use every observation, when we use every second observation, and so on and so forth. Given a log price process, , containing observations and a sampling interval, , the unbiased maximum likelihood estimator for the parameter is given by,

This can be computed in R using the following function,

View the code on Gist.

And as with below is a graph illustrating how the estimates for get more accurate as the number of observations in the log price process gets larger (more data = more accurate).

Maximum Likelihood Estimator forwith overlapping samples

The problem with the above estimator for is that as we increase our sampling interval we reduce the number of observations we have available to use to estimate ... and this results in a deterioration of the estimate.

A significant improvement on this estimator can be obtained by using overlapping samples. Whilst this does bias the estimator, Monte Carlo simulations done by the authors at publication (and myself) show that this is negligible and the estimates of using overlapping samples is, more often than not, more accurate.

Given a log price process, , containing observations and a sampling interval, , the overlapping unbiased maximum likelihood estimator for the parameter is given by,

where

This can be computed in R using the following function,

View the code on Gist.

And as with the previous estimators we see that as our sample size increases the estimates improve. It is hard to see in this graph, but the error is smaller than the previous estimator for .

The critical characteristic of our two models is that volatility is linear in the sampling interval, meaning that the volatility calculated using half as many points ( ) should be twice as large as a variance estimated using all of the points ( ). The estimators take this into consideration so whether you estimate with , , or even , you should get similar estimates. This observation is the heart of the variance ratio test.

Variance Ratio Properties and Statistics

Because estimates of at different sampling intervals should converge to the same true value, we can define two test statistics whose expected value under the model is zero. These statistics are called variance ratios.

: differences using the overlapping samples estimator.

This statistic is computed as the difference between the estimate for given a sampling interval and the estimate for given a sampling interval of 1. Given that and should converge, the expected value of is zero. Mathematically this variance ratio is expressed as follows,

And in R we can compute with the following function,

View the code on Gist.

Now let's test to see whether or not this is true. Below is a graph of the computed statistics for and for 500 randomly generated ten year long log price processes with random values for and random values for with and without stochastic volatility. As can be seen, the values for are always close to zero.

One observation that can be made is that as the sampling interval increases there is a perceived "degradation" in performance of the statistic; this is actually expected as the limiting distribution of the statistic widens as we increase . A much more significant observation is that the statistic with and without stochastic volatility for any is almost indistinguishable, which speaks to the robustness of the statistic.

: differences using the overlapping samples estimator.

This statistic is computed as the ratio of the estimate for given a sampling interval and the estimate for given a sampling interval of 1 minus one. As with the expected value of this statistic is zero,

And in R we can compute with the following function,

View the code on Gist.

Let us again test to see whether or not this is true. Below is a graph of the computed statistics for and for 500 randomly generated ten year long log price processes with random values for and random values for with and without stochastic volatility. As can be seen, the values for are always close to zero.

As with the statistic, we see a widening of the limiting distribution and an almost indistinguishable difference between the log price processes with and without stochastic volatility. We also see that this statistic is more sensitive than the differences statistic. This is a good characteristic.

A Heteroskedasticity-consistent Variance Ratio Test

I should warn you, this is the part of the article when things get a bit hairy 😯, it took me quite a few readings of Lo and MacKinlay's paper for the idea below to click and for me to implement it in R. I've tried my best to explain it.

Because most academics and practitioners agree that the volatility of asset prices do fluctuate over time, Lo and MacKinlay wanted to make the variance ratio robust to changing variances a.k.a heteroskedasticity and stochastic volatility. Thus any rejection of the random walk hypothesis using their test would not be due to the stochastic nature of volatility or long-run drifts, but rather due to the presence of autocorrelation in the increments of .

Any rejection of the random walk hypothesis using Lo and MacKinlay's test is not due to the stochastic nature of volatility or long-run drifts

The reason why they wanted to focus on testing for the presence of autocorrelation in the increments of is because this indicates whether or not satisfies the Markov property and, as per the logic laid out at the start of this article, whether or not future prices could (at least in theory) be partially forecasted using historical prices. To achieve this objective we start with a rather simple observation, namely:

"As long as the increments are uncorrelated, even in the presence of heteroskedasticity, the variance ratio must still approach unity as the number of observations increase without bound. This is because the variance of the sum of uncorrelated increments must still equal the sum of the variances." - Lo and MacKinlay

That having been said, how quickly these variance ratios tend toward unity and their asymptotic variance depends on the nature of the heteroskedasticity present in . In light of this there are two options: either you can specify the model of heteroskedasticity which you are testing for upfront (limiting); or you can make some simplifying assumptions about the nature of the heteroskedasticity present in in order to generalize the test to any model of heteroskedasticity which satisfies the simplifying assumptions (more widely applicable).

Lo and MacKinlay opted for the latter approach, assumed that the model of heteroskedasticity has a finite variance, and developed their heteroskedastic-consistent variance ratio test accordingly. What this means is that the statistical test is valid for most forms of stochastic volatility used in mathematical finance but not all. In particular the variance ratio test they defined is not applicable to models of heteroskedasticity from the Pareto-Levy family. The assumptions made by Lo and MacKinlay for the null hypothesis are shown in the box below,

We consider the following null hypothesis : (A1) For all , , and for any Explanation: contains uncorrelated increments across all periods of time. (A2) is a -mixing with coefficients of size or is a -mixing with coefficients of size , where , such that for all and for any , there exists some for which: (A3) Explanations: The second moment is finite, this is assumed because if it wasn't then the variance ratio is no longer well defined i.e. we can't compute a variance ratio is the variance can be infinite. (A4) For all , for any non-zero and where

I know how heavy that looks, but the intuition is simple. Our null hypothesis states that: possesses uncorrelated increments but does allow for general forms of stochastic volatility provided that the second moments and the estimated values for are also finite. And assuming this is true then we are correct in stating that:

approaches zero under

Given this Lo and MacKinlay derive the asymptotic variance of , from which the distribution of expected values for under heteroskedasticity can be calculated and used to test the random walk hypothesis:

Denote by and the asymptotic variances of (the autocorrelation co-efficient) and , respectively. Then under the null hypothesis : (1) The statistics and all converge almost surely to zero for all as increases without bound. (2) The following is a heteroskedasticity-consistent estimator of : (3) And the following is a heteroskedasticity-consistent estimator of :

Given a log-price process and a sampling interval, , the following R function can be used to estimate (the asymptotic variance of the variance ratio statistic),

View the code on Gist.

An when we have , given the log price process, , and the sampling interval, , we can go ahead and standardize the statistic to arrive at the final standardized test statistic, . Finally!

The above -score can be computed using the following R function,

View the code on Gist.

And since this is still asymptotically standard normal, we can use the very common significance levels to check whether or not the value of for any given asset is statistically significant. If it is, then we are either 95% or 99% certain that the asset prices were not generated by a Geometric Brownian Motion model with stochastic volatility, there is some statistically significant autocorrelation in and, most importantly, the asset probably doesn't evolve according to a random walk and there may be some level of forecast-ability in .

P.S. If the above explanation (which is simplified when compared to the full derivation done by Lo and MacKinlay) did not make much sense, then I would like to direct you to their original paper. And if you are interested in the size and power of the statistic for finite samples, Lo and MacKinlay wrote a follow-up article about this topic as well:

Results obtained on Simulated Asset Prices

Before we get to the results on real asset prices it is a very good idea to test our implementation of Lo and MacKinlay's model by computing for randomly generated log price processes with and without stochastic volatility. If the resulting distribution of -scores is normally distributed (which we will test using the Shapiro-Wilk test) then we can be quite confident that our code works and that it is, hopefully, free of any serious bugs.

The graph below shows the density of a 2500 random numbers samples from a normal distribution (red) versus the density of -scores computed for 2,500 log price prices with the following parameters:

A random number of years uniformly distributed between 5 and 25 years, A randomly selecteduniformly distributed between -0.25 and 0.25, A randomly selecteduniformly distributed between 0.05 and 0.75, Homoskedastic increments ini.e. constant volatility, and Computed using a sampling interval of two,

The two p-values computed using the Shapiro Wilk test were 0.0545 and 0.3476 for the normally distributed random variable and the computed -scores respectively. These both indicate that the distributions are normal. And lastly, the two line graphs below show the QQ-plot of the normally distributed random variable (right) against the QQ-plot of the computed -scores. They both look reasonable to me.

And below we have exactly the same simulation results except that they are with stochastic volatility applied to the log price process. The type of stochastic volatility is the one defined under the model specification section.

The two p-values computed using the Shapiro Wilk test were 0.6363 and 0.7011 for the normally distributed random variable and the computed -scores respectively. These both indicate that the distributions are normal. And lastly, the two line graphs below show the QQ-plot of the normally distributed random variable (right) against the QQ-plot of the computed -scores. Again, they both look reasonable to me.

What can we tell from the above results? First of all we know that if asset prices are generated using a Brownian Motion model with drift and stochastic volatility then they are, more likely than not, going to be marked as random walks using this test (95% or 99% sure depending on the confidence interval). Secondly, we are more confident that my code works. Testing quant code is hard so sometimes you have to resort to Monte Carlo simulations!

Results obtained on Real Asset Prices

The results below are broken up into two sub-sections:

Results obtained on 50 stock market indices from around the world, Results obtained on the current S&P 500 constituent assets, and

Methodology followed in producing results

For each section the following methodology was followed for and ,

Download the data from Quandl using the code.

If possible, extract the adjusted close price, otherwise, extract the close price and represent this as. This was done to mitigate the effect of stock splits which can create "discontinuities".

Compute the log price process as

Check for infinite values, replace these with NA (missing) values.

Omit any and all NA (missing) values from the log price process.

If we are testing the results on individual stocks then: Check if the number of historical days exceeds 10 years. If true, then take the past 10 years as a subset and discard the earlier data. This was done to avoid any small-cap data related issues, it should be noted that the conclusions drawn are the same whether or not you follow this step.

Estimate the value ofandfor the log price process.

Simulate a log price process usingand

Compute and store the-score ofand

Plot the densities of the computed-scores.

Check for normality and then draw conclusions.

Results obtained on 50 global Stock Market indices

The first, and most condemning set of results, was obtained on fifty stock market indices from around the world. The indices considered cover the Americas (South and North), Europe, Africa, and the Middle East, as well as the South Pacific and Asia. For 21 stock market indices (42%) of those tested had statistically significant -scores at the 95% confidence level and 11 stock market indices (22%) had statistically significant -scores at the 99% confidence level. The distribution of computed -scores is shown below,

In order of significance the least likely to be random stock market indices were: The SATRIX Financials Index (ETF), South Africa ( ), The Jakarta Composite Index, Indonesia ( ); The INMEX Index, Mexico ( ), The Colombo All Share Index, Sri Lanka ( ); The RTS Index, Russia ( ); The Nifty Fifty Index, India ( ); The Strait Times Index, Singapore ( ); The NYSE ARCA Mexico Index, Mexico ( ); The Bel 20 Index, Belgium ( ); The BSE 30 Sensitivity Index, India ( ); and The Dow Jones Industrial Average ( ). The full set of results is available as a CSV file.

For the distribution of -scores was almost identical and 9 of the 50 indices had statistically significant -scores. Three of the above indices fell out and, interestingly, the Rusell 1000 Index from the United States fell in with a -score of . The full set of returns for q = 4 is available as a CSV file as well.

For the statistically significant indices, this result means that we are 99% certain that they were not generated by a Brownian Motion model with drift and / or stochastic volatility. We can also say that they they almost surely don't evolve according to a random walk. This will be discussed in more detail in the conclusion.

Results obtained on the current constituents of the S&P 500

The next set of results is for the past ten years of prices for 484 of the 500 stocks on the S&P 500 currently. Some stocks were removed because data was not available on Yahoo! finance, and others were removed due to data-related issues. For 142 of the 484 stocks (30%) had statistically significant -scores at the 95% confidence level, and 88 of the 484 stocks (18%) had statistically significant -scores at the 99% confidence level!

The results here differ from the results on the stock market indices in one major way: the -scores for stock market indices are skewed to the left of the mean whereas the -scores for the constituents of the S&P 500 are skewed to the right of the mean. This observation is discussed in some detail in the conclusion.

I can hear you asking, "so which stock is the least random?". Below I have listed the top ten least random stocks according to our heteroskedastic-consistent variance ratio test at over the past ten years:

If you are wondering why so many REITs (Real Estate Investment Trusts) are in this list, so am I. I have listed a few hypotheses in the conclusion section but I welcome any of your thoughts as well.

Once again, the results for were similar to the results for ; 182 stocks out of the 484 (37.6%) had statistically significant -scores at the 95% level and 116 out of 484 (23.4%) stocks had statistically significant -scores at the 99% level. The results for q = 2 and the results for q = 4 are available as CSV files.

Remarks and Conclusions

In this article we have taken our first step down a Non-Random Walk Down Wall Street. We have understood and implemented the heteroskedasticity-consistent variance ratio test defined by Lo and MacKinlay in their seminal paper, Stock market prices do not follow random walks: Evidence from a simple specification test. The goal of this article was to make this powerful test more accessible to researchers and practitioners. So wherever possible I have focuses on the intuition behind the test rather than its derivation, because the intuition isn't hard to grasp.

The intuition ... in a nut shell:

Using our estimator it doesn't matter if we estimate variance using every observation, or every second observation because they all converge to the same estimate as the number of observations grows. Therefore the ratio of the estimate of variance using every second observation to the estimate of variance using every observation (for example) approaches zero as the number of observations grows. If we assume that prices are uncorrelated and the model of heteroskedasticity has a finite variance then the above statement is true for random walks with constant volatility or stochastic volatility. Given that the ratio of variances approaches zero, we can construct a statistical test around this which is sensitive to autocorrelation but insensitive to drift and stochastic volatility. And, lastly We can then use that statistical test to test the reasonableness of the random walk hypothesis and any failure on the test is a direct statement about the Markov property in the time series being tested.

In this article we implemented Lo and MacKinlay's test in R (all of the code is publicly available), tested that the test works as expected on model data, and then applied that test to two real world data sets: 50 global stock market indices, and 484 constituents of the S&P 500. For the global stock market indices all the available historical data was used and for the the individual stocks the past ten years worth of daily prices were used. Assuming these assets evolved according to a random walk with drift and stochastic volatility we would have expected to see:

Between 0 and 1 stock market indices passing for sampling intervals of 2 and 4 (1% on average).

Between 4 and 5 stocks passing for sampling intervals of 2 and 4 (1% on average).

Instead what we observed is that,

11 and 9 stock market indices passed for sampling intervals of 2 and 4 (18 to 22%)

88 and 116 stocks passed for sampling intervals of 2 and 4 (18 to 23%)

As such, we can quite confidently conclude that the stock market indices and stocks tested do not evolve according to a random walk with drift and stochastic volatility (this is particularly true for the assets which failed the test but could also be true for the others) and therefore: stock markets do not follow random walks.

Stock markets do not follow random walks.

This statement is as true today as it was 28 years ago in 1988 when Lo and MacKinlay concluded the same result on weekly returns data for a number of stocks and a broad based market index in the United States.

During our investigation two unanticipated, but interesting, results were obtained:

The distribution of-scores for market indices was skewed to the left whereas for distribution of-scores for individual stocks was skewed to the right, and A large number of the top-ten least random stocks on the S&P 500 over the past ten years according to our test have been listed REITs (Real Estate Investment Trusts).

For these two observations I offer up the following two, untested, hypotheses:

The autocorrelation of stock market indices is typically positive whereas for individual stocks autocorrelation is typically negative. This might explain the skewness of the-score distribution. The period tested contains the single largest real-estate market crash since the 1929 Great Depression, namely the 2008 Financial Crisis. As such, the apparent non randomness of these stocks in particular, may be a time-period artefact related to the financial crisis. Thoughts are welcome in the comment section below.

And that marks another nail in the coffin for the random walk hypothesis. That said, I really don't think this should come as a surprise. For decades academics and practitioners have been discovering anomalies in price and returns data which simply shouldn't exist under the random walk hypothesis. In a follow up post I hope to elaborate on some of these anomalies and question whether they are actually "anomalies" in an otherwise efficient system ... or rather the emergent statistical properties of a complex adaptive system (the adaptive market hypothesis).

Appendix A: Why R?

Why not R? I am a huge Python fan. I have been using Python for five years now. That said, I don't understand this "rivalry" between R and Python. The truth of the matter is that they are both great programming languages and, ultimately, what tool you use should be determined by your use case, not just your comfort zone. As such, some of my blog posts this year may be in Python, others may be in R, and yet some more still may be in C++, Julia, Scala, Rust, Go, or whatever I feel like. And that is because Computer Science and Quantitative Finance is not about programming languages or technologies, it is about the idea that you can apply scientific thought to the markets.