Markov Chains

Markov Processes, also called Markov Chains are described as a series of “states” which transition from one to another, and have a given probability for each transition. They are used as a statistical model to represent and predict real world events. Below is a representation of a Markov Chain with two states.

If the initial state is state E, there is a 0.3 probability that the current state will remain at E after one step. There is also an arrow from E to A (E -> A) and the probability that this transition will occur in one step. If we start at state A we have a 0.4 probability of transitioning to position 0.4 and a 0.6 probability of maintaining state A after one step.

Discrete-Time Markov Chains

The example above refers to a discrete-time Markov Chain, with a finite number of states. This means that there are only so many “events” or “states” which are represented or representable by the chain. The discreetness of time refers to the idea that time is not directly taken into consideration. This means there are no partial states nor transitions from one state to another, outside of the serial/linearly dependent states. There are several other versions of Markov Chains, yet the most useful (in the sense that it is the basis of all the others) is the standard discrete-time Markov Chain.

Tools of the Chain – Generating Markov Processes

Using a Markov Process is relatively straight forward, you require a time/step dependent problem with statistical probabilities between each step and then a given time/step interval. That’s it! For example,

A pizza delivery boy is attempting to deliver a pizza five blocks to a house. Every block he has a 50% chance of moving to the next block, he gets stuck in traffic with the probability of his current block (i.e. 1/10 for block 1, 2/10 for block 2, etc.) and the remaining probability that he losses his way and goes back a block (this is one stupid delivery boy). What is the probability that in ten steps** he will deliver the pizza? ** In this example a “step” is how long it takes to travel from one block (state) to the next.