consider this toy example. i need to shift each column down by one * (its position in the array). so a kind of diagonal shift:

import numpy as np import pandas as pd df = pd.DataFrame(np.random.randint(1,10,(5,5)),columns=list("ABCDE")) for i,k in enumerate(df): df[k] = df[k].shift(i)

transforms:

A B C D E 0 6 1 6 3 1 1 2 7 5 9 7 2 6 6 6 9 8 3 7 8 8 2 8 4 5 2 9 9 2

into

A B C D E 0 6 NaN NaN NaN NaN 1 2 1 NaN NaN NaN 2 6 7 6 NaN NaN 3 7 6 5 3 NaN 4 5 8 6 9 1

which is what i want.

however for larger dataframes with hierarchical indexes, this looping method does not seem feasible. in fact, i've got an ipython notebook that has been running for almost an hour now with no end in sight.

this makes me think that there must be an easier, perhaps vectorized way... perhaps using some kind of "apply", however i'm not sure how to do that when each column needs to be shifted down as a function of its position in the array.