I think I got it why Google colab allocates a whopping 66GB to your program.

Since X gets allocated the batch size of elements

X = [pair[0] for pair in pairs[idx:idx+batch_size]]

when converting to one_hot_encoding

n_values = np.max(indexes) + 1 return np.eye(n_values)[indexes]

X gets assigned a matrix of dimension (64,64) and since the indexes also are from (0:63). It essentially returns (64,64) matrix.

Warning:- This is only for x consider y too.

Now repeat this process for like N times. Each time X & Y are (64,64) matrices and also there is pairs variable which also is a big list and so memory keeps on increasing.

Hint:- Y is list of strings and np.max(Y) cannot be done.