BERT is deeply bidirectional, OpenAI GPT is unidirectional, and ELMo is shallowly bidirectional.