It's not you, it's them.

You can interact with the map here. Try not to use your fists while interacting.