LONDON—A small Irish republican terror group apologized for killing a journalist during unrest in Northern Ireland last week, a death that underscores the region’s ongoing struggle with violent dissidents opposed to a decades-old peace accord.

The death of Lyra McKee, 29 years old, also casts a light on the reasons behind a nervousness in Belfast, Dublin and London over the risks to peace from Britain’s planned withdrawal from the European Union.

Irish...