1890s, of unknown origin, according to OED, first found in British newspaper police-court reports in the summer of 1898, almost certainly from the variant form of the Irish surname Houlihan, which figured as a characteristic comic Irish name in music hall songs and newspapers of the 1880s and '90s.

As an "inventor" and adapter to general purposes of the tools used by navvies and hodmen, "Hooligan" is an Irish character who occupies week by week the front of a comic literary journal called Nuggets , one of the series of papers published by Mr. James Henderson at Red Lion House. Previous to publication in London, "Hooligan" appears, I believe, in New York in a comic weekly, and in London he is set off against "Schneider," a German, whose contrainventions and adaptations appear in the Garland (a very similar paper to Nuggets ), which also comes from Mr. Henderson's office. "Hooligan" and "Schneider" have been running, I should think, for four or five years. [Notes and Queries, Oct. 15, 1898]

Internationalized 20c. in communist rhetoric as Russian khuligan, opprobrium for "scofflaws, political dissenters, etc."