SAINT JOHN—Four years ago, Kenneth Irving, the scion of one of Canada’s richest and most secretive business dynasties, began a speech in a Boston hotel with an admission. “For the rest of my life I am always going to get asked, whose boy are you?” Mr. Irving told his conference audience.

“I’m Arthur’s boy,” he said, words that seemed to signal fealty to a tightknit clan that for years had escaped many of the woes that commonly plague wealthy business-owning families as they transfer the reins from generation to generation.

But within a few months, Mr. Irving had left Irving Oil Ltd. amid an acrimonious behind-closed-doors split from his father, Arthur Irving Sr. His exit exacerbated a schism within one of the more powerful, but least known, family empires in North America.

Based and dominant in the province of New Brunswick, the Irvings’ influence stretches into the U.S., where they are Maine’s largest landowner and a supplier of almost a fifth of U.S. oil product imports. Their closely held interests in oil, lumber, and shipbuilding, among others, rank them as Canada’s second or third wealthiest family and make Arthur and his elder brother James Irving—the two surviving sons of founder Kenneth Colin (K.C.) Irving—each worth over $6 billion, according to Forbes.

Once so powerful in Atlantic Canada that a Canadian Senate report called their position “unique” in the western world, the family and its half century of dominance face new challenges, as the clan struggles to get along.