OTTAWA—The Canadian economy contracted slightly in January, falling short of expectations as tougher mortgage-lending rules contributed to the biggest decline in real-estate agents and brokers’ output in more than nine years.

Lower output of nonconventional oil extraction also contributed to the overall decline in January.

Canada’s gross domestic...