A Victorian judge has criticised the tax office for making the victims of a "greedy" accountant pay almost $3 million in debts after the agency let him continue working despite being notified of his fraud.

Richard Hogg, 56, was jailed for at least three years by the County Court of Victoria on Thursday for fraud and tax evasion totalling $4.1 million, including $2.8 million in unpaid taxes that his clients must pay.

"How the taxation department can claim (clients) should pay ... when (the tax office) were already on notice to the real risk that you were putting on clients, is a loss to me," Judge Duncan Allen told Hogg.

"An extremely unfortunate state of affairs, to put it mildly."