The Federal Reserve could begin chipping away at its $4.5 trillion balance sheet later this year, New York Fed President William Dudley said Friday.

As the central bank continues down a gradual path of interest rate hikes, officials there also will have to contend with what to do with the bonds the Fed has amassed over the years of monetary policy aimed at juicing the economy.

Under current policy, when the bonds mature the Fed rolls them over and reinvests the profits. However, those days seem to be coming to an end.

"It wouldn't surprise me if sometime later this year or sometime in 2018, should the economy perform in line with our expectations, that we will gradually start to let securities mature rather than reinvest them," Dudley said in an interview with Bloomberg.

Investors care about the Fed's balance sheet because letting those bonds back into the market likely would boost supply and, with it, interest rates.