The European banking system needs consolidation and "as time goes by, it will become more and more inevitable," the head of one of the largest banks in Europe told CNBC on Tuesday.

Often investors, policy-makers and other industry experts refer to fragmentation as one of the biggest hurdles to profitability in European banks. UBS chief Sergio Ermotti told CNBC in an exclusive interview at the UBS European Conference that the issue is "not sustainable."

"That's something that as time goes by will become more and more inevitable, is part of the solutions. For sure consolidation needs to happen, in particular in Europe, where we see a lot of fragmentation that it is not sustainable," Ermotti told CNBC's Joumanna Bercetche.

He further added that technology will make the sector more "effective and more efficient."

European banks have suffered from a trust deficit and have struggled to convince investors, since the global financial crisis. Since the start of the year, the European banking index is down about 20 percent.

Going forward it is hard to predict how investors will see European banks, Ermotti told CNBC.

"Difficult to predict sentiment and investors' approach. I think it is quite clear that investors are very reluctant to invest in our sector," he said. "I do respect the merits and a more balanced approach to valuations in our industry but we have to recognize this is a very challenging moment," he added.

Banks have had to deal with several crisis-legacy issues, such as bad loans, amid a low-interest rate environment and tougher regulations.