A proposed three-way merger in Malaysia will create the world's first Islamic bank that will have enough clout to challenge the dominance of conventional, often Western, banks in the industry and influence the way Islamic finance deals are made.

The Islamic units of market leaders such as HSBC Holdings and Standard Chartered have long used their parents' vast networks to win leading roles in global deals. Most of such deals involve designing and distributing Islamic bonds, or sukuk, a fast-growing segment in the sector.

By contrast, local banks in the Islamic world lack the scale and reach to entice corporate and sovereign issuers from Europe and Asia. Many of them are owned by wealthy families or the state, and fear of losing control has limited mergers and consolidation in a still highly fragmented industry.

Read MoreRussian President Putin blames Malaysian plane tragedy onUkraine

In comes Malaysia, whose government is keen to make leaders out of the country's banks in global Islamic finance. Last week, CIMB Group Holdings, RHB Capital and Malaysia Building Society secured regulatory approval to begin their merger talks.

Their amalgamation would yield a stand-alone Islamic bank that is separate from the conventional banking operation of the enlarged bank. This Islamic bank will have a cross-border presence sizable enough to win big mandates, as well as a Malaysian model of Islamic finance that is regarded by many investors to be simpler and less confusing, analysts say.

"The next meaningful stage for Islamic banking is a bank with connectivity across key markets and separate markets," said Abdul Rauf Rashid, country managing partner for EY Malaysia. "Putting together and creating this large entity is a start, but the next step is integrating them into one powerful unit that can go out and build the market."

Read More Malaysia banks in mega-merger talks

The merger would include the creation of a so-called mega Islamic bank. Malaysia's central bank defines such an entity as having a minimum $1 billion in paid-up capital.

A successful merger may pave the way for other Malaysian banks to follow suit as they expand globally. That would help internationalize the Malaysian model of Islamic finance, analysts say.

Malaysia also has the capacity to standardize industry practices, which vary due to different interpretations of Islamic law, they say.

Malaysia has a centralized, top-down approach to creating sharia-compliant products. Other countries have allowed sharia boards of individual Islamic banks to decide whether their products and activities obey religious principles. The latter approach sometimes leads to disagreement over the design of a product and even confusion among investors, analysts say.

Read MoreMalaysia rate hike: Will they or won't they?



Some investors have decided to stick with the big conventional banks such as Standard Chartered. The banks have their own Islamic scholars and the financial clout to lead in the structuring of Islamic finance deals.

"When we have mega banks, and trim down to fewer players, it will be easier to harmonize things. Bearing in mind that it will go global, we are positioning Malaysia as the leader for Islamic banking," said Johan Lee, partner at Kuala Lumpur-based law firm J.Lee & Associates.

No longer a concept

The mega Islamic bank concept has been discussed in the industry for years. Previous efforts have failed to take off partly because of scant interest from the private sector.

Read MoreMalaysia's central bank raises rates to help debt

Most banks in the Middle East are retail-oriented or lack the financial expertise to compete with large multinationals. Al Rajhi Bank - Saudi Arabia's largest listed lender and the world's biggest Islamic bank - has never issued a sukuk and is only now developing its investment banking business.

Smaller Islamic banks in the region such as Qatar Islamic Bank and Al Baraka Banking Group are hardly in the league of Western banks when it comes to Islamic finance.