Google is to acquire Motorola Mobility, the US mobile company's smartphone business, in a $12.5bn (£7.6bn) cash deal.

The takeover will boost the rise of Google's Android software in the nascent smartphone market. The $40-a-share deal – which is Google's biggest acquisition to date – is a 63% premium on Motorola Mobility's closing price on the New York Stock Exchange on Friday.

Larry Page, Google chief executive, said: "Motorola Mobility's total commitment to Android has created a natural fit for our two companies. Together, we will create amazing user experiences that supercharge the entire Android ecosystem for the benefit of consumers, partners and developers. I look forward to welcoming Motorolans to our family of Googlers."

Sanjay Jha, chief executive of Motorola Mobility, added: "This transaction offers significant value for Motorola Mobility's stockholders and provides compelling new opportunities for our employees, customers, and partners around the world.

"We have shared a productive partnership with Google to advance the Android platform, and now through this combination we will be able to do even more to innovate and deliver outstanding mobility solutions across our mobile devices and home businesses."

The deal represents Google's biggest challenge yet to Apple, which has led the way in the smartphone and tablet markets with the iPhone and iPad.

Other manufacturers, including Samsung and HTC, will be free to release phones using Google's Android software. Google will run Motorola Mobility as a separate business.

The takeover also pits Google, which has traditionally avoided involvement in hardware, against the manufacturing giant Nokia. The move comes just six months after the Finnish phone maker signed a strategic deal with Microsoft in an effort to rebuild its ailing fortunes.

Motorola was one of the first mobile makers to partner with Google and release phones based on its Android operating system.

Motorola spun off Mobility as a separate business in January this year. The manufacturing division primarily produces smartphones, such as the Motorola Droid and the Defy, but also makes tablet computers and digital set-top boxes.

Analysts have long predicted that half of the world's smartphones will be using Android software by the end of 2012, as manufacturers have rushed to adopt Google's operating system rather than develop their own.

The deal is subject to US regulatory approval, which could prove a larger hurdle than usual given that Google's Android division is already being probed by anti-trust investigators. The companies said they expect the takeover to be completed in late 2011 or early 2012.

Andy Rubin, senior vice president of mobile at Google, said: "We expect that this combination will enable us to break new ground for the Android ecosystem.

"However, our vision for Android is unchanged and Google remains firmly committed to Android as an open platform and a vibrant open source community. We will continue to work with all of our valued Android partners to develop and distribute innovative Android-powered devices."

In a blogpost announcing the acquisition, Page said the deal would allow Google to "better protect" Android from "anti-competitive threats" over patents from Microsoft, Apple and other companies.

The highly competitive smartphone market has escalated into a bitterly fought patents war between the major manufacturers and software giants. Microsoft and Apple are suing Motorola and Google over a string of alleged patent infringements relating to Android.

Monday's deal means that Google now owns Motorola Mobility's swathe of patents – thought to total about 17,000 – and strengthen its position in the market. However, those patents failed to deter the ongoing lawsuits from Microsoft and Apple.

Page said: "In 2008, Motorola bet big on Android as the sole operating system across all of its smartphone devices. It was a smart bet and we're thrilled at the success they've achieved so far. We believe that their mobile business is on an upward trajectory and poised for explosive growth."

Motorola Mobility shares soared 60% to $39.20 – just below Google's offer price – in early trading on the New York Stock Exchange.

• This article was amended on 16 August 2011 because the original said Motorola was the first mobile maker to partner with Google.