JUST as native English-speakers stumble with Japanese, the Japanese struggle mightily with English, not to mention Korean or Chinese. In the widely used TOEFL tests of English as a foreign language, Japan invariably ranks second from bottom among the 29 countries participating in the scheme. Compared with the 150m people around the world who speak English as a second language, there are only 9m non-native speakers of Japanese—and most of those were forced to learn the language during Japan's era of colonial occupation, and are now dying of old age.

For those who put their faith in technology, therefore, it was encouraging to hear Shinzo Abe, Japan's prime minister, demonstrate his linguistic skills a few weeks ago with a palm-sized gizmo that provided instantaneous translations of spoken Japanese into near-flawless English and Chinese.

Mr Abe can manage perfectly well without such a device, being one of the few Japanese prime ministers in recent years to speak English fluently. And the question he asked the gadget in Japanese (“Is there a department store nearby?”) was hardly the nuanced dialogue of politics and diplomacy. Even so, the fact that a pocket-sized device could interpret tourist-type phrases accurately and on the fly, from one language to several others, says much about the improvements that have been made lately in machine translation.

This device, developed by the Advanced Telecommunications Research Institute International near Kyoto, is made so compact and effective by wireless connectivity. A database containing upwards of 1m examples of Japanese-English translations and 500,000 Japanese-Chinese ones is stored on a remote computer, which is interrogated wirelessly by the hand-held device.

EPA

One day this may fit in your pocket

Machine translation has been an elusive goal since the earliest days of computer science. The Pentagon poured millions of dollars into efforts to get computers to translate Russian sentences into English. But disillusionment set in the 1960s when it became clear that producing results indistinguishable from a human translator wasn't going to happen soon, if ever. The major obstacles were not computational but linguistic. The missing ingredient was a fuller understanding of language itself.

That is still true. But computational linguists are nowadays making greater strides by being less ambitious. They've come to realise that there are many tasks that need only rough translations—such as initial drafts for professional translators, or for experts who know the topic well. Success is also being achieved in fields where the subject domain is itself limited—weather reports, for example, or patent law, or medical diagnosis.

The main drivers for this more pragmatic approach to machine translation have been the enlargement of the European Union and the spread of the internet. Both have generated a pressing need for cheap and cheerful translations between numerous languages. In turn, this has spawned a wealth of new translation approaches.

The Pentagon poured millions of dollars into efforts to get computers to translate Russian sentences into English. But disillusionment set in ... The missing ingredient was a fuller understanding of language itself

The earliest method, “direct translation”, is applied strictly to a single pair of languages, usually in just one direction (say, from Japanese to English only). In this case, the translation is done direct from the source language to the target language using as little analysis of syntax and semantics as necessary. All that is needed is a colossal bilingual dictionary and a program for analysing the source text and generating the target text.

This kind of approach still crops up in digital phrase books—though, to be fair to Mr Abe's gizmo, it had many more smarts in it than just a huge online dictionary. By and large, however, such systems have been eclipsed by “rule-based” methods, which translate one language into another in either a two- or three-step process.

Here, the source text is first translated into a set of rules about syntax and semantic meaning that is common to the target language, or even a number of target languages. Then, the text that has been translated into an intermediate language, or “interlingua”, is translated into the target language. This way, the same analysis software and interlingua can be used for translating one language into many others.

Now even rule-based translators are giving way to “statistical translation” methods borrowed from information theory. The idea is that a phrase in one language has a certain probability of being a particular phrase in a second language.

This probability-matching has become possible thanks to the enormous collections of texts, or “corpora”, which have been built up on computer databases for linguistics research, and tagged with their parts of speech (verbs, nouns, adjectives, etc). Such resources have been pounced upon already by engineers hoping to build speech-recognition and synthetic-voice systems. Now machine-translation researchers are taking advantage of them, too.

With so much of today's discourse taking place online, machine-readable corpora in dozens of languages are being accumulated at a phenomenal rate. Matching an original phrase in one language with its most-probable translations in several other languages is becoming almost a routine task. Given time, that elusive goal of speaking down the phone in your own language, and having the person at the other end unaware that you not speaking in his or her tongue, actually looks achievable. Even the bilingual Mr Abe should be pleased.