The Eiken tests have traditionally been seen as the standard for English proficiency certification in Japan, and a ticket to a well-paying job. Hundreds of thousands of people pass the tests every year, which makes you wonder where they are all hiding, as proficient English speakers can seem few and far between in this country.

The Eiken tests, which are backed by the ministry of education, are designed and administered by Eikyo, the Eiken Foundation of Japan (formerly the Society for Testing English Proficiency), a public-interest incorporated foundation established in 1963 and based in Tokyo. Many employers see the certificate as a valuable asset in a prospective employee’s portfolio, and cram schools offer courses dedicated to Eiken test preparation.

But the tests, which have been taken by more than 90 million people, are problematic in many ways, according to some researchers. In academic circles, the term “validity” is used to refer to the extent a test does what it is supposed to do. So how well do Eiken tests really measure proficiency? Eikyo appears to have no research data available on the validity of the Eiken, although it “has begun carrying out a research program designed to gather evidence which will be able to support the construction of a comprehensive and cohesive validity argument,” according to their site.

Foreign researchers have been complaining about a lack of data for decades. “Eikyo has been operating a successful testing business in Japan for more than 30 years,” Laura MacGregor of Sapporo International University wrote in 1997. “In all likelihood, this trend will continue. However, published reports of studies by Eikyo on item construction, reliability and validity are urgently needed to help consumers become better informed about the test, and to encourage research that would improve the quality of the test so that someday the Eiken tests might approach (a more acceptable) reliability.”

Although this report was published nearly 20 years ago, it could have been written yesterday.

‘English as scribblings on paper’

I spoke to an English teacher who has been involved in preparing Japanese children for the Eiken at a cram school. He asked to use a pseudonym — let’s call him John — due to worries about repercussions at work if he were identified. The training is good business for the school, he says, as it takes a huge amount of time to prepare students and the hourly fees are substantial.

“I began to have serious doubts about whether the Eiken is useful to English learners,” he explains. “Not only are there many head-scratchers among the test items, but I think there is something flawed about the whole idea of promoting English proficiency by the means of paper-based examinations. After all, the origin of any language is oral — that’s the natural mode of human communication. But with the Eiken’s heavy emphasis on grammar and reading comprehension, the language comes over to students as a formula to be solved on paper.”

On its website, Eikyo claims the tests contribute to a higher level of English proficiency in Japan. However, recent international comparisons do not seem to support the claim. Although it would be a stretch to suggest this is cause and effect, the number of takers of the Eiken test has been increasing over the past few years while Japan’s ranking in the EF English Proficiency Index has been falling, from 14 in 2011 to No. 30 last year, overtaken even by countries like Vietnam, which rose from a “poor” proficiency level at No. 39 in the list in 2011 to a “moderate” 29 last year.

What does this mean in reality? Any tourist will quickly learn that English as a lingua franca has yet to take root in this country. Attempts to correct the problem are stumbling. Data collected by the education ministry last June and July indicate it is falling way short of meeting its goals for English proficiency among Japanese students. For junior high school graduates — the first generation who studied English from elementary school — the goal is for 50 percent of junior high school students to have Eiken Grade 3-level competency by the time they finish school. Yet according to the survey, only 26.1 percent of final-year students nationwide were proficient to that level in reading, 20.2 percent in listening, 43.2 percent in writing and 32.6 percent in the relatively easy speaking component.

John believes the Eiken, with its heavy emphasis on the written language, is part of the problem.

“If by ‘proficiency’ you mean ‘communicative skills in situations ranging from everyday’s life to business,’ as the Eikyo defines it, it’s really hard to see why the tests are all about grammar and reading comprehension,” he says. “The listening part is far easier, and the speaking test is little more than an afterthought. Would you, for example, consider reading a text aloud to constitute communication? That is one part of the so called speaking test. So the way the test is designed makes students think of English as scribblings on paper, not a living language.”

Tough vocab, easy listening

So what are the test-takers themselves saying? Although it would be wrong to suggest there is a great clamor for change, there is no lack of critical voices online. One blogger, who took the top Grade 1 test, writes the following about the first, paper-based part and second speaking section of the Eiken:

“The total score necessary for passing part 1 is set low. At the same time, the level of difficulty of the test items is too high for the reading and vocabulary sections, and too low for listening comprehension. So even if you do horribly in reading or vocabulary, you can make up for it by scoring high in listening comprehension, and still pass.”

The first part, consisting of reading and listening, is paper-based, while the second is an interview-form speaking test held on a separate occasion. Passing part one is a prerequisite for taking part two.

The blogger is unhappy with the design of the tests. “The vocabulary test in part one is so hard that there is no way you could attain the necessary level by reading newspapers, for example,” the blogger writes. “To learn the vocabulary necessary for Grade 1, you basically end up cramming word lists from old tests. Wasn’t that what we were supposed to leave behind in the age of modern education?”

The blogger argues that you can’t even find the Eiken test vocabulary in newspapers. But are the words really that uncommon?

Word frequency in various text material can be checked by consulting the Corpus of Contemporary American English. I ran a couple of queries on items that came up in a Grade 1 test from 2015. Test-takers were asked to fill in the blanks with one of four multiple-choice items. “Frequency” here means the number of occurrences of a given word per million words in the corpus, in this case COCA, which holds 520 million words of spoken, fiction, newspaper and other text material.

Test item 1: “Jeffrey, tuck in your shirt. And brush your hair! You look so ___.”

1) disheveled 2) derivative 3) reticent 4) pervasive

Frequency: spoken: 0.61; fiction: 4.86; magazine: 1.06; newspaper: 0.82

Test item 2: “Although the advertisement for the cruise had promised ___ dinners, the guests were disappointed by the quality of the food aboard the ship.”

1) elliptical 2) abject 3) sumptuous 4) imperative

Frequency: spoken: 0.13; fiction: 1.90; magazine: 3.55; newspaper: 2.18

In other words, if you read a million words of the fiction included in COCA, your would likely encounter “disheveled” four or five times, and not even once if you read a million words of newspaper articles. You would find “sumptuous” about twice in the fiction texts, three times in magazines, and twice in newspapers. It seems like a daunting project to acquire this vocabulary by reading, given the little spare time the average Japanese student has. On the other hand, rote learning by cramming word lists — the only viable alternative — will most likely not help proficiency, as research indicates that rote learning results in poor retainment.

There does seem to be some truth to the test-taker’s claim that the vocab is quite uncommon, and perhaps irrelevant if the test is designed to test proficiency in “communicative skills in situations ranging from everyday life to business,” considering that the words are rarely used, particularly in spoken language.

“The listening part of the test, on the other hand,” continues the blogger, “is dead simple. But the English spoken seems unnatural, pronunciation is overly clear, and very different from what you hear on TV. The test seems to be about reading speed more than anything else, since listening comprehension is easy but you have very little time to select an answer.”

But perhaps all this concentration on the Eiken is a distraction from the real issue: Japan’s obsession with “teaching to test” rather than for fluency in communication. Some of the same criticisms of Eiken could also be leveled at the two other popular tests, TOEIC and TOEFL, but there are also crucial differences.

Jun Saito, a former assistant professor of political science at Yale University, now heads a Japanese prep school with the mission “to better the overall quality of the Japanese educational system from bottom-up, rather than merely teaching conversational English or preparing for entrance exams in Japan.” In a recent article in Japanese business magazine President, he stressed that TOEIC is all about the ability to engage in natural English communication at natural speed. The grammar, on the other hand, is comparable to what Japanese students learn at middle school. The balance in Eiken is arguably skewed the other way.

What’s the alternative?

While there does indeed still seem to be validity questions surrounding the Eiken, the difficulty of preparing students for the tests further reduce the value of the tests as an educational tool, says John.

“The problem is not only the design of the tests, but the teaching situation as well,” he says. To explain the fine points of the grammar and vocabulary tests in English is of course an impossibility. Our students’ command of English was simply too poor. Teachers fall back on Japanese, so we essentially spent 90 percent of the time explaining hairy English grammar problems in Japanese.

“It’s the brute-force method. What you do is essentially have the kids do loads of old tests and hope that something similar will come up when it is showtime. The point is not what is actually correct, but what is the ‘right’ answer — the one that the test designers expect you to choose. In some cases it is hard even for the instructor to know.”

So what’s the alternative to this way of teaching? President magazine also re-published an article by Shin Sasaki, contributing editor with the English Hacker learning portal site. Sasaki raised his TOEIC score from 400 to a perfect 990 by completely changing his style of study, ditching the traditional Japanese juken (exam preparation) style, in which grammar is king, in favor of focussing on speed-reading texts for gist (rather than word-by-word understanding) and honing his listening comprehension at natural speed. In other words, he worked on skills you need in everyday life.

Sasaki mentions how he talked to friends who studied English and found that many had the same, positive experience from “escape from juken-style English study.” Juken study, he points out, is very high-level but demands constant translation from English to Japanese, never allowing students to forget about their mother tongue and start thinking in the foreign language.

Although immersion abroad is still seen as the the holy grail, there are many alternative paths to English proficiency, even in Japan. You can study English on your smartphone with Duolingo, take cheap conversation classes over Skype or make English part of your life by watching English-language movies with English subtitles. When you have improved a bit, you can join an English-speaking MeetUp group and pursue your hobbies there. But none of these methods will lead to a certificate, and that is the core of the problem. Today Japanese students and job-seekers must choose between committing themselves to getting a ticket to a job or something of practical value for the future — between English proficiency on paper and true communicative skills.

Will this ever change? Well, attitudes among the young generation — the test-takers — may be changing already. “What people want today,” writes one Japanese blogger, “is not mastery of advanced English, but of English that you can actually use. If tests don’t change to emphasise practical usage and proficiency more, I think they will become obsolete.”

John agrees. “I think there is something tremendously static about the Eiken tests,” he says. “The second-stage speaking test illustrates this really well — everything about it is reality in suspended motion: The student knocks on the door, is asked to enter, and greets the test official; he steps up to his desk and hands over his test card from the exact right distance; he is then asked to read a short passage, describe what is happening in a couple of pictures and reply to a few questions, with predetermined follow-ups. All this has been drilled over and over at the prep school. The student will recognize much of what is happening like deja vu.

“But now we are rapidly moving towards a world with shrinking distances, virtual-reality commuting, automated factories and workplaces, and jobs that don’t even have names yet. Who will ask for a person who can only function in a world where everything is predictable, including what is said, and the responses to what is said? How much is a paper that certifies such a skill worth then? If I was in charge of designing the Eiken, I would begin to consider taking the red pill and looking that future in the eye.”

Your comments and story ideas: community@japantimes.co.jp