The woman stood in her roadside stall in a quiet neighbourhood in the Indonesian city of Yogyakarta, chopping tomatoes, beans and spinach, plus one red chilli. Mixing everything in a peanut sauce, she handed the salad, called lotek, to customers who puttered up on motorbikes and waited on blue plastic stools. She was curious about me, full of questions, and the feeling was mutual. It was to chat with people like her that I had moved to Indonesia and enrolled in intensive language study. Yet after hundreds of hours of classes, I couldn’t understand what she was saying.

Everything she said sounded to me like it had half a syllable. I did make out familiar words, but painfully rarely. I wondered what her life was like in this city, how she felt about escalating political and cultural tension in this young democracy and the world’s most populous Muslim-majority nation. But I wasn’t to find out.

She handed me my meal wrapped in newspaper, the text of which I could understand. ‘Bahasa Indonesia baku’, I thought to myself – textbook Bahasa Indonesia. My teachers had referred to the language as ‘baku’, or ‘standard’, in class, emphasising that it was this version of Indonesian, the nation’s official language, we were learning. The addendum hadn’t struck me as overly important, but it should have.

You may also be interested in:

• The Asian language designed to unify

• The pidgin language uniting 83 islands

• The island that forever changed science

Bahasa Indonesia’s antecedent, Malay, evolved and spread during the last millennium because of the need in maritime South-East Asia – where hundreds of languages are still spoken across the thousands of islands that now comprise the modern nations of Indonesia, Malaysia and Singapore – for a lingua franca for trade and other exchanges. Malay was seen to be grammatically simple, non-hierarchical and easier to learn than other regional languages. It was the mother tongue of few, but as people travelled around the region, it became their accepted means to communicate.

Then, in the early 20th Century, Indonesian nationalists, plotting independence from Dutch colonial rule, agreed that a reformed version of Malay, with an expanded vocabulary and a new name – Bahasa Indonesia – should become the official language of the soon-to-be independent nation. Malay, according to Cornell University Indonesian scholar Benedict Anderson, was “simple and flexible enough to be rapidly developed into a modern political language”.

The goal for Bahasa Indonesia was to break down communication barriers and facilitate inclusion of more than 300 ethnic groups in the new nation, whose independence was officially recognised in 1949. Because no major ethnic group, including the Javanese (whose highly complex language was at the time spoken by about 40% of the population), would have its mother tongue as the official language, inequality would not be created or reinforced. Bahasa Indonesia would help draw unity out of diversity.

But in reality, things aren’t so simple. Today, standard Bahasa Indonesia, which hasn’t evolved too drastically from Malay, is rarely spoken in casual conversations. People think it’s too ‘kaku’, meaning rigid and stiff, my language teacher Andini told me after I admitted my difficulties at the roadside stall. Moreover, people sometimes find Bahasa Indonesia inadequate to express what they want. Andini confessed she often shares this frustration, wanting to use words and expressions from a sub-dialect of East Javanese spoken in her hometown.

People sometimes find Bahasa Indonesia inadequate to express what they want

Part of the problem lies in the language itself: Bahasa Indonesia has fewer words than most languages. Endy Bayuni of The Jakarta Post has written that foreign translations of Indonesian novels tend to read better, while Indonesian translations of foreign novels sound ‘verbose and repetitive’. But there’s also a political dimension. Because Indonesians learn Bahasa Indonesia in school, then hear it as adults primarily in political speech, they associate it with homogeneity, according to Dr Nancy J Smith-Hefner, associate professor of anthropology at Boston University. This is exacerbated because Bahasa Indonesia was heavily promoted during the Suharto dictatorship that ruled from the mid-1960s until 1998 and stifled many forms of individual and cultural expression. Because of this, those who speak it risk looking “theatrical, bookish or pompous”, explained Nelly Martin-Anatias of the Institute of Culture, Discourse and Communication at the Auckland University of Technology.

It turns out that a means to linguistically unite the Indonesian nation has instead, due to the language’s simplicity and rigidity, created a new barrier that prevents communication on a deeper level – one that Indonesians circumvent by employing their own particularised speech, tailored to their specific regions, generations or social classes.

People dissatisfied with Bahasa Indonesia have plenty of options. There are hundreds of regional languages and dialects, sometimes spoken intact, sometimes blended with Bahasa Indonesia. In Yogyakarta, where I am – located in the centre of Java and the traditional heartland of Javanese culture – Javanese is commonly spoken, partly as a reflection of cultural pride. A food vendor who pushes his wooden cart along my street every morning selling soto ayam (spicy chicken soup) often breaks into Javanese, making our conversations difficult for me to follow. He recently asked me something three times before I understood. The question, when I got it, revealed a pride in his heritage: had I yet seen wayang kulit (shadow puppet play), the quintessentially Javanese cultural performance?

Meanwhile, Indonesia’s youth continue to form their own, cooler language variants, gleefully challenging older ears, with the internet becoming colloquial Bahasa Indonesia’s new frontier. The country has close to the freest speech in Asia, and young Indonesians are fanatical fans of Twitter, Facebook, WhatsApp and Instagram, using the platforms to evolve their own language with new and borrowed words. As Andini and I scrolled through Indonesian Twitter feeds during class one day, road-bumps of slang brought me to abrupt and frequent halts.

By supplementing various informal and regional speeches, Martin-Anatias told me, young Indonesians “establish intimacy and identity” when conversing, so that they can more accurately convey emotions, express needs and tell jokes.

Yet standard Indonesian – Bahasa Indonesia baku – remains the best way I have to communicate here, and for me, the language serves its original purpose. As I operate in standard Bahasa Indonesia, I’m pleased to find plenty of people happy to meet me there. When someone speaks to me in a way I easily understand, I read significance into it, knowing they are likely tailoring it for me, adapting themselves, breaking things down as a conscious act of inclusion.

By supplementing various regional speeches, Indonesians can more accurately convey emotions, express needs and tell jokes

This happened when I took a motorbike taxi home from class. I understood my young driver near-perfectly. His questions were simply phrased: “In your country what season is it now?”; “In your country are there transport apps?”. My own questions he answered in a way designed to ensure clarity. I awkwardly said some just-memorised slang, and he offered a thumbs-up.

Knowing when to scale up speech styles and when to scale them back, and how to successfully balance differing impulses to unity and diversity – that is Bahasa Indonesia’s and this country’s challenge.

Lost in Translation is a BBC Travel series exploring encounters with languages and how they are reflected in a place, people and culture.

Join more than three million BBC Travel fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Travel, Capital, Culture, Earth and Future, delivered to your inbox every Friday.