Bangla is missing from the AI revolution

The typewriter could not hold our alphabet. The computer split it into rival fonts. Now artificial intelligence is learning the world’s seventh most spoken language last, and mostly from strangers.

Bangla AI
Illustration: TBS

“Language is the dress of thought,” wrote Samuel Johnson. If so, every new writing machine is a tailor, and for a century and a half, the tailors have been cutting for English.

The first commercially successful typewriter, sold by Remington in 1874, had a keyboard laid out for 26 letters. Bangla has roughly 50, plus a dozen vowel signs can be attached before, after, above and below their consonants, plus several hundred conjunct letters in which two or three consonants fuse into a single shape.

A typewriter offered only about 90 positions; therefore, something had to give, and what gave was the script. Early Bangla typewriters broke conjuncts apart, shrank letters, and asked typists to strike keys in an order the eye never reads. In the Pakistan years, the convenience of the machine was even offered as a reason to write Bangla in another alphabet altogether.

The man who eventually taught the typewriter to speak Bangla properly was a playwright. Munier Choudhury had been jailed in 1952 for protesting after the Language Movement killings. In 1965, working with a typewriter manufacturer in East Germany, he redesigned the Bangla keyboard so that the script’s logic, not the machine’s, came first. The Munir Optima became the most-used typewriter in Bangladesh until personal computers replaced it. Choudhury did not live to see that. On 14 December 1971, two days before our victory, he was taken from his home in Mohammadpur by Al-Badr and killed. His body was never identified.

The computer repeated the whole pattern. The Bijoy layout, released in 1988, made Bangla typing possible on a PC, but it stored text in a private encoding that only displayed correctly if the reader owned the same font.

A document that looked perfect on one screen was gibberish on another. By the early 2000s, at least five keyboard systems were in use at once, and the Bangladesh Computer Council had to step in with a national standard. The free, Unicode-based Avro keyboard arrived in 2003. The slogan that carried the day was “Let language be free.” Mehdi Hasan Khan built Avro, and Omicron Lab released it at a time when every rival keyboard either cost money or locked users into a proprietary font.

Its real innovation was phonetic typing: a user could write “amar” and watch it become “আমার”, spelling Bangla the way it sounded rather than hunting for a script-specific key layout. That single decision did more to spread Bangla typing than any government standard.

That is the history in one sentence: with every new generation of technology, Bangla arrives late, arrives fragmented, and depends on a handful of volunteers to catch up. We are living through the third round now, and the stakes are higher than fonts.

Anyone who has tried to use artificial intelligence in Bangla already knows this. One turns on automatic captions for a Bangla lecture on YouTube and watches it produce sentences no Bangladeshi has ever spoken.

One asks a meeting tool for live translation and hears the tenses collapse. A recent study from Bangladeshi researchers at the Department of Computer Science and Engineering, AUST, fine-tuned OpenAI’s Whisper, one of the strongest speech recognisers available, on every hour of labelled Bangla speech they could find.

The best they achieved was a word error rate near 29 per cent. Roughly one word in three or four was wrong, after months of specialist work, on clean recordings. For English, error rates in the low single digits are routine. Translations were not satisfactory. When feeding a paragraph of ordinary Bangla prose to any of the leading systems, the meaning survives, but the register does not: the respectful “apni” and the familiar “tui” flatten into the same English “you”, idioms turn literal, and a sentence that was warm comes back merely correct.

The reason is not that Bangla is harder. It is that the machine has barely heard it. Mozilla’s Common Voice project, the largest open speech collection in the world, holds about 54 hours of verified Bangla.

The Bangla text these systems read is still thinner. The models themselves are built to see English: when researchers behind TituLLMs, a family of Bangla language models, examined how a standard tokenizer chopped Bangla text, it needed nearly eight tokens per word, where English typically needs little more than one.

An Oxford study of 17 tokenisers found that some languages require more than 15 times as many tokens as English to say the same thing. In practice, that means a Bangla speaker pays several times more for the same AI service, waits longer for an answer, and gets a system with a shorter memory. The machine is not built to be neutral. It is tailored for the West, since the Global South, in most of modern history, has not been in the forefront of building such capabilities.

Consider the absurdity. Ethnologue ranks Bangla seventh in the world with about 265 million speakers, more than German, French, Japanese or Korean. Yet in the vocabulary of AI research, it is a “low-resource language”.

Most of our literature is not online in machine-readable form. Our newspaper archives are scanned images, not searchable text. Our dialects, from Sylhet to Chittagong to Noakhali, are almost entirely unrecorded.

This matters far beyond just convenience. Courts record testimony in Bangla. Doctors take histories in Bangla. Government offices, schools, call centres and radio stations are run in Bangla. Every one of these will be reshaped by AI over the next decade, and if the tools work in English and stumble in Bangla, the benefits will flow to the English-speaking few and the errors will land on everyone else.

There is also a quieter question of ownership. Right now, the Bangla that AI systems know was scraped by foreign companies from whatever they could find. The shape of our language inside these machines is being decided by people who do not speak it.

The good news is that Bangladeshis have started, as they always do, without waiting. Researchers at Buet released BanglaBERT in 2022, the first serious Bangla language model with a proper benchmark. Volunteers at Bengali.AI have run open competitions that drew thousands of participants to build Bangla speech recognition. A local company assembled 37 billion Bangla text tokens to train TituLLMs. There is now an annual academic workshop devoted to Bangla language processing. The government, to its credit, saw the need early: the Bangladesh Computer Council’s Tk159 crore project to build 16 Bangla language tools, from speech-to-text to machine translation, was launched in 2016. It has moved slowly, with only one tool complete by 2022, but the ambition was correct and it deserves to be finished rather than forgotten.

What would finishing look like?

First, a national open Bangla corpus: thousands of hours of speech, recorded with consent across regions, ages and dialects, and hundreds of millions of words of text, released under an open licence so that anyone, at home or abroad, can build on it.

India’s Bhashini programme shows that a state can do this at scale within a few years. The cost is modest beside what the country already spends on digital infrastructure, and unlike a bridge or a data centre, a corpus grows more valuable every time someone uses it.

Second, digitise what we already own. Bangla Academy, the National Library, the newspaper groups and the universities hold a century of text that could be converted into machine-readable form. Out-of-copyright literature, from Bankim to Tagore, belongs to the public and should be available to every model, not only the ones with lawyers.

Third, make the standards boring and settled, which is the highest compliment one can pay a standard. Unicode may have won the encoding war, but Bangla still suffers from inconsistent spellings, rival conjunct forms and half a dozen ways to write the same word. A national style and normalisation standard, maintained by the Computer Council with the Academy, would save every researcher months.

Fourth, put Bangla into procurement. When the state buys AI services for courts, hospitals or call centres, the contract should require Bangla performance to be measured and published, not assumed. Vendors respond to what large customers demand, and few customers are larger than a government of 170 million people.

Munier Choudhury’s achievement was to make a machine built for another language bend to ours. Sixty years later, the machines have changed, but the choice has not. We can let Bangla be fitted, late and badly, into systems designed elsewhere, or we can decide that this time the language comes first and the data is ours to give.

Samuel Johnson was right that language is the dress of thought. Two hundred and sixty-five million people should not have to think in borrowed clothes.


This article is co-authored by S M Mashrur Arafin Ayon, a Research Associate at SIPG, North South University, and Mursalin Haque, an undergraduate student at North South University.