From our Obsession
Machines with Brains
AI is upending companies, industries, and humanity.
The algorithm able to understand English best is one designed to learn Chinese.
As noted by Karen Hao of the MIT Technology Review, a computer model built by Chinese search engine and commerce firm Baidu now leads all other models on a benchmark of general English-sentence comprehension known as GLUE.
The model is named ERNIE, and its core advantage is that it was trained to guess phrases, a key to understanding Chinese.
Classic models for natural language understanding, such as GPT-2, are trained to guess a next word based on what preceded it. So, for example, predicting the missing word here:
Harry Potter is a series of fantasy novels written by J. K. ███
Models such as Google’s BERT are trained instead to guess words—known in the business as “tokens”—based on what came before and afterward, taking advantage of the additional context:
███ Potter is a series ███ fantasy novels ███ by J. ███ Rowling.
The Baidu researchers trained ERNIE to predict sets of missing tokens. This is essential for understanding Chinese, in which individual characters rarely work alone, as Hao explains:
While certain characters do have inherent meaning, like fire (火, huŏ), water (水, shuĭ), or wood (木, mù), most do not until they are strung together with others. The character 灵 (líng), for example, can either mean clever (机灵, jīlíng) or soul (灵魂, línghún), depending on its match. And the characters in a proper noun like Boston (波士顿, bōshìdùn) or the US (美国, měiguó) do not mean the same thing once split apart.
So training the ERNIE becomes a guessing game of phrases. This skill, it turns out, transfers nicely to English, enabling it to do well at predicting entire sets of words, such as those missing here:
Harry Potter is ███ ███ ███ fantasy novels ███ by ███ ███ ███.
And that, for that moment, helps make ERNIE English’s next top model.