Translation Effect; Performance of Sentiment providers across different languages

In continuation with the series of blogs on Sentiment Analysis, the Connexun team recently evaluated the translation effect on the performance of sentiment providers across different languages.

The previous blog (Different Providers Different Sentiment Scores; not all sentiments are equal) compared the sentiment output from different AI providers for agreements or not, on how positive or negative the sentiment of a given text was.

Connexun’s team again used Eden AI for the Translation Effect analysis and comparison of results of sentiment analysis scores from four providers (Microsoft, IBM, Google, and of course Connexun) and for languages (Chinese, German, Hindi, Italian, Ukrainian).

We took 300 news stories from a week in English and then translated that into 5 different languages: Italian, German, Hindi, Ukrainian, and Chinese. To keep things even for translation, we used Google Translator API.

All languages being from different language families would allow us to check how well the models from different providers were in terms of consistency for these languages. Not all providers on Eden AI supported all these languages. German and Italian were the ones that were common among all. As a general rule, we expected the translation to not change the sentiment of the sentence barring minor differences in the score.

The plots below show the performance of different providers. The x-axis represents the sentiment for the original text and the y-axis represents the sentiment score of translated text in the corresponding language.

Results of Microsoft

The first set of plots below shows the performance of Microsoft. Results showed good consistency with over 88% of news preserving the sentiment after translation except for Hindi where this value dropped to 81%. Pearson correlation coefficient was also very high at 0.8 for German and Italian with slightly lower values of 0.75 for Chinese and Hindi.

Microsoft correlation plots

Results of Google

For Google, the values saw a drop in numbers with only 65% of Italian text preserving their sentiment whereas for German this value dropped to 30%. The translated text also showed a heavy bias toward positive sentiment for the German language. However, the coefficient correlation stayed at 0.75 for both languages stating slightly high variability from the original score.

Google correlation plots

Results of IBM

For IBM, the correlation coefficient was very low around 0.6 with only 60% of news preserving their sentiment after translation.

IBM correlation plots

Results of Connexun

Connexun showed the best performance with translation maintaining a general correlation coefficient of over 0.8 across all languages. Ukrainian was best performing with a correlation coefficient of 0.89 and Hindi scored lowest with a correlation coefficient of 0.8. The percentage of news retaining their sentiments after the translation was also consistently high across the languages with a value > 84% for all.

Connexun correlation plots

Summary tables of all correlation coefficients evaluated and percentage of the same sentiment classes after translation from English into available languages and providers highlight those differences observed when using models on the same translated text.

Through this work, we saw that machine translation of the text can have some impact on how the sentiments are evaluated by current models. Translator imperfection could be among one the factors affecting the results. But our use of the same translated sentence obtained using Google Translate is bound to mitigate that factor for all providers. The most surprising response was seen with Google’s sentiment mode with a systematic shift towards positive sentiment for German translations, whereas for Italian the same model showed satisfactory results. Connexun’s Sentiment model was observed to be more robust across languages maintaining the highest scores in terms of percentage of news retained with the same classes and sentiment score correlation.

About Connexun

Connexun is an innovative tech startup based in Milano, Italy.

Connexun crawls news content from tens of thousands of open web sources worldwide; turning unstructured web content into machine-readable news data APIs. Its AI-powered news engine is B.I.R.B.AL. empowers organizations to transform the world’s news into real-time business insight.

To learn more about Connexun, subscribe to our medium blog, follow us on Linkedin, Twitter, and Facebook, and visit our demos. For more information, do not hesitate to reach us at admin@connexun.com

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Connexun | news api

Connexun is the ultimate AI news engine — turning unstructured news content into multi-purpose actionable data.