Translation Effect; Performance of Sentiment providers across different languages
In continuation with the series of blogs on Sentiment Analysis, the Connexun team recently evaluated the translation effect on the performance of sentiment providers across different languages.
The previous blog (Different Providers Different Sentiment Scores; not all sentiments are equal) compared the sentiment output from different AI providers for agreements or not, on how positive or negative the sentiment of a given text was.
Connexun’s team again used Eden AI for the Translation Effect analysis and comparison of results of sentiment analysis scores from four providers (Microsoft, IBM, Google, and of course Connexun) and for languages (Chinese, German, Hindi, Italian, Ukrainian).
We took 300 news stories from a week in English and then translated that into 5 different languages: Italian, German, Hindi, Ukrainian, and Chinese. To keep things even for translation, we used Google Translator API.
All languages being from different language families would allow us to check how well the models from different providers were in terms of consistency for these languages. Not all providers on Eden AI supported all these languages. German and Italian were the ones that were common among all. As a general rule, we expected the translation to not change the sentiment of the sentence barring minor differences in the score.
The plots below show the performance of different providers. The x-axis represents the sentiment for the original text and the y-axis represents the sentiment score of translated text in the corresponding language.
Results of Microsoft
The first set of plots below shows the performance of Microsoft. Results showed good consistency with over 88% of news preserving the sentiment after translation except for Hindi where this value dropped to 81%. Pearson correlation coefficient was also very high at 0.8 for German and Italian with slightly lower values of 0.75 for Chinese and Hindi.
Results of Google
For Google, the values saw a drop in numbers with only 65% of Italian text preserving their sentiment whereas for German this value dropped to 30%. The translated text also showed a heavy bias toward positive sentiment for the German language. However, the coefficient correlation stayed at 0.75 for both languages stating slightly high variability from the original score.
Results of IBM
For IBM, the correlation coefficient was very low around 0.6 with only 60% of news preserving their sentiment after translation.
Results of Connexun
Connexun showed the best performance with translation maintaining a general correlation coefficient of over 0.8 across all languages. Ukrainian was best performing with a correlation coefficient of 0.89 and Hindi scored lowest with a correlation coefficient of 0.8. The percentage of news retaining their sentiments after the translation was also consistently high across the languages with a value > 84% for all.
Summary tables of all correlation coefficients evaluated and percentage of the same sentiment classes after translation from English into available languages and providers highlight those differences observed when using models on the same translated text.
Through this work, we saw that machine translation of the text can have some impact on how the sentiments are evaluated by current models. Translator imperfection could be among one the factors affecting the results. But our use of the same translated sentence obtained using Google Translate is bound to mitigate that factor for all providers. The most surprising response was seen with Google’s sentiment mode with a systematic shift towards positive sentiment for German translations, whereas for Italian the same model showed satisfactory results. Connexun’s Sentiment model was observed to be more robust across languages maintaining the highest scores in terms of percentage of news retained with the same classes and sentiment score correlation.
About Connexun
Connexun is an innovative tech startup based in Milano, Italy.
Connexun crawls news content from tens of thousands of open web sources worldwide; turning unstructured web content into machine-readable news data APIs. Its AI-powered news engine is B.I.R.B.AL. empowers organizations to transform the world’s news into real-time business insight.
To learn more about Connexun, subscribe to our medium blog, follow us on Linkedin, Twitter, and Facebook, and visit our demos. For more information, do not hesitate to reach us at admin@connexun.com
