Man against the machine
Tony Parr looks at how new machine translation tool DeepL performs compared with two more established translation machines – and with real humans.
Last year, Marcel Lemmens and I conducted a ‘mystery shopper’ experiment in which we assessed the quality of translations we sourced on the open market in both the Netherlands and the UK (ITI Bulletin, September-October 2017). Of the eight translations, half proved to be totally unfit for purpose. In short, any unsuspecting individual or firm looking for a translation agency on the internet would appear to run a good chance of being sold damaged goods.
In the meantime, all the talk is now about a new kid on the block called DeepL. As you may have read in Jost Zetsche’s appraisal in the November-December edition of ITI Bulletin, it looks pretty impressive. A number of colleagues have also tested the new machine and blogged about the results. One of the recurring findings is that DeepL is deceptively good: although its solutions are sometimes way off the mark, they are nevertheless well phrased, and hence convincing.
With some commentators already claiming that human translators will be superseded by machines in the future, perhaps within as little as 10 or 20 years, I was curious to see how DeepL – and the two other big-name machines, Google and Bing – compared with our human translators. Would our mystery
shopper have been better off simply uploading his marketing brochure to one of these machines?
It would certainly have been cheaper, given that they’re all free. And our research had already shown that there seems to be very little meaningful communication between some clients and their human translators. Clearly, a
personal service is not really on the cards anyway. But is there a difference in quality?
“DeepL already scores better than the worst of the human translations (for which we paid good money) and is almost as good (or bad) as the second worst”
Colour test
I decided to put the same Dutch marketing text that we used for the ‘mystery shopper’ experiment through Google Translate, Bing Translator and DeepL. I then subjected the resulting translations to the same colour test we used for our eight human translators, marking glaring errors in red, stylistic problems in yellow and nice ideas in green. We allotted points as follows: red = minus 3; yellow = minus 1; green = plus 2. I then compared the scores with the averages from our own experiment (see table).
What we see is that, on the whole, humans still perform better than machines. Just. Interestingly, though, DeepL does considerably better than both Google and Bing. The main difference lies in the relatively low number of reds scored by DeepL. In other words, where both Google and Bing produce vast quantities of gibberish, leaving the reader totally flummoxed, DeepL does a lot better. DeepL’s translation is much easier to follow, but does run into difficulties with idiomatic expressions. At the same time, it treads carefully around the traps into which Google and Bing so charmingly tumble.
Interestingly, none of the machines scores any greens for ‘nice ideas’. That’s perhaps not surprising: they’re not designed to be inventive and so don’t come up with any startling turns of phrase, clever witticisms or eye-catching
alliterations.
In conclusion, we translators don’t need to worry too much about Google or Bing, whose core competence seems to be an ability to churn out bewildering gobbledygook. Simply chaining together a series of English words is not a guaranteed means of producing genuinely English prose.
DeepL is a different story. The English produced by the new kid reads well and is easy to follow. DeepL already scores better than the worst of the human translations (for which we paid good money) and is almost as good (or bad) as the second worst. Is DeepL a quick learner and set to improve at lightning speed? Only time will tell.
by: Tony Parr
Tony Parr is a professional business translator, language editor, and translator trainer based in the Netherlands.
He is the co-author of Handboek voor de Vertaler Nederlands-Engels. Operating under the name of Teamwork
(www.teamworkvertaalworkshops.nl), he and his business partner Marcel Lemmens have organised workshops
and conferences for translators since 1993.
