Latin America. ChatGPT and its ability to hold conversations and generate written content have focused much of the attention over the past year in the field of technology and artificial intelligence.
But in reality, AI has been around for a while, assisting us with all sorts of everyday tasks, from navigation systems to social media algorithms to machine translation.
Since neural machine translation (TAN) systems began to be widely used a few years ago, AI has increased its presence in the translation industry exponentially. And that has opened up new challenges in the relationship between human and machine translators.
Today, machine translation post-editing is the second most in-demand skill among language service providers and the task with the greatest growth potential, according to the European Language Industry Survey. Translators edit the raw machine translations, correcting the text generated by artificial intelligence. This brings numerous advantages for human translators, but it also poses significant obstacles if the quality of the automatic work is not good. That's why being able to objectively assess the quality of machine translation tools is key for the industry.
Two researchers from the Open University of Catalonia (UOC), Antoni Oliver, researcher in the Interuniversity Research Group on Linguistic Applications (GRIAL), coordinator of the TAN-IBE project and professor at the UOC's Faculty of Arts and Humanities, and Sergi Álvarez-Vidal, also a researcher at GRIAL-UOC, have developed a new method for evaluating the work of AI with the aim of improving the work of translators. reinforcing its capabilities with the potential of machine translation and increasing the quality of the final result for all users.
A New Method for Evaluating AI in Translation
Most translation and language services companies analyze the quality of AI tools in a similar way: using automated metrics. In their latest study, Assessing MT with measures of PE effort, Oliver and Álvarez-Vidal analysed the extent to which these automatic assessment systems led to the choice of tools that would actually facilitate the subsequent work of human translators. To do this, they measured the so-called post-editing effort, calculating the time, pauses, and keys used by the translator to know in detail the difficulty of editing and correcting a text generated with machine translation.
"We have concluded that there is no direct relationship between what the automatic quality assessment measures indicate and the post-editing effort," says Antoni Oliver. "Thus, we understood that it was necessary to add another step in the quality assessment system." In this way, the researchers propose to complement the automatic evaluation system with another program that allows the evaluation of the real post-editing effort. Thus, the company can choose the AI tool that really improves the efficiency of the translation process.
"We have added one more step: the translators translate a sample of machine translation with a special program developed by us. This program allows us to collect a series of data and decide whether or not the translators' effort is less than with other systems," explains Sergi Álvarez-Vidal. "If it's lower, it means that that machine translation tool is valid for the translation company's workflow."
AI, a reinforcement for human translators
Automated systems are widely used in the translation industry, although the final results are always reviewed by people. During post-editing work, human professionals accept, modify, and correct or even reject the machine-generated output entirely. "At this point it's very important to reflect on who is at the center of this task, the human post-editor or the artificial intelligence system?" asks Oliver. "We are convinced that the protagonist is the human and that the artificial intelligence system must be at their service, allowing them to be more productive and maintain the final quality of the product."
According to the researchers, the quality of machine translation directly affects the work of translators. A greater effort leads to more time and difficulty in post-editing and this has two clear consequences: it increases the risk that the final result will be of poorer quality because the translator cannot detect all the errors, and it increases the time that the translator spends post-editing, which is also a waste of money. "The quality of the machine translation is essential for a proper post-editing process," adds Oliver.
Studies such as that of the two UOC researchers also have a major indirect consequence: improving knowledge about machine translation tools in order to democratise access to them and ensure that their use does not affect the working conditions of human translators. "It is very important that knowledge about these technologies and access to these tools do not fall to a few specialists and a few companies," concludes Álvarez-Vidal. "Universities in general, and the UOC in particular, are making a great effort to include knowledge about these technologies in their studies, both undergraduate and master's."
This UOC research supports the United Nations Sustainable Development Goal (SDG) number 8, to promote inclusive and sustainable economic growth, employment and decent work for all.
