Announcing a new translator quality score
We know that quality is never black and white, nor can be summed up perfectly in a number. However, as a data-driven translation platform, we believe in empowering our translators with information and statistics on their performance so they can deliver quality translations at scale.
So, we’re happy to introduce a new and improved translator quality score (TQS) that will provide our translators with one number as a fair and representative indicator of their quality performance. More importantly, we believe it will serve as a helpful tool for our translators to gauge their quality and further foster a culture of learning and progression.
Quality scores are provided through an assessment tool we developed to numerically assess the quality of a translation. This is based on objective rule-based errors rather than relying on subjective opinions, and gives our active translators new check scores each month.
However, until now, translators were only shown a simple average of their last five check scores. This TQS proved to be too simplistic and failed to reflect the recent performance of a translator. In addition, we often received feedback that a translator’s TQS was being brought down by an outlying low score they received several months ago. After experimenting with different models, we’ve found one that resolves these issues and puts the translator’s needs at the forefront.
For example, with the new calculation, a 500-word job submitted two weeks ago will influence a translator’s TQS much more than a 100-word job submitted five months ago.
The main differences between the old and new TQS are summed up below.
- Non-weighted vs. weighted average
The old TQS was a non-weighted average (the most normal type of average). This meant that all five check scores equally influenced the TQS and, thus, an older outlying low score could substantially bring down a translator’s TQS.
The new TQS is instead a weighted average. This means that certain scores will influence the TQS more than others. The two determinants that will set the influence level of each score are the job size and job recency.
Our research shows that longer texts more accurately reflect the ability of a translator, so a score of a larger job will influence the TQS more than that of a smaller one.
We also recognize that the TQS should take into account quality improvements (or deterioration), so a score of a job recently submitted will influence the TQS more than that of an older job. Job size and recency have the same weight (50/50).
- Five vs. 20 check scores
The old TQS was implemented when less checks were performed on translators’ work. Our current quality system handles a much higher volume of randomized checks, and so more check scores are included in the new TQS—a higher representative sample of a translator’s work.
- All-inclusive vs. one score per language pair
The old TQS was language-pair agnostic, so some of our most active translators couldn’t differentiate their quality performance between the various language pairs they are qualified in.
We believe it’s essential for translators to know their quality performance for each of their language pairs, so the TQS is now calculated on a per-language basis for each translator. For now, the translator dashboard will display the TQS for a translator’s first language pair (by qualification date), but we plan to display one TQS per language pair in the future.
Want to become a Gengo translator?