ter

---

                 TRANSLATION ERROR RATE (TER) 7.0

    Matthew Snover
    Shuguang Wang
    Spyros Matsoukas

Copyright (C) 2007 BBN Technologies and University of Maryland      

---

TER contains a Java program for computing the translation error rate, which is an error metric for machine translation that messures the number of edits required to change a system output into one of the references. More details about the translation error rate can be found at http://www.cs.umd.edu/~snover/tercom/.

TER parses input reference and hypothesis files as SGML(NIST format), XML, or Trans. Both reference and hypothesis files should be in the same formats.

The code has been compiled into an executable jar file: tercom.jar.

References to TER should cite:

Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul, "A Study of Translation Edit Rate with Targeted Human Annotation," Proceedings of Association for Machine Translation in the Americas, 2006.

Custom Version

Additionally to the original version, this repository contains a custom script for computing TER.

Usage

You can easily compute TER by doing:

./TER.sh -r reference -h hypothesis

where reference is the reference text and hypothesis is the text whose quality you want to assess.

Original Version

Usage

Currently, the following options are supported:

   -N normalization, optional, default is no.
   -s case sensitivity, optional, default is insensitive
   -P no punctuations, default is with punctuations.
   -r reference file path, required.
   -h hypothesis file path, required.
   -o output formats, optional, default are all formats.
      Valid formats include "ter", "xml", "sum", "sum_nbest" and "pra".
      "ter", plain text file contains four columns: chunkid, numerrs, numwrds, ter%
      "xml", XML format with detailed alignment for each word
      "pra", plain text with alignment details, simpler than that of tercom_v6b.
      "pra_more", identical to pra output as tercom_v6b.
      "sum", same summary output as that of tercom_v6b.
      "sum_nbest", same nbest summary output as that of tercom_v6b.
   -n output name prefix, optional, no output will be generated if it is not set.
   -b beam width, optional, default is 20.
   -S translation span prefix, optional, this option only works with single reference.
   -a alternative reference path, optional, this file will be only used to compute the reference length.
   -d maximum shift distance, optional, default is 50 words.   

Examples

1. Use default values for all options and output in all formats.

java -jar tercom.jar -r <ref_file> -h <hyp_file> -n <output_prefix>

2. Enable normalization and case sensitivity, and set the beam width to 10.

java -jar tercom.jar -N -s -b 10 -r <ref_file> -h <hyp_file> -n <output_prefix>

3. Output only summary output.

java -jar tercom.jar -r <ref_file> -h <hyp_file> -o xml -n <output_prefix>

Sample Data

Sample Data and output is provided in the sample-data directory:

• sample-data/hyp.txt contains the hypothesis text.
• sample-data/ref.txt contains the reference text.
• sample-data/out.txt contains the output of running the following command.

cd sample-data/
java -jar ../tercom.jar -r ref.txt -h hyp.txt > out.txt