Phrasal verbs annotation

Last modified by Alexander Kobzar on 2014-02-10

1. Instructions to download and use BLAST

The first thing to do is to download some files:

Now, before going any further, please rename the two annotation files phrasalverbs-adequacy_.blast and phrasalverbs-fluency_.blast by appending your last name to the file name, e.g. phrasalverbs-adequacy_LASTNAME.blast. This will help us identify your annotation among the other annotators.

1.1. Running BLAST from the command line

  • Open a command line terminal and navigate to the folder where you downloaded all the files. Run BLAST with this command: 
java -jar Blast-v1.3.jar -c <category_file> -a <annotation_file>

According to the annotation you are performing, for adequacy or fluency, specify the corresponding category and annotation files named accordingly.

1.2. Using BLAST's annotation interface

To navigate between sentences, use left/right arrow keys of your keyboard. By default, all the phrasal verbs are annotated as "Undefined". In order to change current annotation, just click the grade you want to annotate current phrasal verb with. We suggest that, when you finish the annotation, you can go to "Search mode" and look if there is any "Undefined" annotation left, which you might have forgotten.

NOTE: if you ran Blast without -a parameter, each time you open a file, you should click the "Edit mode" button before starting the annotation.

32.jpg

2. Annotation guidelines

When you run the command above, you will see the annotation interface of BLAST. In the annotation interface, you will see two sentences: the original (or source) sentence, in English, and the corresponding translation (or target) sentence in French. While the source is a genuine English sentence written by an English native speaker, the French translation was generated automatically using a machine translation system, and therefore it may contain errors. Your task will be to locate and classify how serious some of these errors are.

We are particularly interested in phrasal verbs in English and their translation into French verbs. Phrasal verbs are constructions in which a verb (e.g. give) is associated to a particle, preposition or adverb (e.g. up), generating a new verb (e.g. give up) with a different, new meaning (e.g. to give up means to abandon). Phrasal verbs are hard to translate specially because sometimes the verb and the particle are split by some intervening material (e.g. give something up). The sentences you are going to annotate contain such split phrasal verb occurrences. 

In all the sentences which you will annotate, capitalization and spacing errors must be ignored. For example, if there is a space before the comma, this is normal and should not be considered as an error. The same applies if the first word of the sentence or a proper name is not capitalized, for example. Even if the present guidelines are in English, you should be a native French speaker who understands English very well (or vice-versa) to perform the annotation.

On the top part of the interface, you will see a sentence in English with a highlighted phrasal verb. Below it, you will see the corresponding translated sentence in French, also with highlighted words corresponding to the translation of the English phrasal verb.

  • Example:
    • And I gave it up .
    • Et je l' ai donné .

In the example above, the English phrasal verb give up is highlighted, and the corresponding French translation is ai donné (which is not correct, by the way). In your evaluation, you should consider ONLY THE HIGHLIGHTED PARTS OF THE SENTENCES, ignoring completely whether the rest of the translation contains errors. The example below, for instance, would be considered as CORRECT for meaning, that is, the meaning of the highlighted part is preserved in the translation, regardless of the error induced by the introduction of ne in the French sentence.

  • Example:
    • And I just ate it up .
    • et je ne l' a mangé .

In this annotation task, you will be  asked to annotate two independent aspects of the translations: adequacy and fluency. Adequacy concerns meaning preservation, that is, do you understand the same thing when you read the source phrasal verb and the target verb? Fluency concerns only the French corresponding verb, whether it is a grammatical and natural way of saying something in French. The example above, for instance, is 100% adequate but not very fluent (the auxiliary verb a is not inflected correctly, as it should be ai). Please read the following detailed instructions carefully before starting the annotation. 

As specified above, you should annotate adequacy and fluency separately in two different files. Therefore, first run BLAST using the adequacy categories, save the result once you annotated all the translations, and then close BLAST. Now, in a second moment, you will run blast again using the fluency categories and repeat the annotation for all the sentences, now judging fluency instead of adequacy.

2.1. Adequacy

You should assess a phrasal verb and its translation based on the extent to which the meaning of the original English phrasal verb is preserved in the French translation. Your grade will be based on how easy and precisely you can infer the intended meaning conveyed by the translation. You should assess a sentence with a grade from 3 to 0, with 3 being the highest/best one.

  • 3. Full: the French translated and highlighted words convey exactly the same meaning as their counterparts in English.
    • Example:
      • Do you think a lot of classes next school year should try this thing out ?
      • pensez-vous que beaucoup de classes prochaine année scolaire devrait essayer cette chose ?
  • 2. Partial: the general meaning can be inferred without referring to the English sentence. The translated highlighted words sound clumsy, unnatural and/or funny, less relevant words might be missing or spurious words were added, but one can roughly understand the global meaning. As a native French speaker, though, you would prefer to translate the phrasal verb by another verb or construction than the one chosen by the machine translation system (e.g. distribuer in the example below).
    • Example:
      •  And then we gave condoms out everywhere on the street 
      •  Et puis nous avons_ donné des préservatifs sorti partout 
  • 1. None: the meaning is not conveyed in the translated sentence. In other words, the meaning of the French highlighted words cannot be understood without reading and understanding the English sentence. Some proficiency in the usage of phrasal verbs in English might be required to understand the French translation. In the examples below, example 1 shows a case where the French highlighted words mean the opposite of the English phrasal verb, example 2 shows a case where the French translation makes no sense and in example 3 the phrasal verb disappeared from the translation.
    • Examples:
      • 1.a) we would all do more to put the fire out .
      • 1.b)  nous serions tous faire plus de mettre le feu

      • 2.a) as soon as someone turns it on , 
      • 2.b) dès que quelqu' un s' en sert sur , 

      • 3.a) Look at the pentagon , for instance , pull this out -- it becomes a boat shape trapezium .
      • 3.b) regardez les pentagone , par exemple , ceci -- c' est une forme de bateau trapezium . 
  • 0. Unable to judge: There is a problem with the source English sentence, which looks very weird, broken and/or too short to be understood. If you cannot grasp the meaning of the phrasal verb, you can consult Wordnet glosses. If there is no definition for it or you still experience problems in understanding its meaning  in the context of the given sentence, you can further look it up at WordReference. If you still do not understand the meaning of expression after having looked it up in both sources, you can consider as being unable to judge.

2.2. Fluency

You will assess a translated phrsal verb based on its grammatical correctness in French, regardless of its meaning. Your grade will be based on how well the highlighted French words are inflected, specially regarding verb agreement in tense, number and gender. In this evaluation, you should NOT LOOK AT THE ORIGINAL ENGLISH SENTENCE, since the meaning is not important, but only the fluency of the French sentence. You should assess each French highlighted construction with a grade from 4 to 1, with 4 being the highest/best translation.

  • 4. Fluent: the highlighted words in French show neither spelling nor syntax errors. If there are several highlighted words, word order among them as well as among constituents they have syntactic relationship with should be respected. Verbs should have the required subject and object(s), and agree with them. There should be no extra particles and prepositions in the sentence. In example 1 below, even if the translation sense is wrong, the verb is correctly inflected in the infinitive form and its object (le feu) occurs after it. In example 2, both sense and inflection in the first person singular are correct.
    • Examples:
      • 1.a) we would all do more to put the fire out .
      • 1.b)  nous serions tous faire plus de mettre le feu

      • 2.a) and I am not making this up 
      • 2.b)  et je n' invente rien
  • 3. Non-native: the verb form and/or its agreement with subject/object are wrong. The translation quality of subject/objects themselves should not be considered. Errors can be easily corrected. In example 1 below, the auxiliary verb is missing and, as in example 2, the person is not correctly inflected (should be a battu and imprimez).
    • Examples:
      •  1.a) The day before , her husband beat her up
      • 1.b) La veille , son mari battre sa 

      • 2.a) well , if you print it out , it looks like a thousand pounds of material
      • 2.b) Eh bien , si vous l' imprimer , ça ressemble à un millier de kilos de matériel 
  • 2. Disfluent: the use of the highlighted words makes the sentence syntactically incoherent. This may be due to the use of a noun instead of a verb (example 1), the presence of an extra preposition (example 2), wrong word order, or absence of a required preposition.
    • Examples:
      • 1.a) and then we jack it up into the air .
      • 1.b) et ensuite nous valet il en l' air .

      • 2.a) I stepped down as chairman, but someone else has taken it on 
      • 2.b)  je suis descendu comme président, mais quelqu' un d' autre a pris sur 
  • 1. Incomprehensible: there is no highlighted word in the translation (example 1) or these are English words which remained untranslated (example 2).
    • Example:
      • 1.a) And here we are coiling them up . 
      • 1.b) et ici nous sommes coiling eux . 

      • 2.a) Look at the pentagon , for instance , pull this out -- it becomes a boat shape trapezium .
      • 2.b) regardez les pentagone , par exemple , ceci -- c' est une forme de bateau trapezium . 
Tags:
Created by Carlos Ramisch on 2013-04-12
Copyright 2004-2017 XWiki
2.4.30467