Modeling The Complexity Of Manual Annotation Tasks - ((not So ...

Copy and paste this link to your website, so they can see this document directly without any plugins.



Keywords

al.,, annotate?, annotation, freedom, into, What, possibilities, bias, Introduction, Level, (degree, Proceedings, yppB, discriminated, Some, [Marcus, gene, 2010], several, tagset, simple, implement, gain, time, pages, strain., alleles, were, null, annotated

Transcript

Modeling the Complexity of Manual Annotation Tasks:
a Grid of Analysis
Karën Fort, Adeline Nazarenko, Sophie Rosset
December 13th
1 / 24
Modeling the Complexity of Manual Annotation Tasks:
a Grid of Analysis
Karën Fort, Adeline Nazarenko, Sophie Rosset
December 13th
1 / 24
Introduction
Manual annotation: notoriously costly
Penn Treebank [Marcus et al., 1993]:
4.8 million tokens annotated with POS ⇒ learning phase of 1 month,
to reach 3,000 words/h
3 million tokens annotated in syntax ⇒ learning phase of 2 months, to
reach 475 words/h
Prague Dependency Treebank [Böhmová et al., 2001]:
1.8 million tokens annotated with POS and syntax
⇒ 5 years, 22 persons (max. 17 in parallel), 600,000 dollars
GENIA [Kim et al., 2008]:
9,372 sentences annotated in microbiology (proteins and gene names)
⇒ 5 part-time annotators, 1 senior coordinator and 1 junior for 1.5 year
2 / 24
Introduction
Some solutions...
tag dictionary [Carmen et al., 2010]:
+ simple to implement
- bias
pre-annotation:
+ gain in time and consistency
[Marcus et al., 1993, Fort and Sagot, 2010]
- bias
active learning:
+ gain in time [Cohn et al., 1995, Engelson and Dagan, 1996]
- bias and not so simple to implement
crowdsourcing:
I GWAPs: real cost rarely estimated [Chamberlain et al., 2013]
I microworking (MTurk): quality and ethical issues [Fort et al., 2011]
3 / 24
Introduction
Some solutions...
tag dictionary [Carmen et al., 2010]:
+ simple to implement
- bias
pre-annotation:
+ gain in time and consistency
[Marcus et al., 1993, Fort and Sagot, 2010]
- bias
active learning:
+ gain in time [Cohn et al., 1995, Engelson and Dagan, 1996]
- bias and not so simple to implement
crowdsourcing:
I GWAPs: real cost rarely estimated [Chamberlain et al., 2013]
I microworking (MTurk): quality and ethical issues [Fort et al., 2011]
3 / 24
Introduction
Some solutions...
tag dictionary [Carmen et al., 2010]:
+ simple to implement
- bias
pre-annotation:
+ gain in time and consistency
[Marcus et al., 1993, Fort and Sagot, 2010]
- bias
active learning:
+ gain in time [Cohn et al., 1995, Engelson and Dagan, 1996]
- bias and not so simple to implement
crowdsourcing:
I GWAPs: real cost rarely estimated [Chamberlain et al., 2013]
I microworking (MTurk): quality and ethical issues [Fort et al., 2011]
3 / 24
Introduction
Some solutions...
tag dictionary [Carmen et al., 2010]:
+ simple to implement
- bias
pre-annotation:
+ gain in time and consistency
[Marcus et al., 1993, Fort and Sagot, 2010]
- bias
active learning:
+ gain in time [Cohn et al., 1995, Engelson and Dagan, 1996]
- bias and not so simple to implement
crowdsourcing:
I GWAPs: real cost rarely estimated [Chamberlain et al., 2013]
I microworking (MTurk): quality and ethical issues [Fort et al., 2011]
3 / 24
Introduction
Some solutions...
4 / 24
Introduction
... to a problem that is still little known
4 / 24
Introduction
Some traces
Large-scale campaigns feedback
[Marcus et al., 1993, Abeillé et al., 2003]
Good practices:
I formats [Ide and Romary, 2006]
I organization [Bontcheva et al., 2010]
I evaluation [Krippendorff, 2004]
Partial methodologies: agile annotation
[Bonneau-Maynard et al., 2005, Voormann and Gut, 2008]
Some insights from cognitive science [Tomanek et al., 2010]
What is complex in manual annotation?
5 / 24
Introduction
Some traces
Large-scale campaigns feedback
[Marcus et al., 1993, Abeillé et al., 2003]
Good practices:
I formats [Ide and Romary, 2006]
I organization [Bontcheva et al., 2010]
I evaluation [Krippendorff, 2004]
Partial methodologies: agile annotation
[Bonneau-Maynard et al., 2005, Voormann and Gut, 2008]
Some insights from cognitive science [Tomanek et al., 2010]
What is complex in manual annotation?
5 / 24
Analysing the complexity of an annotation campaign
1 Introduction
2 Analysing the complexity of an annotation campaign
3 What to annotate?
4 How to annotate?
5 Synthesis
6 Conclusion and prospects
6 / 24
Analysing the complexity of an annotation campaign
Complexity dimensions
5 independent dimensions:
I 2 related to the localisation of
annotations
I 3 related to the characterisation of
annotations
1 not independent: the context
Discrimination
Delimitation
Expressivity
Tagset
Ambiguity
Context
Scale from 0 (null complexity) to 1 (maximal complexity) to allow for
the comparison between campaigns
Independent from the volume to annotate and the number of
annotators
6 / 24
Analysing the complexity of an annotation campaign
Elementary Annotation Task (EAT)
From a complex task, to several elementary tasks:
Criteria
An annotation task may be decomposed into at least two EATs if the used
tagset can be decomposed into reduced and independent tagsets.
→ may correspond to several successive annotation steps or not
7 / 24
Analysing the complexity of an annotation campaign
Example: gene renaming
1 Identification of gene names in the source signal:
The yppB gene complemented the defect of the recG40
strain. yppB and ypbC and their respective null alleles were
termed “recU” and “recU1” (recU:cat) and “recS” and
“recS1” (recS:cat), respectively.
2 Identification of gene couples expressing a renaming relation:
The yppB gene complemented the defect of the recG40
strain. yppB and ypbC and their respective null alleles were
termed “recU” and “recU1” (recU:cat) and “recS” and
“recS1” (recS:cat), respectively.
8 / 24
What to annotate?
Discrimination
Parts-of-speech [Marcus et al., 1993], pre-annotated :
I/PRP do/VBP n’t/RB feel/VB very/RB ferocious/JJ ./.
Gene renaming[Fort et al., 2012], no pre-annotation:
The yppB:cat and ypbC:cat null alleles rendered cells sensitive to
DNA-damaging agents, impaired plasmid transformation (25- and 100-fold),
and moderately affected chromosomal transformation when present in an
otherwise Rec+ B. subtilis strain. The yppB gene complemented the defect
of the recG40 strain. yppB and ypbC and their respective null alleles were
termed recU and “recU1” (recU:cat) and recS and “recS1” (recS:cat),
respectively. The recU and recS mutations were introduced into rec-deficient
strains representative of the alpha (recF), beta (addA5 addB72), gamma
(recH342), and epsilon (recG40) epistatic groups.
⇒ more difficult if the units to annotate are scattered, in particular if the
segmentation is not obvious.
9 / 24
What to annotate?
Discrimination
Parts-of-speech [Marcus et al., 1993], pre-annotated :
I/PRP do/VBP n’t/RB feel/VB very/RB ferocious/JJ ./.
Gene renaming[Fort et al., 2012], no pre-annotation:
The yppB:cat and ypbC:cat null alleles rendered cells sensitive to
DNA-damaging agents, impaired plasmid transformation (25- and 100-fold),
and moderately affected chromosomal transformation when present in an
otherwise Rec+ B. subtilis strain. The yppB gene complemented the defect
of the recG40 strain. yppB and ypbC and their respective null alleles were
termed recU and “recU1” (recU:cat) and recS and “recS1” (recS:cat),
respectively. The recU and recS mutations were introduced into rec-deficient
strains representative of the alpha (recF), beta (addA5 addB72), gamma
(recH342), and epsilon (recG40) epistatic groups.
⇒ more difficult if the units to annotate are scattered, in particular if the
segmentation is not obvious.
9 / 24
What to annotate?
Discrimination
The discrimination weight is all the more high as the proportion of what
should be annotated as compared to what could be annotated is low.
Definition
Discrimination(Flow) = 1− |Annotations(Flow)|∑LevelSeg
i=1 |UnitsObtainedBySegi (Flow)|
⇒ Need for a reference segmentation
10 / 24
What to annotate?
Parts-of-speech[Marcus et al., 1993] :
I/PRP do/VBP n’t/RB feel/VB very/RB ferocious/JJ ./.
DiscriminationPTBPOS = 0
Gene renaming[Fort et al., 2012] :
The yppB:cat and ypbC:cat null alleles rendered cells sensitive to
DNA-damaging agents, impaired plasmid transformation (25- and 100-fold),
and moderately affected chromosomal transformation when present in an
otherwise Rec+ B. subtilis strain. The yppB gene complemented the defect
of the recG40 strain. yppB and ypbC and their respective null alleles were
termed recU and “recU1” (recU:cat) and recS and “recS1” (recS:cat),
respectively. The recU and recS mutations were introduced into rec-deficient
strains representative of the alpha (recF), beta (addA5 addB72), gamma
(recH342), and epsilon (recG40) epistatic groups.
DiscriminationIdentification = 0, 9
DiscriminationRenaming = 0, 95
11 / 24
What to annotate?
Boundaries delimitation
extending or shrinking the discriminated unit:
Madame Chirac → Monsieur et Madame Chirac
decompose a discriminated unit into several elements:
le préfet Érignac → le préfet Érignac
or group together several discriminated units into one unique
annotation:
Sa Majesté
le roi Mohamed VI → Sa Majesté le roi Mohamed VI
12 / 24
What to annotate?
Boundaries delimitation
extending or shrinking the discriminated unit:
Madame Chirac → Monsieur et Madame Chirac
decompose a discriminated unit into several elements:
le préfet Érignac → le préfet Érignac
or group together several discriminated units into one unique
annotation:
Sa Majesté
le roi Mohamed VI → Sa Majesté le roi Mohamed VI
12 / 24
What to annotate?
Boundaries delimitation
extending or shrinking the discriminated unit:
Madame Chirac → Monsieur et Madame Chirac
decompose a discriminated unit into several elements:
le préfet Érignac → le préfet Érignac
or group together several discriminated units into one unique
annotation:
Sa Majesté
le roi Mohamed VI → Sa Majesté le roi Mohamed VI
12 / 24
What to annotate?
Boundaries delimitation
Definition
Delimitation(Flow) = min
(
Substitutions + Additions + Deletions
|Annotations(Flow)|
, 1
)
DelimitationIdentification = 0
DelimitationRenaming = 0
DelimitationPTBPOS = 0
DélimitationENTypesSubtypes = 1
DélimitationENComponents = 0, 3
13 / 24
How to annotate?
Expressiveness of the annotation language
Definition
The degrees of expressiveness of the annotation language are the following:
0.25: type languages
0.5: relational languages of arity 2
0.75: relational languages of arity higher than 2
1: higher-order languages
ExpressivenessIdentification = 0.25
ExpressivenessRenaming = 0.25
14 / 24
How to annotate?
Dimension of the tagset
Types and sub-types used for structured NE
annotation [Grouin et al., 2011]
Level 1: pers, func, loc, prod, org, time, amount → 7 possibilities (degree
of freedom = 6).
Level 2: prod.object, prod.serv, prod.fin, prod.soft, prod.doctr, prod.rule,
prod.art, prod.media, prod.award → 9 possibilities (degree of freedom = 8).
Level 3: loc.adm.town, loc.adm.reg, loc.adm.nat, loc.adm.sup → 4
possibilities (degree of freedom = 3).
15 / 24
How to annotate?
Dimension of the tagset
Level 1: pers, func, loc, prod, org, time, amount → 7 possibilities (degree
of freedom = 6).
Level 2: prod.object, prod.serv, prod.fin, prod.soft, prod.doctr, prod.rule,
prod.art, prod.media, prod.award → 9 possibilities (degree of freedom = 8).
Level 3: loc.adm.town, loc.adm.reg, loc.adm.nat, loc.adm.sup → 4
possibilities (degree of freedom = 3).
15 / 24
How to annotate?
Dimension of the tagset
Level 1: pers, func, loc, prod, org, time, amount → 7 possibilities (degree
of freedom = 6).
Level 2: prod.object, prod.serv, prod.fin, prod.soft, prod.doctr, prod.rule,
prod.art, prod.media, prod.award → 9 possibilities (degree of freedom = 8).
Level 3: loc.adm.town, loc.adm.reg, loc.adm.nat, loc.adm.sup → 4
possibilities (degree of freedom = 3).
15 / 24
How to annotate?
Dimension of the tagset
Level 1: pers, func, loc, prod, org, time, amount → 7 possibilities (degree
of freedom = 6).
Level 2: prod.object, prod.serv, prod.fin, prod.soft, prod.doctr, prod.rule,
prod.art, prod.media, prod.award → 9 possibilities (degree of freedom = 8).
Level 3: loc.adm.town, loc.adm.reg, loc.adm.nat, loc.adm.sup → 4
possibilities (degree of freedom = 3).
15 / 24
How to annotate?
Dimension of the tagset
Degree of freedom
ν = ν1 + ν2 + . . .+ νm
where νi is the maximal degree of freedom the annotator has when choosing the i th sub-type
(νi = ni − 1).
Dimension of the tagset
Dimension(Flow) = min(
ν
τ
, 1)
where τ is the threshold from which we consider the tagset to be very large (experimentally
determined).
DimensionIdentification = 0
DimensionRenaming = 0.04
DimensionNETypesSubtypes = 0.34 16 / 24
How to annotate?
Degree of ambiguity: residual ambiguity
Using the traces left by the annotators:
[...] 3CDproM can process both structural and
nonstructural precursors of the "too-generic">poliovirus polyprotein [...].
Définition
AmbiguityRes(Flow) =
|Annotationsamb|
|Annotations|
AmbiguityResIdentification = 0.04
AmbiguityResRenaming = 0.02
17 / 24
How to annotate?
Degree of ambiguity: theoretical ambiguity
Proportion of the units to annotate that corresponds to ambiguous
vocables.
Definition
AmbiguityTh(Flow) =
∑|Voc(Flow)|
voci=1 (Ambig(voci ) ∗ freq(voci ,Flow))
|Units(Flow)|
with
Ambig(voci ) =
{
1 if |Tags(voci )| > 1
0 else
AmbiguityThIdentification = 0.01
→ Does not apply to renaming relations (2 EATs).
18 / 24
How to annotate? Weight of the context
Context to take into account
size of the window to take into account in the source signal:
I The sentence:
I/PRP do/VBP n’t/RB feel/VB very/RB ferocious/JJ ./.
I ... or more:
number of knowledge elements to be rallied or degree of
accessibility of the knowledge sources that are consulted:
I annotation guidelines
I nomenclatures (Swiss-Prot)
I new sources to be found (Wikipedia, etc.)
19 / 24
How to annotate? Weight of the context
Weight of the context
0
0.5
0.75
Co-text size
Accessibility of
knowledge
sourcesAnnotation
guide
Paragraph
Full text
Sentence
Identified
external
sources
New
sources
to indentify
0.25
1
20 / 24
Synthesis
Synthesis of the complexity dimensions
Discrimination
Delimitation
Expressivity
Tagset
Ambiguity
Context
Classification of it pronouns as
anaphoric or impersonal
Discrimination
Delimitation
Expressivity
Tagset
Ambiguity
Context
Gene names identification
21 / 24
Synthesis
Synthesis
Discrimination
Delimitation
Expressivity
Tagset
Ambiguity
Context
Gene renaming campaign (2 EATs)
22 / 24
Conclusion and prospects Conclusion
1 Introduction
2 Analysing the complexity of an annotation campaign
3 What to annotate?
4 How to annotate?
5 Synthesis
6 Conclusion and prospects
23 / 24
Conclusion and prospects Conclusion
Conclusion and prospects
A grid of analysis:
to use during preparatory work
to help asking the right questions and finding the appropriate solutions
→ that should be computed more or less automatically
→ that should be integrated as part of annotation tools
[Kaplan et al., 2010, Bontcheva et al., 2010]
23 / 24
Thank you
Thank you for your attention!
24 / 24
Bibliographie
Abeillé, A., Clément, L., and Toussenel, F. (2003).
Building a treebank for French.
In Abeillé, A., editor, Treebanks, pages 165 –187. Kluwer, Dordrecht.
Böhmová, A., Hajič, J., Hajičová, E., and Hladká, B. (2001).
The prague dependency treebank: Three-level annotation scenario.
In Abeillé, A., editor, Treebanks: Building and Using Syntactically
Annotated Corpora. Kluwer Academic Publishers.
Bonneau-Maynard, H., Rosset, S., Ayache, C., Kuhn, A., and Mostefa,
D. (2005).
Semantic annotation of the French Media dialog corpus.
In Proceedings of the InterSpeech, Lisbonne, Portugal.
Bontcheva, K., Cunningham, H., Roberts, I., and Tablan, V. (2010).
Web-based collaborative corpus annotation: Requirements and a
framework implementation.
In Witte, R., Cunningham, H., Patrick, J., Beisswanger, E., Buyko, E.,
Hahn, U., Verspoor, K., and Coden, A. R., editors, Proceedings of the
24 / 24
Bibliographie
workshop on New Challenges for NLP Frameworks (NLPFrameworks
2010), La Valette, Malte. ELRA.
Carmen, M., Felt, P., Haertel, R., Lonsdale, D., McClanahan, P.,
Merkling, O., Ringger, E., and Seppi, K. (2010).
Tag dictionaries accelerate manual annotation.
In Proceedings of the International Conference on Language Resources
and Evaluation (LREC), La Valette, Malte. European Language
Resources Association (ELRA).
Chamberlain, J., Fort, K., Kruschwitz, U., Lafourcade, M., and Poesio,
M. (2013).
Using games to create language resources: Successes and limitations of
the approach.
In Gurevych, I. and Kim, J., editors, The People’s Web Meets NLP,
Theory and Applications of Natural Language Processing, pages 3–44.
Springer Berlin Heidelberg.
Cohn, D. A., Ghahramani, Z., and Jordan, M. I. (1995).
Active learning with statistical models.
24 / 24
Bibliographie
In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in
Neural Information Processing Systems, volume 7, pages 705–712. The
MIT Press.
Engelson, S. P. and Dagan, I. (1996).
Minimizing manual annotation cost in supervised training from
corpora.
In Proceedings of the 34th annual meeting on Association for
Computational Linguistics, pages 319–326, Morristown, NJ, USA.
Association for Computational Linguistics.
Fort, K., Adda, G., and Cohen, K. B. (2011).
Amazon Mechanical Turk: Gold mine or coal mine?
Computational Linguistics (editorial), 37(2):413–420.
Fort, K., François, C., Galibert, O., and Ghribi, M. (2012).
Analyzing the impact of prevalence on the evaluation of a manual
annotation campaign.
In Proceedings of the International Conference on Language Resources
and Evaluation (LREC), Istanbul, Turquie.
24 / 24
Bibliographie
7 pages.
Fort, K. and Sagot, B. (2010).
Influence of pre-annotation on POS-tagged corpus development.
In Proceedings of the Fourth ACL Linguistic Annotation Workshop,
pages 56–63, Uppsala, Suède.
Grouin, C., Rosset, S., Zweigenbaum, P., Fort, K., Galibert, O., and
Quintard, L. (2011).
Proposal for an extension of traditional named entities: From
guidelines to evaluation, an overview.
In Proceedings of the 5th Linguistic Annotation Workshop, pages
92–100, Portland, Oregon, USA.
Poster.
Ide, N. and Romary, L. (2006).
Representing linguistic corpora and their annotations.
In Proceedings of the International Conference on Language Resources
and Evaluation (LREC), Gène, Italie.
24 / 24
Bibliographie
Kaplan, D., Iida, R., and Tokunaga, T. (2010).
Annotation process management revisited.
In Proceedings of the International Conference on Language Resources
and Evaluation (LREC), pages 365 – 366, La Valette, Malte.
Kim, J.-D., Ohta, T., and Tsujii, J. (2008).
Corpus annotation for mining biomedical events from literature.
BMC Bioinformatics, 9(1):10.
Krippendorff, K. (2004).
Content Analysis: An Introduction to Its Methodology,.
Sage, Thousand Oaks, CA., USA, second edition edition.
Marcus, M., Santorini, B., and Marcinkiewicz, M. A. (1993).
Building a large annotated corpus of English : The Penn Treebank.
Computational Linguistics, 19(2):313–330.
Tomanek, K., Hahn, U., Lohmann, S., and Ziegler, J. (2010).
A cognitive cost model of annotations based on eye-tracking data.
24 / 24
Bibliographie
In Proceedings of the Annual Meeting of the Association for
Computational Linguistics (ACL), ACL’10, pages 1158–1167,
Stroudsburg, PA, USA. Association for Computational Linguistics.
Voormann, H. and Gut, U. (2008).
Agile corpus creation.
Corpus Linguistics and Linguistic Theory, 4(2):235–251.
24 / 24

PDF Document reader online

This website is focused on providing document in readable format, online without need to install any type of software on your computer. If you are using thin client, or are not allowed to install document reader of particular type, this application may come in hand for you. Simply upload your document, and Docureader.top will transform it into readable format in a few seconds. Why choose Docureader.top?

  1. Unlimited sharing - you can upload document of any size. If we are able to convert it into readable format, you have it here - saved for later or immediate reading
  2. Cross-platform - no compromised when reading your document. We support most of modern browers without the need of installing any of external plugins. If your device can oper a browser - then you can read any document on it
  3. Simple uploading - no need to register. Just enter your email, title of document and select the file, we do the rest. Once the document is ready for you, you will receive automatic email from us.

Previous 10

Next 10