Empirical Analysis of Aggregation Methods for Collective Annotation
Ciyang Qing, Ulle Endriss, Raquel Fernandez, Justin Kruger

Abstract:
We investigate methods for aggregating the judgements
of multiple individuals in a linguistic annotation task into
a collective judgement. We define several aggregators
that take the reliability of annotators into account and
thus go beyond the commonly used majority vote, and
we empirically analyse their performance on new datasets 
of crowdsourced data.