Krippendorff's alpha coefficient, named after academic Klaus Krippendorff, is a statistical measure of the agreement achieved when coding a set of units of analysis. Since the 1970s, alpha has been used in content analysis where textual units are categorized by trained readers, in counseling and survey research where experts code open-ended interview data into analyzable terms, in psychological testing where alternative tests of the same phenomena need to be compared, or in observational studies where unstructured happenings are recorded for subsequent analysis. Krippendorff's alpha generalizes several known statistics, often called measures of inter-coder agreement, inter-rater reliability, reliability of coding given sets of units (as distinct from unitizing) but it also distinguishes itself from statistics that are called reliability coefficients but are unsuitable to the particulars of coding data generated for subsequent analysis. Krippendorff's alpha is applicable to any number of coders, each assigning one value to one unit of analysis, to incomplete (missing) data, to any number of values available for coding a variable, to binary, nominal, ordinal, interval, ratio, polar, and circular metrics (note that this is not a metric in the mathematical sense, but often the square of a mathematical metric, see levels of measurement), and it adjusts itself to small sample sizes of the reliability data. The virtue of a single coefficient with these variations is that computed reliabilities are comparable across any numbers of coders, values, different metrics, and unequal sample sizes. Software for calculating Krippendorff's alpha is available.
Reliability data Reliability data are generated in a situation in which m ≥ 2 jointly instructed (e.g., by a code book) but independently working coders assign any one of a set of values 1,...,V to a common set of N units of analysis. In their canonical form, reliability data are tabulated in an m-by-N matrix containing N values vij that coder ci has assigned to unit uj. Define mj as the number of values assigned to unit j across all coders c. When data are incomplete, mj may be less than m. Reliability data require that values be pairable, i.e., mj ≥ 2. The total number of pairable values is ∑ j = 1 N m j = {\displaystyle \sum _{j=1}^{N}m_{j}=} n ≤ mN. To help clarify, here is what the canonical form looks like, in the abstract:
General form of alpha We denote by R {\displaystyle R} the set of all possible responses an observer can give. The responses of all observers for an example is called a unit (it forms a multiset). We denote a multiset with these units as the items, U {\displaystyle U} . Alpha is given by:
α = 1 − D o D e {\displaystyle \alpha =1-{\frac {D_{o}}{D_{e}}}}
where D o {\displaystyle D_{o}} is the disagreement observed and D e {\displaystyle D_{e}} is the disagreement expected by chance.
D o = 1 n ∑ c ∈ R ∑ k ∈ R δ ( c , k ) ∑ u ∈ U m u n c k u P ( m u , 2 ) {\displaystyle D_{o}={\frac {1}{n}}\sum _{c\in R}\sum _{k\in R}\delta (c,k)\sum _{u\in U}m_{u}{\frac {n_{cku}}{P(m_{u},2)}}}
where δ {\displaystyle \delta } is a metric function (note that this is not a metric in the mathematical sense, but often the square of a mathematical metric, see below), n {\displaystyle n} is the total number of pairable elements, m u {\displaystyle m_{u}} is the number of items in a unit, n c k u {\displaystyle n_{cku}} number of ( c , k ) {\displaystyle (c,k)} pairs in unit u {\displaystyle u} , and P {\displaystyle P} is the permutation function. Rearranging terms, the sum can be interpreted in a conceptual way as the weighted average of the disagreements of the individual units weighted by the number of coders assigned to unit j:
D o = 1 n ∑ j = 1 N m j E ( δ j ) {\displaystyle D_{o}={\frac {1}{n}}\sum _{j=1}^{N}m_{j}\,\mathbb {E} (\delta _{j})}
… excerpt ends here. Continue reading the full article.
