In statistics and psychometrics, reliability is the freedom of a measure from measurement error. In Classical Test Theory, reliability is the proportion of the observed test score variation due to "true" score variation. A measure with high reliability tends to produce similar measurements under consistent conditions:It is the characteristic of a set of test scores that relates to the amount of random error from the measurement process that might be embedded in the scores. Scores that are highly reliable are precise, reproducible, and consistent from one testing occasion to another. That is, if the testing process were repeated with a group of test takers, essentially the same results would be obtained. Various kinds of reliability coefficients, with values ranging between 0.00 (much error) and 1.00 (no error), are usually used to indicate the amount of error in the scores. For example, measurements of people's height and weight are often extremely reliable.
Types There are several general classes of reliability estimates:
Inter-rater reliability assesses the degree of agreement between two or more raters in their appraisals. For example, a person gets a stomach ache and different doctors all give the same diagnosis. Test-retest reliability assesses the degree to which test scores are consistent from one test administration to the next. Measurements are gathered from a single rater who uses the same methods or instruments and the same testing conditions. This includes intra-rater reliability. Inter-method reliability assesses the degree to which test scores are consistent when there is a variation in the methods or instruments used. This allows inter-rater reliability to be ruled out. When dealing with forms, it may be termed parallel-forms reliability. Internal consistency reliability, assesses the consistency of results across items within a test.
Difference from validity
Reliability does not imply validity. That is, a reliable measure that is measuring something consistently is not necessarily measuring what is supposed to be measured. For example, while there are many reliable tests of specific abilities, not all of them would be valid for predicting, say, job performance. While reliability does not imply validity, reliability does place a limit on the overall validity of a test. A test that is not perfectly reliable cannot be perfectly valid, either as a means of measuring attributes of a person or as a means of predicting scores on a criterion. While a reliable test may provide useful valid information, a test that is not reliable cannot possibly be valid. For example, if a set of weighing scales consistently measured the weight of an object as 500 grams over the true weight, then the scale would be very reliable, but it would not be valid (as the returned weight is not the true weight). For the scale to be valid, it should return the true weight of an object. This example demonstrates that a perfectly reliable measure is not necessarily valid, but that a valid measure necessarily must be reliable.
General model In practice, testing measures are never perfectly consistent. Theories of test reliability have been developed to estimate the effects of inconsistency on the accuracy of measurement. The basic starting point for almost all theories of test reliability is the idea that test scores reflect the influence of two sorts of factors:
Consistency factors: stable characteristics of the individual or the attribute that one is trying to measure. Inconsistency factors: features of the individual or the situation that can affect test scores but have nothing to do with the attribute being measured. These factors include:
Temporary but general characteristics of the individual: health, fatigue, motivation, emotional strain Temporary and specific characteristics of individual: comprehension of the specific test task, specific tricks or techniques of dealing with the particular test materials, fluctuations of memory, attention or accuracy Aspects of the testing situation: freedom from distractions, clarity of instructions, interaction of personality, etc. Chance factors: luck in selection of answers by sheer guessing, momentary distractions The goal of estimating reliability is to determine how much of the variability in test scores is due to measurement errors and how much is due to variability in true scores (true value). A true score is the replicable feature of the concept being measured. It is the part of the observed score that would recur across different measurement occasions in the absence of error. Errors of measurement are composed of both random error and systematic error. It represents the discrepancies between scores obtained on tests and the corresponding true scores. This conceptual breakdown is typically represented by the simple equation:
X = T + E {\displaystyle X=T+E} where X is the observed test score, T is the true score, and E is the measurement error
Classical test theory
The goal of reliability theory is to estimate errors in measurement and to suggest ways of improving tests so that errors are minimized. The central assumption of reliability theory is that measurement errors are essentially random. This does not mean that errors arise from random processes. For any individual, an error in measurement is not a completely random event. However, across a large number of individuals, the causes of measurement error are assumed to be so varied that measure errors act as random variables. If errors have the essential characteristics of random variables, then it is reasonable to assume that errors are equally likely to be positive or negative, and that they are not correlated with true scores or with errors on other tests. It is assumed that:
Mean error of measurement = 0 True scores and errors are uncorrelated Errors on different measures are uncorrelated Reliability theory shows that the variance of obtained scores is simply the sum of the variance of true scores plus the variance of errors of measurement.
… excerpt ends here. Continue reading the full article.
