In probability theory and statistics, a collection of random variables is independent and identically distributed (i.i.d., iid, or IID) if each random variable has the same probability distribution as the others and all are mutually independent. IID was first defined in statistics and finds application in many fields, such as data mining and signal processing.
Introduction Statistics commonly deals with random samples. A random sample can be thought of as a set of objects that are chosen randomly. More formally, it is "a sequence of independent, identically distributed (IID) random data points." In other words, the terms random sample and IID are synonymous. In statistics, "random sample" is the typical terminology, but in probability, it is more common to say "IID."
Identically distributed means that there are no overall trends — the distribution does not fluctuate and all items in the sample are taken from the same probability distribution. Independent means that the sample items are all independent events. In other words, they are not connected to each other in any way; knowledge of the value of one variable gives no information about the value of the other and vice versa.
Application Independent and identically distributed random variables are often used as an assumption, which tends to simplify the underlying mathematics. In practical applications of statistical modeling, however, this assumption may or may not be realistic. The i.i.d. assumption is also used in the central limit theorem, which states that the probability distribution of the sum (or average) of i.i.d. variables with finite variance approaches a normal distribution. The i.i.d. assumption frequently arises in the context of sequences of random variables. Then, "independent and identically distributed" implies that an element in the sequence is independent of the random variables that came before it. In this way, an i.i.d. sequence is different from a Markov sequence, where the probability distribution for the nth random variable is a function of the previous random variable in the sequence (for a first-order Markov sequence). An i.i.d. sequence does not imply the probabilities for all elements of the sample space or event space must be the same. For example, repeated throws of loaded dice will produce a sequence that is i.i.d., despite the outcomes being biased. In signal processing and image processing, the notion of transformation to i.i.d. implies two specifications, the "i.d." part and the "i." part: i.d. – The signal level must be balanced on the time axis. i. – The signal spectrum must be flattened, i.e. transformed by filtering (such as deconvolution) to a white noise signal (i.e., a signal where all frequencies are equally present).
Definition
Definition for two random variables Suppose that the random variables X {\displaystyle X} and Y {\displaystyle Y} are defined to assume values in I ⊆ R {\displaystyle I\subseteq \mathbb {R} } . Let F X ( x ) = P ( X ≤ x ) {\displaystyle F_{X}(x)=\operatorname {P} (X\leq x)} and F Y ( y ) = P ( Y ≤ y ) {\displaystyle F_{Y}(y)=\operatorname {P} (Y\leq y)} be the cumulative distribution functions of X {\displaystyle X} and Y {\displaystyle Y} , respectively, and denote their joint cumulative distribution function by F X , Y ( x , y ) = P ( X ≤ x ∧ Y ≤ y ) {\displaystyle F_{X,Y}(x,y)=\operatorname {P} (X\leq x\land Y\leq y)} . Two random variables X {\displaystyle X} and Y {\displaystyle Y} are independent if and only if F X , Y ( x , y ) = F X ( x ) ⋅ F Y ( y ) {\displaystyle F_{X,Y}(x,y)=F_{X}(x)\cdot F_{Y}(y)} for all x , y ∈ I {\displaystyle x,y\in I} . (For the simpler case of events, two events A {\displaystyle A} and B {\displaystyle B} are independent if and only if P ( A ∧ B ) = P ( A ) ⋅ P ( B ) {\displaystyle P(A\land B)=P(A)\cdot P(B)} , see also Independence (probability theory) § Two random variables.) Two random variables X {\displaystyle X} and Y {\displaystyle Y} are identically distributed if and only if F X ( x ) = F Y ( x ) {\displaystyle F_{X}(x)=F_{Y}(x)} for all x ∈ I {\displaystyle x\in I} . Two random variables X {\displaystyle X} and Y {\displaystyle Y} are i.i.d. if they are independent and identically distributed, i.e. if and only if
… excerpt ends here. Continue reading the full article.
