Info-metrics is an interdisciplinary approach to scientific modeling, inference and efficient information processing. It is the science of modeling, reasoning, and drawing inferences under conditions of noisy and limited information. From the point of view of the sciences, this framework is at the intersection of information theory, statistical methods of inference, applied mathematics, computer science, econometrics, complexity theory, decision analysis, modeling, and the philosophy of science. Info-metrics provides a constrained optimization framework to tackle under-determined or ill-posed problems – problems where there is not sufficient information for finding a unique solution. Such problems are very common across all sciences: available information is incomplete, limited, noisy and uncertain. Info-metrics is useful for modelling, information processing, theory building, and inference problems across the scientific spectrum. The info-metrics framework can also be used to test hypotheses about competing theories or causal mechanisms.
History Info-metrics evolved from the classical maximum entropy formalism, which is based on the work of Shannon. Early contributions were mostly in the natural and mathematical/statistical sciences. Since the mid 1980s and especially in the mid 1990s the maximum entropy approach was generalized and extended to handle a larger class of problems in the social and behavioral sciences, especially for complex problems and data. The word info-metrics was coined in 2009 by Amos Golan, right before the interdisciplinary Info-Metrics Institute was inaugurated.
Preliminary definitions Consider a random variable X {\textstyle X} that can result in one of K distinct outcomes. The probability p k {\textstyle p_{k}} of each outcome x k {\textstyle x_{k}} is p k = p ( x k ) {\textstyle p_{k}=p(x_{k})} for k = 1 , 2 , … , K {\textstyle k=1,2,\ldots ,K} . Thus, P {\textstyle P} is a K-dimensional probability distribution defined for X {\textstyle X} such that p k ϵ [ 0 , 1 ] {\displaystyle p_{k}\epsilon [0,1]} and ∑ k p k = 1 {\textstyle \sum _{k}p_{k}=1} . Define the informational content of a single outcome x k {\textstyle x_{k}} to be h ( x k ) = h ( p k ) = log 2 ( 1 / p k ) {\textstyle h(x_{k})=h(p_{k})=\log _{2}(1/p_{k})} (e.g., Shannon). Observing an outcome at the tails of the distribution (a rare event) provides much more information than observing another, more probable, outcome. The entropy is the expected information content of an outcome of the random variable X whose probability distribution is P:
H ( P ) = ∑ k = 1 K p k log 2 ( 1 p k ) = − ∑ k = 1 K p k log 2 ( p k ) = E [ log 2 ( 1 P ( X ) ) ] {\displaystyle H(P)=\sum _{k=1}^{K}p_{k}\log _{2}\left({\frac {1}{p_{k}}}\right)=-\sum _{k=1}^{K}p_{k}\log _{2}(p_{k})=\operatorname {E} \left[\log _{2}\left({\frac {1}{P(X)}}\right)\right]}
Here p k log 2 ( p k ) ≡ 0 {\displaystyle p_{k}\log _{2}(p_{k})\equiv 0} if p k = 0 {\displaystyle p_{k}=0} , and E {\displaystyle \operatorname {E} } is the expectation operator.
… excerpt ends here. Continue reading the full article.
