ArticleslgStudy

computer science

Infomax

Infomax is a computer science topic covered in the lgStudy science library. This page brings together a partial reference excerpt, illustrations, worked examples, real-world applications and a short study plan, so you can understand Infomax rather than just read about it. In short: Infomax', or the principle of maximum information preservation, is an optimization principle for artificial neural networks and other information processing systems. It prescribes that a function that maps a set of input values x {\displaystyle x} to a set of output values z ( x ) {\displaystyle z(x)} should be chosen or learned so as to maximize the average Shannon mutual information between x {\displaystyle x} and…

Key takeaways

  • Infomax belongs to computer science; place it in that map before memorising details.
  • Learn the definition first, then one example that makes the definition concrete.
  • Connect Infomax to a quantity you can measure, compute or draw — that is where exam questions come from.
  • Reproduce the core statement of Infomax from memory before moving on to harder problems.

Reference excerpt

Infomax', or the principle of maximum information preservation, is an optimization principle for artificial neural networks and other information processing systems. It prescribes that a function that maps a set of input values x {\displaystyle x} to a set of output values z ( x ) {\displaystyle z(x)} should be chosen or learned so as to maximize the average Shannon mutual information between x {\displaystyle x} and z ( x ) {\displaystyle z(x)} , subject to a set of specified constraints and/or noise processes. Infomax algorithms are learning algorithms that perform this optimization process. The principle was described by Linsker in 1988. The objective function is called the InfoMax objective. As the InfoMax objective is difficult to compute exactly, a related notion uses two models giving two outputs z 1 ( x ) , z 2 ( x ) {\displaystyle z_{1}(x),z_{2}(x)} , and maximizes the mutual information between these. This contrastive InfoMax objective is a lower bound to the InfoMax objective. Infomax, in its zero-noise limit, is related to the principle of redundancy reduction proposed for biological sensory processing by Horace Barlow in 1961, and applied quantitatively to retinal processing by Atick and Redlich.

Applications (Becker and Hinton, 1992) showed that the contrastive InfoMax objective allows a neural network to learn to identify surfaces in random dot stereograms (in one dimension). One of the applications of infomax has been to an independent component analysis algorithm that finds independent signals by maximizing entropy. Infomax-based ICA was described by (Bell and Sejnowski, 1995), and (Nadal and Parga, 1995).

See also FastICA

References

Bell AJ, Sejnowski TJ (December 1997). "The "Independent Components" of Natural Scenes are Edge Filters". Vision Res. 37 (23): 3327–38. doi:10.1016/S0042-6989(97)00121-1. PMC 2882863. PMID 9425547. Linsker R (1997). "A local learning rule that enables information maximization for arbitrary input distributions". Neural Computation. 9 (8): 1661–65. doi:10.1162/neco.1997.9.8.1661. S2CID 42857188. Stone, J. V. (2004). Independent Component Analysis: A tutorial introduction. Cambridge MA: MIT Press. ISBN 978-0-262-69315-8.

Worked examples

Example 1 — a first encounter with Infomax

Start with the simplest possible case. Write down what Infomax claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In computer science, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to Infomax before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about Infomax ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of Infomax

In research
Infomax appears in computer science research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry
Engineering practice reuses Infomax in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom
Infomax is common in secondary-school and first-year university syllabi. It links to neighbouring topics Applied mathematics stubs, Artificial neural networks, Computational neuroscience, so understanding it makes those chapters shorter.
In everyday life
Look for Infomax outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

Affiliate

Preply — study more efficiently by working with a personal tutor. 50% off.

How to study Infomax in 20 minutes

  1. Read the reference excerpt below once, without taking notes.
  2. Close the page and write down what Infomax means in your own words.
  3. Compare your version with the excerpt and mark what you missed.
  4. Work through the three examples above with pen and paper.
  5. Explain Infomax out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is Infomax in simple terms?

Infomax', or the principle of maximum information preservation, is an optimization principle for artificial neural networks and other information processing systems. It prescribes that a function that maps a set of input values x {\displaystyle x} to a set of output values z ( x ) {\displaystyle z(…

Why does Infomax matter?

Because it connects several computer science ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study Infomax?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on Infomax.

Tags

  • Applied mathematics stubs
  • Artificial neural networks
  • Computational neuroscience

Keep exploring