Information gain ratio

Key takeaways

Information gain ratio belongs to mathematics; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect Information gain ratio to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of Information gain ratio from memory before moving on to harder problems.

Reference excerpt

In decision tree learning, information gain ratio is a ratio of information gain to the intrinsic information. It was proposed by Ross Quinlan, to reduce a bias towards multi-valued attributes by taking the number and size of branches into account when choosing an attribute. Information gain is also known as mutual information.

Information gain calculation Information gain is the reduction in entropy produced from partitioning a set with attributes a {\displaystyle a} and finding the optimal candidate that produces the highest value:

IG ( T , a ) = H ( T ) − H ( T | a ) , {\displaystyle {\text{IG}}(T,a)=\mathrm {H} {(T)}-\mathrm {H} {(T|a)},}

where T {\displaystyle T} is a random variable and H ( T | a ) {\displaystyle \mathrm {H} {(T|a)}} is the entropy of T {\displaystyle T} given the value of attribute a {\displaystyle a} . The information gain is equal to the total entropy for an attribute if for each of the attribute values a unique classification can be made for the result attribute. In this case the relative entropies subtracted from the total entropy are 0.

Split information calculation The split information value for a test is defined as follows:

SplitInformation ( X ) = − ∑ i = 1 n N ( x i ) N ( x ) ∗ log ⁡ 2 N ( x i ) N ( x ) {\displaystyle {\text{SplitInformation}}(X)=-\sum _{i=1}^{n}{{\frac {\mathrm {N} (x_{i})}{\mathrm {N} (x)}}*\log {_{2}}{\frac {\mathrm {N} (x_{i})}{\mathrm {N} (x)}}}}

where X {\displaystyle X} is a discrete random variable with possible values x 1 , x 2 , . . . , x n {\displaystyle {x_{1},x_{2},...,x_{n}}} and

N ( x i ) {\displaystyle N(x_{i})} being the number of times that x i {\displaystyle x_{i}} occurs divided by the total count of events N ( x ) {\displaystyle N(x)} where x {\displaystyle x} is the set of events. The split information value is a positive number that describes the potential worth of splitting a branch from a node. This in turn is the intrinsic value that the random variable possesses and will be used to remove the bias in the information gain ratio calculation.

Information gain ratio calculation The information gain ratio is the ratio between the information gain and the split information value:

IGR ( T , a ) = IG ( T , a ) / SplitInformation ( T ) {\displaystyle {\text{IGR}}(T,a)={\text{IG}}(T,a)/{\text{SplitInformation}}(T)}

… excerpt ends here. Continue reading the full article.

Illustrations

Worked examples

Example 1 — a first encounter with Information gain ratio

Start with the simplest possible case. Write down what Information gain ratio claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In mathematics, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to Information gain ratio before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about Information gain ratio ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of Information gain ratio

In research: Information gain ratio appears in mathematics research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses Information gain ratio in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: Information gain ratio is common in secondary-school and first-year university syllabi. It links to neighbouring topics Classification algorithms, Decision trees, Entropy and information, so understanding it makes those chapters shorter.
In everyday life: Look for Information gain ratio outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

How to study Information gain ratio in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what Information gain ratio means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain Information gain ratio out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is Information gain ratio in simple terms?

In decision tree learning, information gain ratio is a ratio of information gain to the intrinsic information. It was proposed by Ross Quinlan, to reduce a bias towards multi-valued attributes by taking the number and size of branches into account when choosing an attribute.

Why does Information gain ratio matter?

Because it connects several mathematics ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study Information gain ratio?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on Information gain ratio.