mathematics

Negative log predictive density

Negative log predictive density is a mathematics topic covered in the lgStudy science library. This page brings together a partial reference excerpt, illustrations, worked examples, real-world applications and a short study plan, so you can understand Negative log predictive density rather than just read about it. In short: In statistics, the negative log predictive density (NLPD) is a measure of error between a model's predictions and associated true values. A smaller value is better.

Key takeaways

Negative log predictive density belongs to mathematics; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect Negative log predictive density to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of Negative log predictive density from memory before moving on to harder problems.

Reference excerpt

In statistics, the negative log predictive density (NLPD) is a measure of error between a model's predictions and associated true values. A smaller value is better. Importantly the NLPD assesses the quality of the model's uncertainty quantification. It is used for both regression and classification. To compute: (1) find the probabilities given by the model to the true labels. (2) find the negative log of this product. (we actually find the negative of the sum of the logs, for numerical reasons).

Definition

NLPD = − ∑ i = 1 N log ⁡ p ( y i = t i | x i ) {\displaystyle {\text{NLPD}}=-\sum _{i=1}^{N}\log p(y_{i}=t_{i}|\mathbf {x_{i}} )}

where p ( y | x ) {\displaystyle p(y|\mathbf {x} )} is the model, x i {\displaystyle \mathbf {x_{i}} } are the inputs (independent variables) and t i {\displaystyle t_{i}} are the observations outputs (dependent variable). Often the mean rather than the sum is used (by dividing by N),

NLPD = − 1 N ∑ i = 1 N log ⁡ p ( y i = t i | x i ) {\displaystyle {\text{NLPD}}=-{\frac {1}{N}}\sum _{i=1}^{N}\log p(y_{i}=t_{i}|\mathbf {x_{i}} )}

Example

Calculating the NLPD for a simple classification example We have a method that classifies images as dogs or cats. Importantly it assigns probabilities to the two classes. We show it a picture of three dogs and three cats. It predicts that the probability of the first three being dogs as 0.9 and 0.4, 0.7 and of the last three being cats as 0.8 and 0.4, 0.3. The NLPD is: − ( log ⁡ 0.9 + log ⁡ 0.4 + log ⁡ 0.7 + log ⁡ 0.8 + log ⁡ 0.4 + log ⁡ 0.3 ) = 3.72 {\displaystyle -(\log 0.9+\log 0.4+\log 0.7+\log 0.8+\log 0.4+\log 0.3)=3.72} .

Comparing to a classifier with better accuracy but overconfident We compare this to another classifier which predicts the first three as being dogs as 0.95, 0.98, 0.02, and the last three being cats as 0.99, 0.96,0.96. The NLPD for this classifier is 4.08. The first classifier only guessed half correctly, so did worse on a traditional measure of accuracy (compared to 5/6 for the second classifier). However it performs better on the metric of NLPD: The second classifier is effectively 'confidently wrong' which is penalised heavily by this metric.

Compared to a very under-confident classifier A third classifier then just predicts 0.5 for all classes will have an NLPD in this case of 4.15: worse than either of the others.

Usage It is used extensively in probabilistic modelling research. Examples include: - Candela, Joaquin Quinonero, et al. "Propagation of uncertainty in bayesian kernel models-application to multiple-step ahead forecasting." 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP'03).. Vol. 2. IEEE, 2003. - Kersting, Kristian, et al. "Most likely heteroscedastic Gaussian process regression." Proceedings of the 24th international conference on Machine learning. 2007. - See also https://onlinelibrary.wiley.com/doi/pdfdirect/10.1111/coin.12411 for a background of other approaches (confusingly the definition in that reference says that the NLPD is what most others refer to as the *average* NLPD). I.e. 1 N NLPD {\displaystyle {\frac {1}{N}}{\text{NLPD}}} . - Heinonen, Markus, et al. "Non-stationary gaussian process regression with hamiltonian monte carlo." Artificial Intelligence and Statistics. PMLR, 2016.

Worked examples

Example 1 — a first encounter with Negative log predictive density

Start with the simplest possible case. Write down what Negative log predictive density claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In mathematics, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to Negative log predictive density before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about Negative log predictive density ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of Negative log predictive density

In research: Negative log predictive density appears in mathematics research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses Negative log predictive density in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: Negative log predictive density is common in secondary-school and first-year university syllabi. It links to neighbouring topics Statistical theory, so understanding it makes those chapters shorter.
In everyday life: Look for Negative log predictive density outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

Affiliate

Preply — study more efficiently by working with a personal tutor. 50% off.

How to study Negative log predictive density in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what Negative log predictive density means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain Negative log predictive density out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is Negative log predictive density in simple terms?

In statistics, the negative log predictive density (NLPD) is a measure of error between a model's predictions and associated true values. A smaller value is better.

Why does Negative log predictive density matter?

Because it connects several mathematics ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study Negative log predictive density?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on Negative log predictive density.