computer science

Probabilistic programming

Probabilistic programming is a computer science topic covered in the lgStudy science library. This page brings together a partial reference excerpt, illustrations, worked examples, real-world applications and a short study plan, so you can understand Probabilistic programming rather than just read about it. In short: Probabilistic programming (PP) is a programming paradigm based on the declarative specification of probabilistic models, for which inference is performed automatically. Probabilistic programming attempts to unify probabilistic modeling and traditional general purpose programming in order to make the former easier and more widely applicable.

Key takeaways

Probabilistic programming belongs to computer science; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect Probabilistic programming to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of Probabilistic programming from memory before moving on to harder problems.

Reference excerpt

Probabilistic programming (PP) is a programming paradigm based on the declarative specification of probabilistic models, for which inference is performed automatically. Probabilistic programming attempts to unify probabilistic modeling and traditional general purpose programming in order to make the former easier and more widely applicable. It can be used to create systems that help make decisions in the face of uncertainty. Programming languages following the probabilistic programming paradigm are referred to as "probabilistic programming languages" (PPLs).

Applications Probabilistic reasoning has been used for a wide variety of tasks such as predicting stock prices, recommending movies, diagnosing computers, detecting cyber intrusions and image detection. However, until recently (partially due to limited computing power), probabilistic programming was limited in scope, and most inference algorithms had to be written manually for each task. Nevertheless, in 2015, a 50-line probabilistic computer vision program was used to generate 3D models of human faces based on 2D images of those faces. The program used inverse graphics as the basis of its inference method, and was built using the Picture package in Julia. This made possible "in 50 lines of code what used to take thousands". The Gen probabilistic programming library (also written in Julia) has been applied to vision and robotics tasks. More recently, the probabilistic programming system Turing.jl has been applied in various pharmaceutical and economics applications. Probabilistic programming in Julia has also been combined with differentiable programming by combining the Julia package Zygote.jl with Turing.jl. Probabilistic programming languages are also commonly used in Bayesian cognitive science to develop and evaluate models of cognition.

Probabilistic programming languages PPLs often extend from a basic language. For instance, Turing.jl is based on Julia, Infer.NET is based on .NET Framework, while PRISM extends from Prolog. However, some PPLs, such as WinBUGS, offer a self-contained language that maps closely to the mathematical representation of the statistical models, with no obvious origin in another programming language. The language for WinBUGS was implemented to perform Bayesian computation using Gibbs Sampling and related algorithms. Although implemented in a relatively unknown programming language (Component Pascal), this language permits Bayesian inference for a wide variety of statistical models using a flexible computational approach. The same BUGS language may be used to specify Bayesian models for inference via different computational choices ("samplers") and conventions or defaults, using a standalone program WinBUGS (or related R packages, rbugs and r2winbugs) and JAGS (Just Another Gibbs Sampler, another standalone program with related R packages including rjags, R2jags, and runjags). More recently, other languages to support Bayesian model specification and inference allow different or more efficient choices for the underlying Bayesian computation, and are accessible from the R data analysis and programming environment, e.g.: Stan, NIMBLE and NUTS. The influence of the BUGS language is evident in these later languages, which even use the same syntax for some aspects of model specification. Several PPLs are in active development, including some in beta test. Two popular tools are Stan and PyMC.

Relational A probabilistic relational programming language (PRPL) is a PPL specially designed to describe and infer with probabilistic relational models (PRMs). A PRM is usually developed with a set of algorithms for reducing, inference about and discovery of concerned distributions, which are embedded into the corresponding PRPL.

Probabilistic logic programming

Probabilistic logic programming is a programming paradigm that extends logic programming with probabilities. Most approaches to probabilistic logic programming are based on the distribution semantics, which splits a program into a set of probabilistic facts and a logic program. It defines a probability distribution on interpretations of the Herbrand universe of the program.

List of probabilistic programming languages This list summarises the variety of PPLs that are currently available, and clarifies their origins.

Difficulty Reasoning about variables as probability distributions causes difficulties for novice programmers, but these difficulties can be addressed through use of Bayesian network visualizations and graphs of variable distributions embedded within the source code editor. As many PPLs rely on the specification of priors on the variables of interest, specifying informed priors is often difficult for novices. In some cases, libraries such as PyMC provide automated methods to find the parameterization of informed priors.

See also Statistical relational learning Inductive programming Bayesian programming Plate notation

Notes

External links Foundations of Probabilistic Programming List of Probabilistic Model Mini Language Toolkits

Worked examples

Example 1 — a first encounter with Probabilistic programming

Start with the simplest possible case. Write down what Probabilistic programming claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In computer science, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to Probabilistic programming before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about Probabilistic programming ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of Probabilistic programming

In research: Probabilistic programming appears in computer science research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses Probabilistic programming in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: Probabilistic programming is common in secondary-school and first-year university syllabi. It links to neighbouring topics Probabilistic models, Probabilistic software, Programming paradigms, so understanding it makes those chapters shorter.
In everyday life: Look for Probabilistic programming outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

Affiliate

Preply — study more efficiently by working with a personal tutor. 50% off.

How to study Probabilistic programming in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what Probabilistic programming means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain Probabilistic programming out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is Probabilistic programming in simple terms?

Why does Probabilistic programming matter?

Because it connects several computer science ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study Probabilistic programming?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on Probabilistic programming.