science

Test and learn

Test and learn is a science topic covered in the lgStudy science library. This page brings together a partial reference excerpt, illustrations, worked examples, real-world applications and a short study plan, so you can understand Test and learn rather than just read about it. In short: Test and learn is a set of practices followed by retailers, banks and other consumer-focused companies to test ideas in a small number of locations or customers to predict impact. The process is often designed to answer three questions about any tested program before rollout: What impact will the program have on key performance indicators if executed across the network or customer base?

Key takeaways

Test and learn belongs to science; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect Test and learn to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of Test and learn from memory before moving on to harder problems.

Reference excerpt

Test and learn is a set of practices followed by retailers, banks and other consumer-focused companies to test ideas in a small number of locations or customers to predict impact. The process is often designed to answer three questions about any tested program before rollout:

What impact will the program have on key performance indicators if executed across the network or customer base? Will the program have a larger impact on some stores/customers than others? Which components of the idea are actually working?

Historical context Test and learn has been systematically applied as far back as 1988 by Capital One. Capital One has been aggressive about testing since the firm was founded, testing everything from product design to marketing to customer selection to collection policies. In a single year, the company performs tens of thousands of tests, allowing it to offer thousands of different types of credit cards to customers, based on the knowledge gained from their tests. Richard Fairbank, CEO of Capital One, called test and learn "a marketing revolution that can be applied to many businesses".

Examples Retail banks running different commercials in different markets. Fast food restaurants launching a new menu item or product for a limited time in select locations Big-box retailers evaluating “store of the future” programs Companies evaluating pricing strategies Retailers testing merchandising layouts across their door base The UK government, through research projects such as the College for Teaching and Learning's Close the Gap: Test and learn Amazon frequently employs the Test and Learn methodology to determine the effectiveness of different homepage layouts, product recommendation algorithms, and pricing strategies. Netflix often employs the Test and Learn methodology for its algorithms, user interface, and even content creation.

Test and learn in practice Woolworths uses a test-and-learn approach to identify which stores should undergo renovations. Wawa Food Markets, a Mid-Atlantic convenience store chain, uses Test and Learn across several aspects of its business: Measuring the effect of introducing a new product Determining the impact of adding additional staff Understanding how the opening a new location would impact existing nearby locations Subway ran a promotion that lowered the price of its foot-long sandwiches to $5. Before deciding to launch the promotion nationally, Subway ran a test that compared locations offering the $5 foot-long sandwiches to a group of similar control restaurants that were not offering the promotion. Kraft Foods, a manufacturer and distributor of packaged foods, uses testing to optimize the placement of its products on grocery store shelves: Determining the effect of a coffee aisle re-design on sales of premium coffees Measuring the impact on product sales of a change to in-store sales representatives' responsibilities Pier 1 Imports, a recently bankrupt retail chain specializing in home decor and furnishings, uses testing to measure the effectiveness of search advertising campaigns. The UK Government commissioned eight test and learn trials in 2023 to 'improve our evidence-base and understanding of what works to end rough sleeping', which are being run by the Centre for Homelessness Impact. Test and learn is also methodically used online to test impact of various web page layouts on user behavior. Tools such as Google Website Analyzer provide every site with the ability to test various elements and their effect on user engagement. Pat McFadden, the UK government's Chancellor of the Duchy of Lancaster, announced on 9 December 2024 that the government would be using a "test and learn" approach to understanding and resolving public service design and delivery dilemmas.

Test and learn software There are a variety of software tools available today to support systematic testing within an organization. However, there is currently no one software package that covers all types of tests, and in some cases significant knowledge of statistics is still required for effective analysis. Many companies, including Capital One and eBay, have developed experiment management software within the company. Alternatives to internal solutions include broad statistical analysis software tools such as SAS or test and learn focused software from Applied Predictive Technologies, Trial Run and MarketDial. There are also a variety of more narrowly tailored products, focused on particular problems or fields. In web analytics, for example, firms such as Omniture, WebTrends, and Google have created specialized software.

Criticism Companies wanting to conduct Test and Learn must mobilize, track and analyze site and customer data at very granular levels. This can often be a massive and expensive undertaking. Increasingly however, an exponential increase in computing power, and corresponding decrease in its cost, have made testing more accessible. In the Harvard Business Review February 2009 article "How to Design Smart Business Experiments", Thomas Davenport discusses the Test and Learn Process, and describes how several companies have overcome these and other barriers to testing.

See also Clinical trials Scientific method Design of experiments Evidence-based management Evidence-based practices Business process reengineering A/B testing

References

Worked examples

Example 1 — a first encounter with Test and learn

Start with the simplest possible case. Write down what Test and learn claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In science, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to Test and learn before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about Test and learn ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of Test and learn

In research: Test and learn appears in science research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses Test and learn in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: Test and learn is common in secondary-school and first-year university syllabi. It links to neighbouring topics Business terms, Quality, Tests, so understanding it makes those chapters shorter.
In everyday life: Look for Test and learn outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

Ask Teacher Smith questions about this articleOpens your AI tutor with a question about “Test and learn” →

Affiliate

Preply — study more efficiently by working with a personal tutor. 50% off.

How to study Test and learn in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what Test and learn means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain Test and learn out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is Test and learn in simple terms?

Why does Test and learn matter?

Because it connects several science ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study Test and learn?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on Test and learn.