computer science

PaLM

PaLM is a computer science topic covered in the lgStudy science library. This page brings together a partial reference excerpt, illustrations, worked examples, real-world applications and a short study plan, so you can understand PaLM rather than just read about it. In short: PaLM (Pathways Language Model) is a 540 billion-parameter dense decoder-only transformer-based large language model (LLM) developed by Google AI. Researchers also trained smaller versions of PaLM (with 8 and 62 billion parameters) to test the effects of model scale.

Key takeaways

PaLM belongs to computer science; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect PaLM to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of PaLM from memory before moving on to harder problems.

Reference excerpt

PaLM (Pathways Language Model) is a 540 billion-parameter dense decoder-only transformer-based large language model (LLM) developed by Google AI. Researchers also trained smaller versions of PaLM (with 8 and 62 billion parameters) to test the effects of model scale.

Model PaLM is capable of a wide range of tasks, including commonsense reasoning, arithmetic reasoning, joke explanation, code generation, and translation. When combined with chain-of-thought prompting, PaLM achieved significantly better performance on datasets requiring multi-step reasoning, such as word problems and logic-based questions. The model was first announced in April 2022 and remained private until March 2023, when Google launched an API for PaLM and several other technologies. The API was initially available to a limited number of developers who joined a waitlist before it was released to the public. Google and DeepMind developed a version of PaLM 540B (with 540 billion parameters) called Med-PaLM, which is fine-tuned on medical data and outperforms previous models on medical question-answering benchmarks. Med-PaLM was the first to obtain a passing score on U.S. medical licensing questions, and in addition to answering both multiple choice and open-ended questions accurately, it provides reasoning and is able to evaluate its own responses. Google also extended PaLM using a vision transformer to create PaLM-E, a vision-language model that can be used for robotic manipulation without the need for retraining or fine-tuning. In May 2023, Google announced PaLM 2 at the annual Google I/O keynote. PaLM 2 is reported to be a 340 billion-parameter model trained on 3.6 trillion tokens. In June 2023, Google announced AudioPaLM for speech-to-speech translation, which uses the PaLM-2 architecture and initialization.

Training PaLM is pre-trained on a high-quality corpus of 780 billion tokens that comprise various natural language tasks and use cases. This dataset includes filtered webpages, books, Wikipedia articles, news articles, source code obtained from open source repositories on GitHub, and social media conversations. It is based on the dataset used to train Google's LaMDA model. The social media conversation portion of the dataset makes up 50% of the corpus, which aids the model in its conversational capabilities. PaLM 540B was trained over two TPU v4 Pods with 3,072 TPU v4 chips in each Pod attached to 768 hosts, connected using a combination of model and data parallelism, which was the largest TPU configuration. This allowed for efficient training at scale, using 6,144 chips, and marked a record for the highest training efficiency achieved for LLMs at this scale: a hardware FLOPs utilization of 57.8%.

See also LaMDA, PaLM's predecessor Gemini, PaLM's successor Chinchilla

References

Illustrations

Worked examples

Example 1 — a first encounter with PaLM

Start with the simplest possible case. Write down what PaLM claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In computer science, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to PaLM before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about PaLM ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of PaLM

In research: PaLM appears in computer science research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses PaLM in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: PaLM is common in secondary-school and first-year university syllabi. It links to neighbouring topics 2022 in artificial intelligence, 2022 software, Generative pre-trained transformers, so understanding it makes those chapters shorter.
In everyday life: Look for PaLM outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

Ask Teacher Smith questions about this articleOpens your AI tutor with a question about “PaLM” →

Affiliate

Preply — study more efficiently by working with a personal tutor. 50% off.

How to study PaLM in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what PaLM means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain PaLM out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is PaLM in simple terms?

Why does PaLM matter?

Because it connects several computer science ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study PaLM?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on PaLM.