biology

Generative pre-trained transformer

Generative pre-trained transformer is a biology topic covered in the lgStudy science library. This page brings together a partial reference excerpt, illustrations, worked examples, real-world applications and a short study plan, so you can understand Generative pre-trained transformer rather than just read about it. In short: A generative pre-trained transformer (GPT) is a type of large language model (LLM) that is widely used in generative artificial intelligence chatbots. GPTs are based on a deep learning architecture called the transformer.

Key takeaways

Generative pre-trained transformer belongs to biology; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect Generative pre-trained transformer to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of Generative pre-trained transformer from memory before moving on to harder problems.

Reference excerpt

A generative pre-trained transformer (GPT) is a type of large language model (LLM) that is widely used in generative artificial intelligence chatbots. GPTs are based on a deep learning architecture called the transformer. They are pre-trained on large datasets of unlabeled content, and able to generate novel content. OpenAI was the first to apply generative pre-training to the transformer architecture, introducing the GPT-1 model in 2018. The chatbot ChatGPT, released in late 2022 (using GPT-3.5), was followed by many competitor chatbots using their own generative pre-trained transformers to generate text, such as Gemini, DeepSeek and Claude. GPTs are primarily used to generate text, but can be trained to generate other kinds of data. Multimodal GPTs can process or generate multiple types of data, such as text, images and audio. To improve performance on complex tasks, reasoning GPTs allocate more computation time analyzing the problem before generating an output.

Background During the 2010s, improved machine learning algorithms, more powerful computers, and an increase in the amount of digitized material allowed for an AI boom. Separately, the concept of generative pre-training (GP) was a long-established technique in machine learning. GP is a form of self-supervised learning wherein a model is first trained on a large, unlabeled dataset (the "pre-training" step) to learn to generate data points. This pre-trained model is then adapted to a specific task using a labeled dataset (the "fine-tuning" step). The transformer architecture for deep learning is the core technology of a GPT. Developed by researchers at Google, it was introduced in the paper "Attention Is All You Need", which was released in 2017. The transformer architecture solved many of the performance issues that were associated with older recurrent neural network (RNN) designs for natural language processing (NLP). The architecture's use of an attention mechanism allows models to process entire sequences of text at once, enabling the training of much larger and more sophisticated models. Available transformer-based NLP systems have been capable of processing, mining, organizing, connecting, contrasting, and summarizing texts as well as answering questions from textual input.

History On June 11, 2018, OpenAI researchers and engineers published a paper called "Improving Language Understanding by Generative Pre-Training", which introduced GPT-1, the first GPT model. It was designed as a transformer-based large language model that used generative pre-training (GP) on BookCorpus, a diverse text corpus, followed by discriminative fine-tuning to focus on specific language tasks. This semi-supervised approach was seen as a breakthrough. GPT models use only the decoder part of the transformer architecture, pre-trained to predict the next token in a sequence. Previously, the best-performing neural models in natural language processing (NLP) had commonly employed supervised learning from large amounts of manually labeled data – training a large language model with this approach would have been prohibitively expensive and time-consuming. On February 14, 2019, OpenAI introduced GPT-2, a larger model that could generate coherent text. Created as a direct scale-up of its predecessor, it had both its parameter count and dataset size increased by a factor of 10. GPT-2 has 1.5 billion parameters and was trained on WebText, a 40-gigabyte dataset of 8 million web pages. Citing risks of malicious use, OpenAI opted for a "staged release", initially publishing smaller versions of the model before releasing the full 1.5-billion-parameter model in November. On February 10, 2020, Microsoft introduced its Turing Natural Language Generation, which it claimed was the "largest language model ever published at 17 billion parameters." The model outperformed all previous language models at a variety of tasks, including summarizing texts and answering questions. On May 28, 2020, OpenAI introduced GPT-3, a model with 175 billion parameters that was trained on a larger dataset compared to GPT-2. It marked a significant advancement in few-shot and zero-shot learning abilities. With few examples, it could perform various tasks that it was not explicitly trained for. Following the release of GPT-3, OpenAI started using reinforcement learning from human feedback (RLHF) to align models' behavior more closely with human preferences. This led to the development of InstructGPT, a fine-tuned version of GPT-3. OpenAI further refined InstructGPT to create ChatGPT, the flagship chatbot product of OpenAI that was launched on November 30, 2022. ChatGPT was initially based on GPT-3.5, but it was later transitioned to the GPT-4 model, which was released on March 14, 2023. GPT-4 was also integrated into parts of several applications, including Microsoft Copilot, GitHub Copilot, Snapchat, Khan Academy, and Duolingo. The immense popularity of ChatGPT spurred widespread development of competing GPT-based systems from other organizations. EleutherAI released a series of open-weight models, including GPT-J in 2021. Other major technology companies later developed their own GPT models, such as Google's PaLM and Gemini as well as Meta AI's Llama. Many subsequent GPT models have been trained to be multimodal (able to process or to generate multiple types of data). For example, GPT-4o can both process and generate text, images, and audio. Additionally, GPT models like o3 and DeepSeek R1 have been trained with reinforcement learning to generate multi-step chain-of-thought reasoning before producing a final answer, which helps to solve complex problems in domains such as mathematics.

Foundation models A foundation model is an AI model trained on broad data at scale such that it can be adapted to a wide range of downstream tasks. Foundational GPTs can also employ modalities other than text, for input and/or output. Regarding multimodal output, some generative transformer-based models are used for text-to-image technologies such as diffusion and parallel decoding. Such kinds of models can serve as visual foundation models (VFMs) for developing downstream systems that can work with images.

… excerpt ends here. Continue reading the full article.

Illustrations

Worked examples

Example 1 — a first encounter with Generative pre-trained transformer

Start with the simplest possible case. Write down what Generative pre-trained transformer claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In biology, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to Generative pre-trained transformer before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about Generative pre-trained transformer ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of Generative pre-trained transformer

In research: Generative pre-trained transformer appears in biology research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses Generative pre-trained transformer in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: Generative pre-trained transformer is common in secondary-school and first-year university syllabi. It links to neighbouring topics 2018 in artificial intelligence, Artificial neural networks, Generative AI, so understanding it makes those chapters shorter.
In everyday life: Look for Generative pre-trained transformer outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

Ask Teacher Smith questions about this articleOpens your AI tutor with a question about “Generative pre-trained transformer” →

Affiliate

Preply — study more efficiently by working with a personal tutor. 50% off.

How to study Generative pre-trained transformer in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what Generative pre-trained transformer means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain Generative pre-trained transformer out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is Generative pre-trained transformer in simple terms?

Why does Generative pre-trained transformer matter?

Because it connects several biology ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study Generative pre-trained transformer?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on Generative pre-trained transformer.