Loading...
Loading...
Loading Curriculum...
Loading Subject...
Loading Topic...
Loading Lesson...
Loading Lab...
A generative pre-trained transformer (GPT) is a type of large language model (LLM)[1][2][3] that is widely used in generative artificial intelligence chatbots.[4][5] GPTs are based on a deep learning architecture called the transformer. They are pre-trained on large datasets of unlabeled content, and able to generate novel content.[2][3]
OpenAI was the first to apply generative pre-training to the transformer architecture, introducing the GPT-1 model in 2018.[6] The company has since released many bigger GPT models. The chatbot ChatGPT, released in late 2022 (using GPT-3.5), was followed by many competitor chatbots using their own generative pre-trained transformers to generate text, such as Gemini, DeepSeek, and Claude.
During the 2010s, improved machine learning algorithms, more powerful computers, and an increase in the amount of digitized material allowed for an AI boom.[8] Separately, the concept of generative pre-training (GP) was a long-established technique in machine learning. GP is a form of self-supervised learning wherein a model is first trained on a large, unlabeled dataset (the "pre-training" step) to learn to generate data points. This pre-trained model is then adapted to a specific task using a labeled dataset (the "fine-tuning" step).[9]
The transformer architecture for deep learning is the core technology of a GPT. Developed by researchers at Google, it was introduced in the paper "Attention Is All You Need" in 2017. The transformer architecture solved many of the performance issues that were associated with older recurrent neural network (RNN) designs for natural language processing (NLP). The architecture's use of an attention mechanism allows models to process entire sequences of text at once, enabling the training of much larger and more sophisticated models.[10] Since 2017, available transformer-based NLP systems have been capable of processing, mining, organizing, connecting, contrasting, and summarizing texts as well as answering questions from textual input.
GPTs are primarily used to generate text, but can be trained to generate other kinds of data. For example, GPT-4o can process and generate text, images, and audio.[7]
To improve performance on complex tasks, some GPTs, such as OpenAI o3, allocate more computation time analyzing the problem before generating an output, and are called reasoning models.
In 2025, GPT-5 was released with an intelligent router that automatically selects whether to use a faster model or a slower reasoning model based on the provided task.
Click on any model milestone below to explore parameters, training datasets, and architectural breakthroughs.
OpenAI introduced generative pre-training (GP) to the decoder portion of the transformer architecture. Designed for semi-supervised learning—trained on unlabeled books followed by task-specific fine-tuning.
A foundation model is an AI model trained on broad data at scale such that it can be adapted to a wide range of downstream tasks.[28][29] Major examples include OpenAI's GPT-5, Google's PaLM (a broad foundation model compared to GPT-3), Together's GPT-JT (open-source GPT-3 alternative), and Meta's LLaMA.
Standard self-attention has quadratic complexity. To mitigate this scale limit, models utilize sparse attention or memory-efficient structures. Architectures like BigBird, Reformer, and FlashAttention optimize scaling parameters to handle extremely long context sequences.
Emergent abilities refer to capabilities that appear in LLMs only when they reach a certain scale, and are absent in smaller versions of the same models. Examples include multi-step reasoning, in-context zero-shot learning, and sudden optimization jumps on complex benchmarks.[46][47]
Scaling laws show empirical relationships between model loss and parameters, dataset size, and compute. Adjust model variables below to observe the changes in predicted validation loss.
OpenAI claims "GPT" as its own branding, citing its association with ChatGPT. In 2023, they revised brand guidelines to restrict API clients from using "GPT" in names. In the U.S., the USPTO responded to OpenAI's trademark application by determining that "GPT" was both descriptive and generic.[48][53]
In the EU, the EUIPO registered "GPT" as a trademark in spring 2023, but it has since been challenged and is pending cancellation. In Switzerland, the Swiss Federal Institute of Intellectual Property registered it successfully.[62][63]
Evaluation of GPT models is carried out using standard benchmarks (such as accuracy on datasets, robustness, bias, and toxicity). Tasks typically test natural language understanding, reasoning, query answering, and code generation.[65][66]
Evaluating models remains an active area of research, as existing tests may not fully reflect real-world execution performance or the safety risks associated with large-scale generative models.[65]
LLMs can reproduce and amplify patterns present in their training data, including social biases, which may lead to unfair, biased, or discriminatory outputs.[67][68]
These models generate fluent text optimized for probability rather than verifying factual accuracy. This can lead to highly convincing misinformation in production.[69]
Training large-scale models requires massive computational infrastructures, leading to significant energy consumption and carbon footprints. Efficient training methods are highly sought after.[70][71]