Foundation model
Overview
A foundation model is a machine learning model trained on broad, diverse data at scale, designed to serve as a base for adaptation to a wide range of downstream applications. The term emerged in the mid-2020s to describe systems like large language models (LLMs) that acquire general capabilities during pretraining and can be efficiently repurposed for specific tasks without retraining from scratch.
Foundation models differ from task-specific models by their generalist architecture and training approach. Rather than being optimized for a single objective, they acquire linguistic, factual, and reasoning capabilities across many domains during unsupervised pretraining on large corpora. This broad capability set enables transfer to new domains and tasks through prompt engineering, in-context learning, chain-of-thought reasoning, or targeted retrieval-augmented generation.
The foundation model paradigm has become the dominant approach in contemporary AI because it reduces the data and compute requirements for deploying solutions to new problems. Organizations can adapt a single foundation model to multiple downstream tasks—document classification, summarization, question-answering, code generation—rather than training separate models for each application. This efficiency, combined with the emergence of embedding and tool-use capabilities, has made foundation models the substrate for most contemporary AI products.
How it works
Foundation models acquire their broad capabilities through pretraining on large, diverse corpora. During pretraining, the model learns patterns in language, factual associations, mathematical reasoning, and domain-specific knowledge through tokenized text prediction tasks. The resulting model encodes representations in its parameters that capture regularities across many domains.
Adaptation occurs through several mechanisms:
- Fine-tuning: The model's parameters are updated on task-specific labeled data. This approach is most common when labeled data exists and computational resources permit retraining.
- In-context learning: The model adapts to new tasks by observing input-output examples in its context window, without parameter updates. Zero-shot (no examples), few-shot, and many-shot variants exist depending on the number of examples provided.
- Prompt engineering: Task performance is optimized through structured prompts that guide reasoning. Chain-of-thought prompting decomposes problems into intermediate steps to improve accuracy on reasoning tasks.
- Tool use: The model integrates external tools—calculators, code interpreters, APIs, retrieval systems—to extend its capabilities beyond learned parameters.
Foundation model performance is typically documented in model cards, which specify benchmark results, context length, training data composition, and known limitations.
| Term | Distinction |
|---|---|
| Large language model | Foundation model is a broader concept that encompasses LLMs but also includes multimodal models (vision, audio) and domain-specific large models. All LLMs are foundation models, but not all foundation models are language-only. |
| Task-specific model | Foundation models are trained for broad capability acquisition; task-specific models are trained or fine-tuned for a single objective. Foundation models enable efficient transfer; task-specific models optimize for one application at the cost of flexibility. |
| Fine-tuned model | A fine-tuned model is a downstream instantiation of a foundation model adapted to a specific task. The foundation model is the base; the fine-tuned variant is the result of adaptation. |
| Generative engine | A generative engine is an application layer built on foundation models. Foundation models are the underlying components; generative engines are user-facing systems that integrate foundation models with retrieval, ranking, and UI. |
| Embedding model | Embedding models are specialized foundation models trained to produce vector representations of text, code, or images. Embedding models have a narrower output space (vectors) than general-purpose foundation models (text tokens). |
Examples
- GPT-3 and GPT-4: OpenAI's foundation models trained on large text corpora, adapted to tasks via prompting, in-context learning, and fine-tuning. GPT-4 serves as the foundation for AI Overviews integration and zero-click answer systems.
- BERT and T5: Google's encoder and encoder-decoder foundation models. BERT is adapted to classification and information retrieval tasks; T5 supports text generation tasks including summarization and source attribution.
- LLaMA: Meta's open-weight foundation model family, adapted to diverse tasks through prompting and retrieval-augmented generation, with variants optimized for inference efficiency.
See also
- Large language model
- In-context learning
- Prompt engineering
- Retrieval-augmented generation
- Model card
- Benchmark contamination
