Small language models (SLM) or compact language models are artificial intelligence language models designed for human natural language processing including language and text generation. Small language models typically range have less than forty billion parameters. This make them feasible to train or host entirely on consumer electronics such as personal computers, laptops, or smart devices. A large language model typically contains hundreds of billions of training parameters, with some models exceeding a trillion parameters. This enables the model to encode vast amounts of information, improving the generalizability and accuracy of its outputs. However, training and hosting such models demands enormous computational resources, rendering it infeasible for an individual to do so using a single computer and graphics processing unit. Most contemporary small language models use the same architecture as a large language model, but with a smaller parameter count and sometimes lower arithmetic precision. Similar to LLMs, SLMs are optimized via knowledge distillation, pruning, and quantization.
Models Notable families with some models under ten billion parameters include:
Qwen by Alibaba Cloud Gemma by Google DeepMind Granite by IBM Nemotron by Nvidia Muse Glimmer by Meta Superintelligence Labs Mistral Small by Mistral AI SmolLM by Hugging Face family of SMLs, including SmolLM2 models with 135 million, 360 million, and 1.7 billion parameters.
Language model with small pre-training dataset Traditional AI language systems need enormous computers and vast amounts of data. Pre-training matters, even tiny models show significant performance improvements when pre-trained performance increases with larger pre-training datasets. Classification accuracy improves when pre-training and test datasets share similar tokens. Shallow architectures can replicate deep model performance through collaborative learning.
See also Edge computing
References