Gemma 2: Elevating Open Language Models with Practical Efficiency

Here I am again with another language model… This time an open source one: Gemma 2, from Google!

The world of artificial intelligence has witnessed an explosion in the capabilities of Large Language Models (LLMs)… These complex systems, trained on massive datasets, have demonstrated remarkable proficiency in understanding and generating human-like text, pushing the boundaries of what was once considered the exclusive domain of human intelligence. Bbbbut!!! This progress has often come at the cost of immense computational resources, limiting accessibility for many researchers and developers. It is in this context that Gemma 2 is presented, a new generation of open-source LLMs (the first one was… a little stupid) meticulously crafted by Google to deliver cutting-edge performance within a practical size, democratizing access to powerful AI tools.

Do you want to know more about it? If yes, for you:

1. Gemma 2: A Family of Models
2. Architectural Innovations: The Building Blocks of Efficiency
3. Knowledge Distillation: Learning from a Wise Teacher
4. Performance Benchmarks: Setting New Standards
5. Responsibility and Safety: The Ethical Imperative
6. Conclusion

Gemma 2: A Family of Models

Gemma 2 isn’t just a single model but rather a family of models, each tailored for specific computational constraints while maintaining exceptional performance. The current lineup includes 9 billion and 27 billion parameter models, with a 2 billion parameter model slated for release soon. This range provides flexibility, allowing developers to choose the model that best suits their resources and application requirements.

Architectural Innovations: The Building Blocks of Efficiency

Gemma 2 inherits its foundational decoder-only transformer architecture from its predecessor, Gemma (1). However, it incorporates key architectural refinements to further enhance efficiency and performance:

1. Interleaving Local and Global Attention: Gemma 2 strategically alternates between layers of local sliding window attention and global attention. This approach strikes a delicate balance, capturing both local context and broader relationships within the text. Local attention focuses on a fixed window of tokens, reducing computational overhead, while global attention considers all tokens in the sequence, preserving the model’s ability to discern long-range dependencies. Let’s take an example. Consider the sentence “The cat sat on the mat, but it was thinking about the delicious fish it had for dinner”. Local attention might focus on understanding the relationship between “cat” and “sat”, while global attention would connect “it” to both “cat” and “fish”, capturing the pronoun’s reference.
2. Grouped-Query Attention (GQA): GQA, a novel attention mechanism, further reduces computational demands without sacrificing accuracy. Instead of computing attention scores between all possible word pairs, GQA divides queries and keys into smaller groups, calculating attention within these groups. This smart partitioning significantly speeds up processing, particularly for longer sequences.
3. Logit Soft-Capping: To enhance training stability, Gemma 2 employs logit soft-capping. This technique prevents extreme values within the model’s internal representations, leading to more robust and predictable behavior.
4. Post-norm and Pre-norm with RMSNorm: Gemma 2 utilizes RMSNorm (Root Mean Square Layer Normalization) for normalizing the inputs and outputs of each layer. This choice contributes to smoother training and improved model convergence.

A recap of all the features of Gemma 2 is the following:

Feature	Description
Attention Mechanism	Interleaving Local-Global Attention, Grouped-Query Attention (GQA)
Normalization	Post-norm and Pre-norm with RMSNorm
Non-Linearity	GeGLU
Positional Embeddings	Rotary Position Embeddings (RoPE)
Context Length	8192 tokens
Vocabulary Size	256,128
Parameters (2B / 9B / 27B)	2.6 Billion / 9 Billion / 27 Billion
Training Objective (2B / 9B)	Knowledge Distillation
Training Objective (27B)	Next Token Prediction

Knowledge Distillation: Learning from a Wise Teacher

Training LLMs traditionally relies on next-token prediction, where the model learns to predict the next word in a sequence. While effective, this approach can be data-hungry, requiring massive datasets for optimal performance. Gemma 2, particularly the 2B and 9B models, employs a more sophisticated technique called knowledge distillation.

Imagine a student learning from a wise teacher. Instead of merely mimicking the teacher’s actions, the student gains deeper insights by understanding the teacher’s reasoning and thought process. Similarly, in knowledge distillation, a smaller “student” model learns from a larger, pre-trained “teacher” model. The student model is trained not just to predict the next token but to approximate the probability distribution over all possible next tokens, as predicted by the teacher.

This approach offers significant advantages:

• Data Efficiency: The student model benefits from the teacher’s knowledge acquired from a much larger dataset, achieving comparable performance with less training data.
• Faster Training: Distillation accelerates training by providing richer gradients, guiding the student model toward a more optimal solution space.
• Enhanced Performance: By learning from a more powerful model, the student model often surpasses the performance achievable through traditional next-token prediction training on the same dataset.

Performance Benchmarks: Setting New Standards

Gemma 2 shines across a battery of rigorous benchmarks, consistently outperforming other open models of similar sizes and even challenging models significantly larger in scale.

Benchmark	Metric	Gemma 2 (9B)	Gemma 2 (27B)
MMLU	5-shot	71.3	75.2
GSM8K	5-shot	68.6	74.0
HellaSwag	10-shot	81.9	86.4
HumanEval	pass@1	40.2	51.8
MBPP	3-shot	52.4	62.6

It is certainly not comparable with the current SoTA models, GPT-4o and Claude 3.5 Sonnet, able to reach impressive values, like, respectively, 90.2% and 92.0% in HumanEval, the benchmark for python coding, or 96.4% (with 0-shot, not 3-shot!!!) by 3.5 Sonnet in GSM8K, the benchmark for grade school math.

However, it’s important to remember that Gemma 2’s strengths lie in its accessibility and practicality. Its smaller model sizes make it deployable on a wider range of hardware, potentially benefiting developers and researchers with limited resources.

Responsibility and Safety: The Ethical Imperative

The potential impact of LLMs necessitates a strong commitment to responsible development and deployment. Gemma 2 is built with safety and ethical considerations at its core.

• Safety Policies and Training-Time Mitigations: Gemma 2’s training data undergoes rigorous filtering to remove harmful and biased content. Additionally, the models are fine-tuned with safety policies to further minimize the risk of generating inappropriate or harmful outputs.
• Robust and Transparent Evaluations: Gemma 2 is rigorously evaluated using both automated benchmarks and human evaluations to assess its capabilities and potential risks. These evaluations cover a wide range of aspects, including safety, fairness, bias, and robustness.
• Responsible Generative AI Toolkit: To empower developers, Google provides a comprehensive Responsible Generative AI Toolkit. This toolkit offers resources, tools, and best practices to ensure the safe and responsible deployment of Gemma 2 models.

Conclusion

Gemma 2 is a nice breeze in open LLM technology, providing a compelling combination of cutting-edge performance, practical efficiency, and a steadfast commitment to responsible AI: by democratizing access to these powerful tools, Gemma 2 has the potential to fuel a new wave of innovation across a wide range of fields, from research and education to content creation and beyond. To use it, as usual, the models’ weights are available on Hugging Face!