Model | Qwen2

• Related Project: Private
• Category: Paper Review
• Date: 2024-06-09

Qwen2

Not published yet.

• url: https://arxiv.org/abs/
• pdf: https://arxiv.org/pdf/
• html https://arxiv.org/html/
• blog: https://qwenlm.github.io/blog/qwen2/

Contents

• Qwen2
  • TL;DR

TL;DR

성능 표로 대체

The datasets for evaluation include:

• English Tasks: MMLU (5-shot), MMLU-Pro (5-shot), GPQA (5shot), Theorem QA (5-shot), BBH (3-shot), HellaSwag (10-shot), Winogrande (5-shot), TruthfulQA (0-shot), ARC-C (25-shot)
• Coding Tasks: EvalPlus (0-shot) (HumanEval, MBPP, HumanEval+, MBPP+), MultiPL-E (0-shot) (Python, C++, JAVA, PHP, TypeScript, C#, Bash, JavaScript)
• Math Tasks: GSM8K (4-shot), MATH (4-shot)
• Chinese Tasks: C-Eval(5-shot), CMMLU (5-shot)
• Multilingual Tasks: Multi-Exam (M3Exam 5-shot, IndoMMLU 3-shot, ruMMLU 5-shot, mMMLU 5-shot), Multi-Understanding (BELEBELE 5-shot, XCOPA 5-shot, XWinograd 5-shot, XStoryCloze 0-shot, PAWS-X 5-shot), Multi-Mathematics (MGSM 8-shot), Multi-Translation (Flores-101 5-shot)

Qwen Models

(본 블로그 각주) The numbers in orange parentheses represent the benchmark score divided by the parameter size of each model.

Qwen-7B-Instruction

Qwen2-7B

Qwen2-0.5B-Instruct & Qwen2-1.5B-Instruct

This document introduces the Qwen2 series, an advancement from Qwen1.5, featuring pretrained and instruction-tuned models in five sizes: Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B. These models exhibit enhanced performance, including state-of-the-art results in various benchmark evaluations and significant improvements in coding and mathematics. They also support extended context lengths up to 128K tokens.

Key advancements include training on data in 27 additional languages beyond English and Chinese, implementing Group Query Attention (GQA) across all model sizes for faster inference and reduced memory usage, and improved handling of code-switching in multilingual contexts. The Qwen2-72B model, in particular, outperforms its predecessor Qwen1.5-110B and other leading models in natural language understanding, knowledge acquisition, coding proficiency, and multilingual capabilities.

Safety assessments show that Qwen2-72B-Instruct is comparable to GPT-4 and superior to the Mistral-8x22B model in managing multilingual unsafe queries. The models have been open-sourced on Hugging Face and ModelScope under the Apache 2.0 license (except Qwen2-72B, which retains the Qianwen License), promoting broader application and commercial use. Future developments will include larger Qwen2 models and extensions to multimodal capabilities.

Introduction After months of effort, we are pleased to announce the evolution from Qwen1.5 to Qwen2. This time, we bring to you:

• Pretrained and instruction-tuned models of 5 sizes, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B.
• Training on data in 27 additional languages besides English and Chinese.
• State-of-the-art performance in numerous benchmark evaluations.
• Significantly improved performance in coding and mathematics.
• Extended context length support up to 128K tokens with Qwen2-7B-Instruct and Qwen2-72B-Instruct.

Model Information

• All model sizes now apply Group Query Attention (GQA) for faster speed and less memory usage. The context length capabilities of instruction-tuned models, assessed through the Needle in a Haystack task, include up to 128K tokens for Qwen2-7B-Instruct and Qwen2-72B-Instruct models.
• The Qwen2 series includes base and instruction-tuned models of 5 sizes:

Multilingual Capabilities

• Significant efforts were directed towards augmenting both the volume and quality of pretraining and instruction-tuning datasets across a diverse linguistic spectrum. We explicitly highlight the inclusion of 27 additional languages:
  • Western Europe: German, French, Spanish, Portuguese, Italian, Dutch
  • Eastern & Central Europe: Russian, Czech, Polish
  • Middle East: Arabic, Persian, Hebrew, Turkish
  • Eastern Asia: Japanese, Korean
  • South-Eastern Asia: Vietnamese, Thai, Indonesian, Malay, Lao, Burmese, Cebuano, Khmer, Tagalog
  • Southern Asia: Hindi, Bengali, Urdu
• Evaluations confirm enhanced proficiency in handling code-switching across languages.

Performance

• Comparative assessments reveal substantial enhancements in performance for large-scale models (70B+ parameters) relative to Qwen1.5. The Qwen2-72B model exhibits superior performance in natural language understanding, knowledge acquisition, coding proficiency, mathematical skills, and multilingual abilities. Notably, it surpasses the performance of Qwen1.5-110B despite having fewer parameters.
• Our post-training phase is designed to enhance the model’s intelligence, bringing it closer to human capabilities. This phase employs various automated alignment strategies and training methods to improve coding, mathematics, reasoning, instruction following, and multilingual understanding.

Highlights

• Coding & Mathematics: Significant improvements have been made in Qwen2-72B-Instruct for various programming languages and solving mathematical problems using extensive and high-quality datasets.
• Long Context Understanding: All instruction-tuned models have been trained on 32k length contexts, extrapolated to longer contexts using techniques like YARN or Dual Chunk Attention. Qwen2-72B-Instruct can handle information extraction tasks within a 128k context.
• Safety and Responsibility: Evaluation of the model’s safety reveals that Qwen2-72B-Instruct performs comparably to GPT-4 in terms of safety and significantly outperforms the Mistral-8x22B model in handling multilingual unsafe queries.
• Developing with Qwen2: All models have been released on Hugging Face and ModelScope. For detailed usage and more information, refer to the model cards and our official documentation.

Base Language Model Evaluation The evaluation of base models focuses on performance in natural language understanding, general question answering, coding, mathematics, scientific knowledge, reasoning, and multilingual capability using various datasets.

Instruction-tuned Model Evaluation We compare Qwen2 instruction-tuned models with other recent LLMs on several cross-lingual benchmarks and by human evaluation. The results demonstrate the strong multilingual capabilities of Qwen2 instruction-tuned models.