Yesterday, the Orca AI team led by Arindam Mitra, a senior researcher at Microsoft Research, announced a major breakthrough on X, which is the launch of a new type of large language model called Orca-Math. This model is a new variant of Mistral's Mistral 7B model, a French startup, and it excels in "mathematical text problems" while maintaining a smaller training and inference scale. This is part of Microsoft Orca team's efforts to enhance the capabilities of small-scale large language models (LLMs). The performance of Orca Math is particularly impressive as they have achieved new heights in small-scale performance, even surpassing models with 10 times the number of parameters. Mitra's team shared a chart showing that Orca Math outperforms most AI LLMs and variants with sizes ranging from 7 to 70 billion parameters on the GSM8K benchmark test, including MetaMath (70B) and Llemma (34B), second only to Google's Gemini Ultra and OpenAI's GPT-4. GSM8K is a series released by OpenAI that includes 8,500 different mathematical text problems and questions, each requiring 2-8 steps to solve, designed by human writers and suitable for middle school level (up to eighth grade) answers. The achievement is attributed to the innovative approach of the Orca team. Mitra revealed that they generated a new list of 200,000 mathematical text problems completed by "professional agent collaboration," including student AI agents and teacher AI agents, with the latter responsible for correcting answers. These questions were generated by collecting 36,217 mathematical text problem samples from existing open-source datasets, seeking answers from OpenAI's GPT-4, and using these answers to train the new Mistral 7B variant, ultimately resulting in Orca-Math. In addition, the Orca team introduced a "suggester and editor" agent to develop more complex problems for training AI. This innovative approach allows Orca-Math to achieve performance comparable to larger models while maintaining a smaller scale. It is worth noting that the Microsoft Orca team has also released their synthesized set of 200,000 AI-generated mathematical text problems on Hugging Face, adopting a permissive MIT license that allows anyone to explore, build, innovate, and even use it for commercial purposes. This initiative will promote the utilization of this dataset by startups and enterprises, driving the development of the field of artificial intelligence. Since the initial release of Orca 13B by Microsoft in June 2023, the team has been committed to enhancing the model's capabilities. The subsequent Orca 2 versions, 13B and 7B, were released in November 2023, both based on Meta's open-source Llama 2 LLM. It can be foreseen that with the continuous advancement of technology, the Orca family will continue to grow and bring us more surprises.