Cohere releases multilingual AI model Aya 23, supporting 23 languages

2024-05-24

Cohere's non-profit research lab, Cohere for AI, has officially launched Aya 23, an upgraded version of their large-scale language model (LLM). This cutting-edge LLM is released in the form of open weights for both 8B and 35B, supporting up to 23 languages and surpassing its predecessor, Aya 101, in performance. Aya 23 cleverly combines the high-performance pre-training models of the Command family with the newly released Aya collection, creating a powerful multilingual model. This model not only provides state-of-the-art capabilities to nearly half of the world's population but also emphasizes depth by allocating more capacity to fewer languages during the pre-training process. Aya 23 has demonstrated outstanding performance in various discriminative and generative tasks, surpassing widely used models such as Gemma, Mistral, and Mixtral. It is particularly noteworthy that the 8B version achieves the best multilingual performance among similar products, enabling researchers using consumer-grade hardware to easily access it. Under the CC-BY-NC license, Cohere for AI has released the open weights for the 8B and 35B models. This initiative reflects their commitment to advancing the field of multilingual AI and expanding access. The languages supported by Aya 23 include Arabic, Chinese (simplified and traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese.