A new open-source AI code generation tool, AlphaCodium, inspired by Google DeepMind's AlphaCode (as well as last month's Gemini-powered AlphaCode 2), has now surpassed it and caused a sensation on Twitter this week.
"We are one step closer to AI generating better code than humans!" Santiago Valdarrama posted. "AlphaCodium has proven to be the best code generation method we have seen. It beats DeepMind's AlphaCode and their new version AlphaCode 2 without the need for fine-tuning the model!"
Andrej Karpathy, formerly Tesla's AI director and now at OpenAI, emphasized the tool's improvement in code generation through a "process engineering" approach - "transforming from a simple prompt-answer paradigm to a 'process' paradigm, where the answer is built iteratively."
To improve LLMs' performance on specific code problems, AlphaCode's "process engineering" goes beyond prompt engineering with a chain of thought and introduces elements of GAN architecture (developed by Ian Goodfellow in 2014), including a code generation model and an adversarial model that provides code integrity through testing, reflection, and specification matching.
The process starts with an input and includes a series of preprocessing steps, where AlphaCodium reflects on the problem and arrives at the first code solution. It then generates additional tests to help refine the solution and arrive at a practically effective final solution.
AlphaCodium is developed by CodiumAI, a startup based in Tel Aviv - according to its website, the company's mission is to "enable developers to build bug-free software faster." AlphaCodium was tested on the CodeContests dataset, which consists of approximately 10,000 competitive programming problems. Its performance on the CodeContests benchmark test shows that its accuracy improved from 19% to 44% compared to GPT-4. According to CodiumAI, "this result is not just a numerical improvement; it is a leap in LLMs' code generation capability, setting a new standard in the field."
CodiumAI was founded in 2022 and raised $10.6 million in March 2023, sharing the AlphaCodium GitHub repository and a companion paper titled "Code Generation with AlphaCodium: From Prompt Engineering to Process Engineering."
Co-founder and CEO Itamar Friedman expressed surprise at the attention AlphaCodium has garnered so far but added that he knew it was a breakthrough that could help the entire developer community - he emphasized that AlphaCodium is not just a model but a system and algorithm that enables "process" communication between a code generation model and a "critic" model.
"This is a significant thing we bring - it's important to see it as a process, which is why we call it 'process engineering,'" he said. He explained that this process allows AI to generate not only boilerplate code but also effective and accurate code.
OpenAI and Google DeepMind are the biggest coding rivals
Friedman noted that he considers OpenAI (developer of Codex) and Google DeepMind (developer of AlphaCode and AlphaCode 2) as CodiumAI's biggest competitors in the coding race - but the biggest competitor is the code integrity technology itself.
"We are deeply inspired by DeepMind," he said, adding that he has also discussed the importance of code integrity with OpenAI CEO Sam Altman.
"I am in full agreement with Sam that code integrity is crucial not only for the next generation of code building but also for AI's consistency," he said. He explained that AlphaCodium actually provides the 'next generation' of code integrity - "it understands not only my specifications but also my cultural documents, beliefs, and other guidelines."
Google DeepMind included aspects of process engineering in its AlphaGo solution but not in AlphaCode, he said - "I don't know why." Perhaps, he suggested, it is because this idea is not part of the mainstream narrative that simply calls for better large language models.
"The reason why AI cannot generate effective code is not because you need a better LLM," he said. "It's because you need a process."
