Today’s most advanced AI models, such as OpenAI’s ChatGPT and Anthropic’s Claude, are controlled by their creators in order to prevent them from being used in harmful ways. This stands in contrast to ‘open’ models, which can be freely downloaded, modified, and used by anyone for various purposes. A recent report by Epoch AI found that open models currently lag behind closed models by about a year in terms of performance. The lead researcher on the report, Ben Cottier, explains that while the best open model matches closed models in performance, there is still a noticeable delay in advancements.
Meta’s Llama 3.1 405B, an open model released in July, took approximately 16 months to reach the capabilities of the first version of GPT-4. If Meta’s upcoming AI model, Llama 4, is released as an open model as expected, this gap between open and closed models may shrink even further. These findings come at a critical time as policymakers are facing the challenge of regulating increasingly powerful AI systems that have the potential to impact information environments and pose risks such as engineering pandemics, cyberattacks, and other harmful actions.
Epoch AI researchers conducted an analysis of hundreds of AI models released since 2018 to determine the performance gap between open and closed models. They evaluated the models based on technical benchmarks, which assess an AI’s ability to handle tasks such as solving math problems, answering general knowledge questions, and demonstrating logical reasoning. The researchers also considered the amount of computing power used to train the models, as this has traditionally been a reliable indicator of capabilities. However, open models have shown the potential to perform as well as closed models while using less compute due to advancements in AI algorithms.
The time lag between open and closed AI models presents an opportunity for policymakers and AI labs to evaluate the latest advancements in the field. By understanding the differences in performance and capabilities between open and closed models, stakeholders can make more informed decisions about how to regulate and utilize AI technology. As AI continues to evolve and play a significant role in various industries, it is crucial to address concerns about the potential risks associated with powerful AI systems and ensure that they are used responsibly and ethically.
In conclusion, the report by Epoch AI highlights the current disparity in performance between open and closed AI models, with open models trailing by approximately a year. As AI technology advances, policymakers and researchers must consider the implications of these differences and work towards developing strategies to address potential risks associated with powerful AI systems. By understanding the capabilities of AI models and the impact they can have on society, stakeholders can make informed decisions to promote the responsible use of AI technology.
