14:35Machine-translated from Azerbaijani by AIRead in Azerbaijani
Mozilla Report: Open AI Models Trail Closed Models by 4.4 Months
“Mozilla” released version 1.1 of its report on the state of open-source artificial intelligence on September 15. The document covers data collected through September 1. According to the “Artificial Analysis Intelligence Index,” the best open model trailed the leading closed model by just 3 points. Its usage cost was 60% of the leading model’s price. Compared with “Claude Fable 5,” the score gap narrowed to 2 points, while the cost was just 30% of its price.
According to the calculation, the gap between open and closed models is approximately 4.4 months. In the report, an “open-weight model” means a system whose core parameters can be downloaded. The training data and software code of such models do not necessarily have to be open. The document lists 16 notable open models. However, none of them disclose all the information required under the “Open Source Initiative” organization’s definition. The report is based on a survey of approximately 1,400 programmers, “OpenRouter” traffic data, and independent tests. “METR” evaluates a model based on whether it can complete work that takes a human several hours with a success rate of at least 50%. Closed models can perform tasks that take an expert 8–12 hours. Open models reach that level approximately 4 months later. The capabilities of open models double every 3.9 months. For closed models, this growth occurs every 5.5 months.
In the “Terminal-Bench 2.1” test, all models were evaluated using the same tools and software environment. “Z.ai’s” “GLM-5.2” model trailed “Claude Opus 4.7” by 1 point. The gap with “Claude Opus 4.8” was approximately 4 points. Even so, the cost of each test did not reach even 20% of the competing model’s cost. In August, 8 of the 10 most-used models on “OpenRouter” were open. Of those, 7 were developed in China.
Prices were compared based on model use through the software interface—API—and official rates. The best open model that fits on 1 server scored 52.6 points, while a model running on 1 graphics processor scored 40 points. They trailed the leader by 10 and 23 points, respectively. “Kimi K3” occupies approximately 1.56 TB of memory and is divided into 96 parts. At least 8 “GB300” graphics processors are required to run the model. Therefore, although its parameters can be downloaded, it cannot be run on ordinary hardware. “Thinking Machines’” “Inkling-Small” model is considered a more accessible example. Its NVFP4 version runs on 1 “B300” accelerator with 180 GB of memory. Because the report’s data was finalized on September 1, it cannot be directly compared with subsequent tests. A US security warning dated September 8 alleged that “Moonshot” trained its “K3” model using responses from “Claude Fable 5.” However, no evidence supporting this allegation was presented.
According to the calculation, the gap between open and closed models is approximately 4.4 months. In the report, an “open-weight model” means a system whose core parameters can be downloaded. The training data and software code of such models do not necessarily have to be open. The document lists 16 notable open models. However, none of them disclose all the information required under the “Open Source Initiative” organization’s definition. The report is based on a survey of approximately 1,400 programmers, “OpenRouter” traffic data, and independent tests. “METR” evaluates a model based on whether it can complete work that takes a human several hours with a success rate of at least 50%. Closed models can perform tasks that take an expert 8–12 hours. Open models reach that level approximately 4 months later. The capabilities of open models double every 3.9 months. For closed models, this growth occurs every 5.5 months. In the “Terminal-Bench 2.1” test, all models were evaluated using the same tools and software environment. “Z.ai’s” “GLM-5.2” model trailed “Claude Opus 4.7” by 1 point. The gap with “Claude Opus 4.8” was approximately 4 points. Even so, the cost of each test did not reach even 20% of the competing model’s cost. In August, 8 of the 10 most-used models on “OpenRouter” were open. Of those, 7 were developed in China. Prices were compared based on model use through the software interface—API—and official rates. The best open model that fits on 1 server scored 52.6 points, while a model running on 1 graphics processor scored 40 points. They trailed the leader by 10 and 23 points, respectively. “Kimi K3” occupies approximately 1.56 TB of memory and is divided into 96 parts. At least 8 “GB300” graphics processors are required to run the model. Therefore, although its parameters can be downloaded, it cannot be run on ordinary hardware. “Thinking Machines’” “Inkling-Small” model is considered a more accessible example. Its NVFP4 version runs on 1 “B300” accelerator with 180 GB of memory. Because the report’s data was finalized on September 1, it cannot be directly compared with subsequent tests. A US security warning dated September 8 alleged that “Moonshot” trained its “K3” model using responses from “Claude Fable 5.” However, no evidence supporting this allegation was presented.