16:35Machine-translated from Azerbaijani by AIRead in Azerbaijani
2.8-Trillion-Parameter Kimi K3 AI Model Run at Home Without a GPU
An enthusiast who built a server at home has run “Moonshot AI’s” 2.8-trillion-parameter “Kimi K3” artificial intelligence model without a GPU. A GPU is a processor that accelerates graphics computations. The system runs using only a 40-core 3rd-generation “Xeon Scalable” processor. The model’s data is stored in discontinued 2nd-generation “Optane PMem 200” memory. This allowed a large model designed for data centers to run using only a central processing unit.
However, the system ran quite slowly. In 1-bit mode, in which the numbers in the model are simplified, it generated 0.85 text units per second. In 3-bit mode, the speed fell to 0.75 text units per second. At a speed of about 1 unit per second, tasks proceed very slowly. For example, completing a task with 50,000 units could take 13 hours. Moreover, these results were measured without optimizing the processor and memory settings.
“Kimi K3” was not designed for home computers. The manufacturer recommends running the model on data center systems. Even the smallest community version of the model requires at least 610 GB of RAM. One way to obtain that much memory in a home environment is to buy used “Optane” modules. According to the person who built the system, that memory cost considerably less than DRAM of the same capacity.
“Intel” released this memory series in 2020 in 128 GB and 256 GB capacities. The series also includes a 512 GB version. The newer generation transfers data faster in systems with 2 processor sockets. According to “Intel,” speeds are up to 32 percent higher on average than with the previous “PMem 100” generation. The new setup uses “App Direct” mode to give software direct access to the memory. In a previous test, the 1-trillion-parameter “K2.5” model was run. With an “RTX 3060” and “Memory Mode,” the speed reached about 4 text units per second.
The setup’s main problem is high temperatures. “Optane” modules running continuously can heat up to dangerous levels and may also damage nearby DRAM. Active cooling with 2 fans rated at 12 W that blow air toward the modules is therefore considered essential. “Intel” requires airflow of 2.4-2.7 meters per second. Each module consumes 12 or 15 W of power. As a result, 4 modules together can generate 60 W of heat.
When the temperature reaches 83 °C, the modules begin to reduce their speed. At the 85 °C threshold, the system shuts down. Standard DDR4 server memory can operate at up to 85 °C. In extended mode, this limit is 95 °C. The person who built the system plans to add a GPU in the next phase. The more frequently used parts of the model will be moved to faster DRAM, after which new speed tests will be conducted.