16:48Machine-translated from Azerbaijani by AIRead in Azerbaijani
Home Server With Two RTX 3090s Runs 2.8-Trillion-Parameter AI Model

A home server enthusiast added two RTX 3090 graphics cards to an artificial intelligence system built around second-generation Optane memory. The system ran the 2.8-trillion-parameter Moonshot Kimi K3 model at 3.11 tokens per second. A token is a small piece of text processed by the model. The initial prompt processing speed reached 16.8 tokens per second.
The model file is 865 GB and consists of 94 parts. Of these, 45 were moved to a fast virtual disk created in RAM. These parts have a total size of 414 GB. The virtual disk was created using tmpfs, a file system that presents memory as a disk in Linux. When the model was stored only in Optane memory, its text generation speed was 2.20 tokens per second. Prompt processing speed reached 13.3 tokens per second. The RAM disk increased text generation speed by 41% and prompt processing speed by 26%.
In this setup, the graphics cards do not store the model’s entire file in their own memory. Each RTX 3090 graphics card has 24 GB of memory, giving them a total of 48 GB of video memory. This is only 5.5% of the model file. Kimi K3 uses a system known as a mixture of experts. This means that only the necessary parts of the model are activated when each token is generated. Of the model’s 896 experts, 16 are selected each time. In addition, 2 shared experts are always active. As a result, only 104 billion of the 2.8 trillion parameters are active.
If the model were stored uncompressed at 16-bit precision, it would occupy approximately 5.6 TB. The 4-bit MXFP4 version released by Moonshot is approximately 1.56 TB. The Q2_K file used in this system reduces the model’s size to 865 GB. After this compression, each parameter contains less than 3 bits of data on average.
Data belonging to the selected experts is stored on the processor side, in Optane memory and on the RAM disk. The graphics cards perform attention calculations that identify important parts of the preceding text. They also process the shared experts, the model’s non-split layers and the current conversation’s memory. Conventional RAM is slower than video memory but faster than Optane memory. Therefore, moving part of the model to RAM increased its operating speed.
To fit the 2 large graphics cards into the case, part of its metal frame was cut away and the power supply was moved elsewhere. Because the Optane memory modules become very hot, the system has 2 blower fans rated at 12 W each. Each graphics card is in the approximately 350 W power class, so cooling remains one of the main challenges. In the future, custom quantization—that is, further compression of the model data—could increase the speed. However, its effect on the quality of the responses must also be taken into account.
