17:30Machine-translated from Azerbaijani by AIRead in Azerbaijani
8 Used Nvidia GPUs, 1 TB of RAM: Lawyer Upgrades AI Server
Nick Mermiges, a lawyer working in the U.S. state of South Carolina, has upgraded the personal artificial intelligence server he built with 8 used Nvidia V100 graphics cards. The server runs the DeepSeek V4.1 Flash model for his legal work, with custom tests and workflows. Following the upgrade, approximately 80% of computations have moved closer to the 4-bit NVFP4 level, which is more precise than 2-bit. The system supports a context of 524,000 tokens. This means the model can consider that much text at once. According to Mermiges, documents never leave the office.
Mermiges is the founder of PLI Labs and proprietarylegal.ai. PLI is not a law firm but a technology consultancy. From February through July, he developed legal software based on Claude, but tests of local models were unsuccessful. After DeepSeek V4.1 Flash was released, he was able to move the software to his personal server. Claude Code, Opus 5.5, and Fable 5.2 developed the custom kernels, tests, and model-compression solutions under his direction.
The server has 8 V100 cards, each with 32 GB of memory. Their total video memory is 256 GiB, while the model is approximately 269 GiB at its official precision. The previous 2-bit model files occupied 248.8 GB. The new package is approximately 556 GB. The model uses 15,360 expert blocks for computations. All blocks are stored on the cards in 2-bit form, while the precision of the 5,949 most frequently used blocks has been raised to the NVFP4 level. These blocks account for 38.7% of the total. Nevertheless, they handle 80-86% of the computations in each query.
Compared with Nvidia’s official version of the model, the rate at which the same next text unit is selected increased from 92.2% to 95.1%. The KL score, which indicates how much the results differ from the official version, fell from 0.196 to 0.079. The lower this score, the closer the result is to the official model. In a standard 513-question multiple-choice knowledge test, the number of differing answers fell from 79 to 8. However, the test accuracy of all 3 versions remained approximately the same. In other words, the upgrade did not significantly change the number of correct answers; it mainly brought the results closer to those of the official model.
Measurements conducted on November 5 and 12 showed that speed had decreased slightly. Code generation speed was previously 160-215 tokens per second. After the increase in precision, this figure fell to 150-180 tokens per second. A token is a small unit of text generated by the model. Depending on the query length, code output slowed by 3-30%. The decrease for legal texts was 9-23%. When 2 queries were run simultaneously, overall speed increased. The figure rose from 118.9 tokens per second to 159 tokens per second. The first response also arrived faster for 256,000-token queries. The wait time fell from 118.7 seconds to 108.4 seconds.
During the upgrade, Mermiges increased the RAM to 1 TB. An RTX 3090 card was also added to the system. It was determined that the previous malfunctions were most likely caused by a faulty power supply rather than the PCIe connection. After the power supply was replaced, all 9 long-conversation tests were completed without any problems. From now on, the V100 cards will operate only with the Nvidia R580 driver, the final release that supports them. Mermiges already has 2 DGX Spark devices operating with a context of 1 million tokens. He is also preparing a 2nd server. That system will have another 8 V100 cards and 512 GB of RAM.