Data Sovereignty

InferSSD: your SSD is the new VRAM

2026-11-02 • 6 min read

There's a simple problem in local AI that nobody had solved well: large models don't fit in your memory. A model like DeepSeek-V4.1 has 763 billion parameters. It takes up hundreds of gigabytes. Your laptop has 32. Your GPU has 16. The math doesn't add up.

The traditional answer: buy more RAM. Or pay a cloud subscription. Or settle for small models.

InferSSD proposes another answer: use your SSD.

The idea: SSD as extended memory

InferSSD is an open-source inference engine (Apache 2.0 license) created by WiCi AI, built on llama.cpp. Its premise is simple to understand even if the implementation is complex:

Instead of loading the entire model into RAM or VRAM, it stores the weights on the SSD and loads only what's needed, when it's needed.

But there's a catch. It doesn't work with just any model — it works specifically with Mixture-of-Experts (MoE) models. And the reason is elegant.

Why MoE makes it possible

A MoE model doesn't use all its parameters on every token. For each word it generates, the model selects a small subset of "experts" — think of it as a team of specialists where only the relevant ones are consulted for each task.

DeepSeek-V4.1 has 763B total parameters but only activates a fraction for each token. InferSSD exploits exactly this:

What changes (attention, KV cache, shared layers) stays in RAM/VRAM — it's fast and needed constantly
What doesn't change (expert weights) stays on SSD — it's hundreds of gigabytes but only a fraction is needed at any moment
A smart cache keeps frequently-used experts in RAM, avoiding re-reading from SSD

The SSD doesn't replace RAM. It complements it. It's extended memory for models that don't fit, where you only need to load the active part.

The numbers: 800B on an RTX 5090

This isn't theory. InferSSD already works. The benchmarks are surprising:

RTX 5090 (32GB RAM + 32GB VRAM): DeepSeek-V4.1 at 7.27 tokens/s decode, 164.5 tokens/s prefill. That's usable. A 763B model on a consumer graphics card.
MacBook Pro M5 Max (48GB unified): 2.7 tokens/s decode. Slow but functional — a 763B model on a laptop.
RTX 5060 Ti (8GB RAM + 16GB VRAM): 2.55 tokens/s. Yes, 8GB of RAM. The SSD does the heavy lifting.

For perspective: llama.cpp without InferSSD on the same RTX 5090 does 0.87 tokens/s. InferSSD is 8.4 times faster in decode and 20 times faster in prefill.

It's not the only one heading this direction

The idea of using SSD as memory for AI is gaining traction:

SSD-LLaMA (September 2026 paper): academic research on SSD-native inference for trillion-parameter MoE models
llama.cpp Expert-Aware SSD Streaming: active community discussion to integrate expert streaming from SSD directly into llama.cpp
SGLang SSD Expert Pack: joint work between WiCi and SGLang to optimize expert access from storage
Gandalf: study on the impact of using SSD for KV caching and drive durability

The convergence is clear: the industry is discovering that SSD isn't just storage — it's another memory layer.

What about SSD durability?

It's the question everyone asks. Won't running an AI model from SSD kill the drive?

The short answer: not as much as you'd think.

InferSSD reads static weights. It doesn't write. Model weights don't change during inference — they're read and cached in RAM. SSD wear comes mainly from: downloading weights (once at install), system logs, OS swap, and the drive's internal maintenance.

WiCi published a wear model suggesting that, even under intensive use (4 continuous hours per day), a quality SSD can last years before exhausting its write budget. Obviously, this depends on the drive, temperature, and actual workload. But the idea that InferSSD "burns SSDs" is a myth.

Why this changes the rules

If you can run an 800B model on an RTX 5090, the question "do I need the cloud?" becomes much more nuanced.

You're not going to replace GPT-5 at home. But for many applications —offline inference, local fine-tuning, data privacy, experimentation— having access to frontier models on consumer hardware is a paradigm shift.

For researchers: you can experiment with state-of-the-art models without paying API fees
For SMEs: total privacy, no sending data to third parties, with models that previously required data centers
For developers: a server with a decent GPU and a fast SSD can serve 800B models to multiple users in parallel

The bottleneck was always memory. InferSSD moves it to the SSD, and modern NVMe SSDs are surprisingly fast. "It doesn't fit on my machine" is no longer a definitive answer.

Conclusion

InferSSD isn't magic. Large models remain slow on modest hardware. 2.7 tokens/s on a MacBook won't replace a cloud API for intensive use.

But it demonstrates something important: the cloud's monopoly on large models is eroding. With affordable NVMe SSDs, increasingly powerful consumer GPUs, and engineering like InferSSD, the barrier to running frontier models locally drops every month.

Sam Altman says your PC is a waste. InferSSD says your SSD is VRAM waiting to be used. We'll go with the second option.