Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading

Putting an MoE model's expert weights and its KV cache into one shared pool of GPU memory, and shifting how much each side gets as load changes, lets a memory-limited machine serve many more concurrent requests. Rimuru is a serving system built this way. When requests pile up, it moves rarely used experts out of GPU memory to free pages for the KV cache, and when traffic drops, it brings experts back. To make this work, it rewrites the MoE kernel to read expert weights that are not stored contiguously. Existing expert-offloading systems split GPU memory into two fixed regions at startup, so the KV cache runs short as soon as requests pile up. On 5090 machines whose GPU memory just fits all the weights, Rimuru moves some experts out on purpose to admit more requests, and its throughput is 1.1 to 2.88 times that of vLLM and SGLang, which keep all weights in GPU memory.

Teams deploying LLM inference should read it, especially teams that have to serve MoE models on a few consumer GPUs or older servers. It does not build a better expert predictor. Instead, it turns "how many experts stay in GPU memory" from a setting fixed at startup into a quantity that follows the request queue at run time. The core idea in one sentence: expert weights and the KV cache share one pool of pages, and after every scheduling round the adjuster checks how many KV cache pages are missing, evicting experts from coldest to hottest when short and bringing them back from hottest to coldest when there is room. With the same model, the same RTX 5090 and the same request stream, peak concurrency rises from 74 requests under fixed-ratio offloading to 90, and the forward steps needed to finish the batch drop from about 1300 to about 1000 (these are step counts, not elapsed time).

What problem is it solving?

An illustrative scenario first, to show what the fixed memory boundary costs. A team deploys a 30B-class MoE model on two 24 GB GPUs. The weights do not fit, so, as is common, only some experts stay in GPU memory and the rest live in CPU memory and are copied over PCIe when needed. The remaining GPU memory goes to the KV cache. This boundary is fixed at startup.

The problem is not keeping too many or too few experts. It is that the boundary between the two regions is fixed while their demands rise and fall against each other. The paper measures this on one RTX 5090 serving Qwen1.5-MoE-14B with the same ShareGPT request stream (10 requests per second), comparing three designs. vLLM with all weights in GPU memory peaks at only 34 concurrent requests and needs about 2100 forward steps. Fixed-ratio expert offloading enlarges the KV cache region, reaching 74 concurrent requests and about 1300 steps. Rimuru lets the boundary move with load, reaching 90 concurrent requests and about 1000 steps. Offloading experts only frees memory. Letting the freed memory be lent back and forth between the two kinds of data saves another 300 or so forward steps.

The pipeline: what happens in one scheduling round

Rimuru is built on vLLM, with about 4400 lines of Python and 800 lines of C++, and its MoE kernel is written in Triton. The figure below shows the structure; follow one round by the numbers:

1. Requests enter the scheduler's queues.
2. The logical page manager allocates KV cache pages for the requests that will run this round; if the schedule is still short of pages, the shortfall goes to the adjuster, which evicts experts. The adjuster looks at the remaining pages and decides whether the expert cache grows, shrinks or stays put this round.
3. The schedule, the page allocations and a change list of experts to load and evict go to the GPUs together.
4. The GPU side updates its page tables, and execution uses them to find the KV cache and the expert weights.
5. The backend expert manager merges newly cached experts with experts predicted to be needed into one prefetch queue and copies them from CPU memory. The prediction is taken directly from the existing offloading system Fate and is not this paper's contribution; the paper does not say which data from which layer that predictor reads to predict experts for later layers. At execution time the actual routing decides, and any expert not prefetched is copied in before computation.

The key division of labor: the front end only decides how memory is split, and the GPU side makes sure copies and page-table updates happen at safe moments. The next three sections cover how one pool serves data of very different sizes, how the kernel computes on scattered expert weights, and how the adjuster decides when to grow or shrink.

One pool for data whose sizes differ by orders of magnitude

Why this is hard: one weight matrix of an expert is a block of several megabytes, while a KV cache page, as in vLLM, holds the K and V of only 16 tokens. Managing everything in small pages splits each expert into many pieces and makes lookups expensive. Managing everything in large pages wastes space on small KV allocations.

Rimuru handles it at two levels. The first is views (the paper's term is memory view). The whole pool is one contiguous byte range in GPU memory, and experts, the KV cache, the SSM states of Mamba layers and convolution states each get a view that reads the same bytes with its own page size and shape, using PyTorch's .view(dtype) and .view(shape) with no copying. Once an expert's memory is safely reclaimed, the KV cache view can use it right away.

The second level is three tiers of pages. The largest tier is the huge page, sized as the least common multiple of the expert page and SSM page sizes, or simply the expert page size when the model has no SSM. The middle tier holds expert pages and SSM pages. The smallest tier holds KV cache pages and convolution-state pages, whose sizes must divide the huge page. Allocation splits downward on demand. A KV request first uses free small pages that are already split, then splits a free expert page, and only after that turns a huge page into expert pages. It never cuts a huge page straight into KV pages, because the sizes are so different that huge pages would end up chopped into scattered fragments. Reclamation runs the other way. When all KV pages carved from an expert page are freed, it becomes an expert page again, and when every page under a huge page is back, the huge page returns to the pool.

A small worked example (illustrative): KV pages carved from one expert page go to three requests. When the first two requests finish, the expert page still cannot be returned. Only after the third request finishes does the whole page merge back into an expert page that can hold one weight matrix of an expert brought back from host memory (each of an expert's three matrices takes one page). So Rimuru tries to give KV pages from one expert page to the same request, so that they are freed together. In this example, the small pages freed by the first two requests can only keep serving the KV cache and cannot hold an expert, a temporary trapped space that same-request allocation is meant to shorten. Beyond that, the paper argues that only two kinds of waste remain: the unfilled part of KV pages and unused space at the end of convolution states. Each gap is smaller than one small page, and these pages are small to begin with, so the paper judges the total negligible.

Expert weights are scattered across pages. How does the kernel read them?

Why the kernel must change: vLLM's fused MoE kernel expects all expert weights packed in order in one large array. Now the experts cached in GPU memory sit at scattered places in the pool, and experts that are not cached are copied into a temporary loading buffer. Copying them into one contiguous array before each computation would add a round of on-device copies and extra staging memory, giving back much of what caching saved.

Rimuru's kernel takes extra metadata saying where each expert is: a page number for cached experts and a buffer slot for experts just copied in. The gate and up projection matrices of an expert (the paper's $W_1$ and $W_3$) each occupy one page, and the kernel picks the page by whether an output column falls in the first or second half, without physically concatenating them. The address is:
$${\mathrm{addr}}_e(n,t)=B_P+p_e(n)fd+(n\mathrm{mod}f)d+t$$
Here $B_P$ is the start of the page pool, $p_e(n)$ is the page holding output column $n$ of expert $e$, $f$ and $d$ are the intermediate and hidden sizes on this GPU, and $t$ is the offset along the hidden dimension. The intuition: the page lookup happens once per output tile, and the kernel then reads within the page with fixed strides, so the extra addressing cost is small.

Copying also has to stay out of the way of computing. Rimuru keeps two loading buffers and alternates them: while one supplies weights for this layer, the other receives the experts prefetched for the next layer, and the roles swap at the next layer. With a single buffer, the next layer's copies would have to wait until this layer finishes, or the weights would be staged elsewhere and copied again. The buffer size is bounded by expert_offload_limit: a layer that offloads at most that many experts never needs to copy in more than that many at once.

The paper measures the kernel's own cost separately. Against vLLM's original contiguous-weight kernel with identical inputs and routing, it is 0.83% slower on average. With batches of 1024 tokens or more, every configuration stays within 1.31%. With small batches it is at most 9.63% slower in relative terms (with a single token), and at most 3.38 microseconds slower in absolute time. Reading scattered pages is nearly free: the kernel itself barely slows computation.

The adjuster: when to evict an expert, and which one?

Why an adjuster is needed: a shared pool only makes lending possible; something still has to decide who lends to whom. Evicting experts too aggressively means copying them back from the CPU again and again, and keeping too many experts leaves queued requests waiting.

The adjuster decides each round right after the scheduler has placed requests, because the free pages at that moment best reflect current memory pressure. The rules are concrete. Each expert occupies 3 expert pages, one each for the two projection matrices and the down projection $W_2$.

• When the scheduler is short of $R_b$ expert pages, evict $\lceil (R_b+1)/3\rceil$ experts. If the scheduler is short by 8 pages (illustrative), the adjuster evicts $\lceil 9/3\rceil =3$, freeing 9 pages; after the shortfall is covered, the one extra page is a cushion for the next round.
• When nothing is short but fewer than 2 expert pages $F_b$ are free, evict 1 as a precaution.
• Only when more than 9 expert pages are free does it grow, adding $\min (5,\lfloor (F_b-9)/3\rfloor )$ experts per round. With 20 free pages (illustrative), it adds $\lfloor 11/3\rfloor =3$.

The shrink threshold of 2 and grow threshold of 9 leave a gap so that free memory hovering near a threshold does not cause an expert to be evicted one round and loaded back the next. The cap of 5 per round keeps the prefetch path from being flooded.

Which experts to evict or add comes from an offline ranking of hot experts: the routing probability of every expert in every layer is measured on sample data and sorted. Each layer always keeps a top slice of its ranking in GPU memory. To grow, each layer nominates the next uncached expert in its ranking and the adjuster picks those with the highest routing probability. To shrink, each layer nominates the last expert it keeps and the adjuster picks those with the lowest probability. Layers that already offload as many experts as expert_offload_limit allows are left out of shrinking.

Once a decision reaches the GPU, ordering matters. When the adjuster sends the change list, it has already reserved page numbers for each newly added expert. On the GPU side these experts join the next layer's prefetch queue, are de-duplicated against predicted experts and copied first; newly added experts are written straight into their reserved pages, and only experts needed temporarily go into the loading buffers. Before that layer runs, the manager waits for the copies to finish and only then points the page table at the new pages. Evicted experts are not freed at once; they stay usable until the current forward pass ends, and only then is the page table changed and the pages returned to the pool. So pages freed by eviction can be reassigned only after the forward pass that is currently running ends. The paper's system overview does not say in which round the requests waiting for these pages get their KV cache pages and start running. Finish copying before updating the page table, finish computing before reclaiming pages: these two rules keep computation from reading half-copied weights or pages already given to someone else.

Experiments: where it is faster and what it costs

The experiments use two machines: eight RTX 5090s (32 GB each, PCIe 5.0) and two RTX 4090Ds (24 GB each, PCIe 4.0). The models are Qwen3-30B-A3B, Qwen3.5-35B-A3B and Qwen3.5-122B-A10B, requests are sampled from ShareGPT, and the request rate is raised step by step. For fairness, CUDA Graph is turned off in every system that supports it.

The 19.03× figure needs context. The CPU in the 4090D machine lacks Intel's newer matrix-acceleration instructions, so the paper configures KTransformers to use an older generation of vector instructions to compute experts, and its performance drops sharply. On the 5090 machine, whose CPU supports these matrix instructions, the gap is at most 2.30×, and on Qwen3 KTransformers even has lower TTFT and TPOT than Rimuru. Rimuru wins by admitting more concurrent requests under heavy load, not by computing each token faster.

Against vLLM and SGLang, the large TTFT drop comes mainly from shorter queues: under memory pressure Rimuru admits more requests, so they wait less. The price is a higher TPOT than both baselines, because some experts have to be copied from the CPU. Under light load Rimuru brings experts back and its TPOT falls to roughly half of its heavy-load value, but stays above the baselines. The paper says overall request completion time is still lower.

The ablation starts from Fate and adds one part at a time (2×5090). Adding only unified memory performs about the same as the baseline, because it only makes lending possible and nothing borrows yet. Adding the adjuster raises throughput to about 3.85× the baseline and cuts TTFT by 83.9%. Switching to the scattered-page kernel lifts throughput to about 4.19× and cuts TTFT by about 97.8%. Most of the throughput gain comes from the adjuster actually moving memory, while the kernel pushes TTFT down much further, mainly in prefill, where many tokens activate nearly all experts and the old path had to copy a large batch of weights.

Where I think it falls short

First, it trades per-token speed for concurrency. Its TPOT is higher than that of vLLM and SGLang with all weights in GPU memory, and on the 5090 machine, whose CPU has matrix-acceleration instructions, running Qwen3, KTransformers beats it on both TTFT and TPOT. For single-user, low-concurrency interactive use it may not pay off; the advantage holds only when requests pile up.

Second, the cap must be tuned by hand and matters a lot. At its minimum, expert_offload_limit turns Rimuru into a fixed cache like Fate, with queued requests and very high TTFT. Allowing about ten more experts to be offloaded brings throughput near its peak. Setting it too high hurts throughput, because after a burst it takes a long time to copy the experts back. TPOT, by contrast, is largely unaffected by this cap. The paper itself says the best value depends on model, GPU memory and workload and is hard to set in advance. The thresholds of 2 and 9 and the cap of 5 per round are also hard-coded constants.

Third, the hot-expert ranking is measured offline. If live traffic differs from the profiling data (say, a new language or task mix), the top-ranked experts may not be the ones actually in use, and wrong evictions or additions mean more copying. The paper does not test distribution shift, and ShareGPT is its only workload.

How an inference deployment team could try it

1. First check whether you have the mismatch: on your current vLLM deployment, log KV cache occupancy and queued requests over a day. If the KV cache fills at peak and sits mostly empty at night, there is room for this; if the weights fit and the KV cache never fills, you do not need it.
2. Start with fixed-ratio offloading: use a system like Fate to move some cold experts to CPU memory, and measure throughput, TTFT and TPOT at several offload ratios to find the knee for your load. This is the baseline of Rimuru's ablation, and it also shows how much PCIe copying raises TPOT.
3. Then add dynamic adjustment: build the hot-expert ranking from your own real requests, sweep the offload cap from small to large, and watch how long throughput takes to recover after a burst ends. The paper finds that allowing about ten more offloaded experts works best, but measure it again on your own model and GPUs.
4. Judge success by the request rate you can sustain under the same latency requirement, not only by peak throughput. If the product has a hard TPOT limit, put it in the load test and see how much of Rimuru's advantage remains below that line.

Verdict

The main takeaway: when GPU memory is short, the problem is often not the total amount but a fixed split between experts and the KV cache. Rimuru makes that split something adjusted at run time by queue pressure, and it builds the paging and the kernel needed to make it work.