Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads at LinkedIn

Ankit Goyal, Santosh Jha, Zhuo Zhi, Youmin Han, Cong Gu, Chen Xie, Animesh Singh
55 minread4.45scoreNo. 211-11 Infra issue
Contents · 7 sections1The five steps from submitting a job to running it2Why one pod can take jobs with different code3Three isolation rules for reusing pods4Zero-VRAM standby: no twenty-second wait when a pod must change5The GPU cost and limits of warm pods6In benchmarks, reuse made loops 2.2 to 13 times faster7Warm pods do not pay when jobs are long or pauses too long
Rel 4Nov 4Prac 5Evid 5Time 5

LinkedIn's Firepool, which hands out GPUs to internal agent and RL jobs, shows that few-second GPU jobs get faster not by starting pods faster but by landing on pods already running, without paying for idle GPUs or leaking one job's leftover files and processes to the next.

A coding agent works by editing a file, running tests in a sandbox, reading the result and editing again. RL training is similar: the trainer sends out a batch of rollouts, scores them, updates the policy and sends the next batch. Every test run or rollout needs an isolated environment and a GPU, which on Kubernetes means starting a pod. These loops used to be driven by people, who read a result before acting, so steps were minutes apart. Now that programs drive them, steps are seconds apart, jobs arrive in batches, and the load never stops. The rest of this piece follows one illustrative scenario: a coding agent (call it agent-A) fixing bug 42, which runs 4 steps in a GPU sandbox, each taking 2 seconds, with a 1-second pause between steps while it waits for the LLM to return the next action.

The trouble is that starting a pod for 2 seconds of work takes more than twenty seconds. On LinkedIn's H200 nodes, even with the 12.2 GB image already on the node, the median time from pod creation to the first instruction is 22.1 seconds, and p90 is 34.7 seconds. The largest share is the platform init containers that every production pod runs: 14 seconds at the median, 5 of them just to fetch credentials. Applied to agent-A (illustrative): one pod per step means 4 × 22 = 88 seconds waiting for pods, 4 × 2 = 8 seconds of work and 3 pauses totaling 3 seconds, 99 seconds in all, with less than a tenth spent running the task. Without the image on the node it is slower still: over a month, the fleet-wide average pod-creation time was 1.3 to 7.5 minutes, depending on the GPU shape.

The usual fixes each solve only part of this problem. Pre-caching images on nodes saves the network pull, but the 22 seconds above were measured with the image already cached, so agent-A saves nothing. Batch schedulers such as Kueue and Volcano manage queues and quotas, but leave no reusable pod between jobs, so agent-A still starts one pod per step. Existing warm-pool controllers (for example SageMaker Managed Warm Pools) prepare pods in advance, but only serve requests that match the prepared configuration; if, as we assume in the illustration, agent-A's environment does not match, it still needs a cold start (building a pod from scratch). A Ray cluster or a static worker pool removes the wait for all 4 steps, at the cost of holding GPUs through every pause of agent-A and through nights with no work. On such long-lived workers, a job's leftover files and shared memory can also be reached by the next job.

So keeping a started pod for the next job (a warm pod, below) only pays off if three things hold at once: the next job can actually use the pod, the GPU cost of holding pods is bounded, and reuse does not leak the previous job's files and processes to another caller (the agent or trainer that submits jobs). Firepool is a broker service inside LinkedIn that provides pods for coding agents, RL training and other agentic research workloads, and this paper describes how it handles each of the three. In a sampled 14-hour production peak, 94.7% of 3,206 pod requests were served by pods that were already running, with no new pod created. This shows that, at least during this peak, creating a pod had become the exception.

The five steps from submitting a job to running it

Firepool runs one broker service in each Kubernetes cluster. The request agent-A submits for step 1 looks roughly like this (illustrative, fields abbreviated from the paper's description):

sku:       h200-1gpu
package:   sha256:9f3c…   # scripts and verifier for the bug fix, a 4 MB zip
lease_tag: fix-bug-42

sku names the hardware pool, which fixes the GPU model, CPU, memory and base image; here it is the "one H200" tier. package is the code package. lease_tag is a label the caller gives this session, called the session tag below. The caller's identity is not filled in by the request; the broker verifies it from an identity token.

The request then goes through these steps:

StepInputWho does whatOutputDoes the job wait for it
1Job requestThe broker verifies identity and writes the job to durable SQLA queued recordYes
2Queued record, in-memory pod tableThe dispatch loop takes the job and claims a pod in the pod tableagent-A leases pod-7 (an illustrative pod name)Yes, the lookup is very fast
3Code packageThe daemon in the pod unpacks the code, installs dependencies as needed and starts the jobThe job process runsYes, about 0.4 s at the median
4Finished jobThe daemon kills the job process and empties the working directory; once it confirms the pod is clean, the pod is kept for agent-Apod-7, reusable for the next stepThe next step waits for the cleanup
5Queue length and GPU budgetThe scaling controller creates or reclaims pods in the backgroundWarm pods for later jobsNo, it runs in the background

The pod table in step 2 (the paper calls it the tracker) records each pod's name, hardware pool, caller and job, all behind one read/write lock. Finding a pod, validating the state change, checking the caller's limits and recording the lease all happen while holding the lock once. The dispatch loop's workers queue for this single lock, so by design no two workers claim the same pod. With 100,000 pods, the median lock hold is only 1.5 to 2.6 microseconds. Dispatch throughput is therefore bounded not by the lock but by the durable commit that writes the job's state to SQL after the claim.

The flow table above leaves three questions open: why step 3 can accept any job, who the pod in step 4 can be kept for, and how many pods step 5 should keep. The next three sections take them in turn.

Why one pod can take jobs with different code

A warm pod is only worth something if the next job can use it. If each image gets its own pool, warm pods are fragmented, because a job that needs image A can only run on an image-A pod. Illustration: with a budget of 8 GPUs, 4 images each keep 2 warm pods. Then 3 jobs needing image A arrive; only 2 can start right away and the third waits for a cold start, while the 6 pods for images B, C and D sit idle and cannot help.

Firepool's approach is called late binding: decide as little as possible when a pod is created, and fix the code, dependencies and the role of each pod in a multi-pod job only when a job is assigned to the pod. Each hardware pool uses a single base image holding the heavy, rarely changing layers such as frameworks, so the cluster only needs to cache a few base images per GPU generation.

agent-A's code package is a zip named by its content hash, unpacked into the pod's working directory when the job starts; unpacking a package of a few MB takes about 0.4 seconds at the median. Python dependencies that are not in the base image are installed the first time this caller uses this pod and then stay, so the next step that returns to the same pod does not reinstall them.

Hoot saysOne warm pod takes a second job: what changes, what staysClose ✕
The code package is delivered fresh for every job; dependencies stay on the pod, added or upgraded, never uninstalled.
Four boxes: the base image is in place; the dependency layer, code directory and run topology are empty.base imagenative buildsframework pinsdependenciesno extras yetcode directoryemptyrun topologyno rank
Four boxes: the base image is in place; the dependency layer, code directory and run topology are empty.base imagenative buildsframework pinsdependenciesno extras yetcode directoryemptyrun topologyno rank
1/4 · The pod has just been createdThe pod boots as a generic worker carrying only its SKU's shared base image, so any job on this pool can use it.
The base image is unchanged; the dependency layer gains pkg-x v1 and pkg-y v1; the code directory holds P1; the run topology has rank variables.base imageunchangeddependenciespkg-x v1pkg-y v1marker: list Acode directorypackage P1run topologyrank variablesmulti-pod only
The base image is unchanged; the dependency layer gains pkg-x v1 and pkg-y v1; the code directory holds P1; the run topology has rank variables.base imageunchangeddependenciespkg-x v1pkg-y v1marker: list Acode directorypackage P1run topologyrank variablesmulti-pod only
2/4 · The first job startsIllustrative: the job brings code package P1 and dependency list A (pkg-x v1, pkg-y v1). The dispatcher claims the pod and the code is unpacked into the workspace. This is the caller's first job on this pod, so installing dependencies takes extra time, and a completion marker for list A is written afterwards. For a multi-pod job, the executor injects the rank variables now.
The base image is unchanged; the dependency layer still holds pkg-x v1 and pkg-y v1; the code directory is emptied; the run topology is stripped.base imageunchangeddependenciespkg-x v1pkg-y v1keptcode directoryemptiedrun topologystripped
The base image is unchanged; the dependency layer still holds pkg-x v1 and pkg-y v1; the code directory is emptied; the run topology is stripped.base imageunchangeddependenciespkg-x v1pkg-y v1keptcode directoryemptiedrun topologystripped
3/4 · The job ends: resetThe in-pod agent kills the process tree and verifies it is gone, then empties the code directory and /dev/shm, and the reset also strips any ephemeral multi-pod cluster state. Dependencies and the marker stay put, and the pod remains reserved for this caller.
The base image is unchanged; the dependency layer now has pkg-x v2 with pkg-y v1 still present; the code directory holds P2; the run topology is injected for this job.base imageunchangeddependenciespkg-x v2pkg-y v1marker: list Bcode directorypackage P2run topologyinjected anewmulti-pod only
The base image is unchanged; the dependency layer now has pkg-x v2 with pkg-y v1 still present; the code directory holds P2; the run topology is injected for this job.base imageunchangeddependenciespkg-x v2pkg-y v1marker: list Bcode directorypackage P2run topologyinjected anewmulti-pod only
4/4 · The second job changes dependenciesIllustrative: the same caller's next job in the same session brings code package P2 and list B (pkg-x v2, pkg-y dropped). The list does not match, so the old marker is removed and the install runs: pkg-x is upgraded in place, and pkg-y is not uninstalled and stays until the pod retires. If the list had not changed, the marker would match and this step would cost 0.
1/4

Red changed in this step, dark is kept, gray is empty; package names, versions and jobs are illustrative. A pod remembers only the last list installed, so returning to list A reinstalls it. A caller who needs a clean environment asks for a single-use pod; because reuse is owner-only, the residue does not reach other callers. The paper assigns native builds and framework pins to the base image, where incompatibilities surface once at build time.

Sourcepaper §4 (code and dependencies arrive with the job, topology arrives with the job, where the boundary falls, costs and limits); §5 rule 3 (reset between jobs); App. A.2 (incremental dependencies and the completion marker); App. C.5 (dependency install times)

The roles in a multi-pod job are also left until assignment. Kubernetes training operators usually bake ranks (each pod's number in distributed training) into the pod spec, while Firepool pods boot as generic workers. Illustration: when a job needs 4 pods, the dispatch loop claims all 4 under a single lock hold, so two jobs never end up each holding part of the pods they need and waiting on each other. After the claim, Firepool injects torchrun rendezvous and rank environment variables into the 4 pods; when the job ends this information is cleared and the 4 pods become independent warm pods again.

The line between image and code package is drawn like this: natively compiled dependencies and framework versions go in the image; prompts, reward functions, verifiers and experiment logic go in the code package. If agent-A's verifier changes, the next job picks it up. Changing the framework version means rebuilding the base image.

Late binding makes every job pay a little extra time: delivering, unpacking and starting the code package took 0.40 to 0.54 seconds at the median across the production benchmarks. In return, warm pods are no longer fragmented by image. The authors put 64 environments into a simulator and compared held capacity: the total time pods hold GPUs, with a pool kept always on at peak concurrency counted as 1× (explained further below). With one pool per environment and each pool allowed to grow to the full budget, held capacity was 2.15×, against 1.06× for one shared pool. If the total budget is split across the environment pools instead, p90 start latency reaches 400 seconds. Per-environment pools can thus be configured two ways and both lose: without splitting the budget they hold more GPUs, and with a split budget jobs wait longer.

Held capacity and tail latency as the number of environments grows in the simulator: one late-binding shared pool stays roughly flat, one pool per environment roughly doubles held capacity when each may grow to the full budget, and tail latency rises sharply when the budget is split
Held capacity and tail latency as the number of environments grows in the simulator: one late-binding shared pool stays roughly flat, one pool per environment roughly doubles held capacity when each may grow to the full budget, and tail latency rises sharply when the budget is split

Late binding does not help jobs that bring their own image, because each hardware pool has only one base image and such jobs cannot use the warm pods in it. In LinkedIn's pools where each job needs 8 GPUs, these jobs are the majority, so those pools currently get almost no benefit from warm pods.

Three isolation rules for reusing pods

Reusing a pod directly can leave the previous job's files, shared memory and credentials to the next job. The simplest defense is to retire (delete) the pod after every job, but that brings back the 22-second wait at every step. Firepool sets a few rules between these two extremes of "reuse directly" and "delete after use".

The most basic rule: a used pod is only given to the caller that used it last. When a job ends, the pod enters a reserved period (the paper calls it an owner hold; its length is configured per pool, and the pod is retired when it expires), so agent-A's next step can reclaim it before any other caller. The broker searches for a pod in this order:

OrderWhat it looks foragent-A's step 2 (illustrative)
1An idle pod last used by the same callerpod-7 is in its owner hold and matches; its page cache and installed dependencies are still there
2An idle pod never used beforeOnly if item 1 misses
3A zero-VRAM standby pod on a busy GPUCovered in the next section
4A new podOnly when none of the first three exist; waits more than twenty seconds

The second rule handles the boundary between sessions: when a caller switches to a new session tag, its next job can still run once on the old pod, and the old pod is retired after that job. Illustration: agent-A finishes bug 42 and starts the next bug with the new tag fix-bug-43. Step 1 of the new session can still run once on pod-7 without waiting; after it, pod-7 is retired. So state left by bug 42 can reach at most step 1 of bug 43, not step 2; under the default configuration, pods are never reused across callers either.

The third rule handles cleanup between jobs. The daemon in the pod kills the job's entire process tree, checks against /proc that all processes are gone, then empties the code directory, /dev/shm and any volume marked for wiping, keeping read-only caches. If it cannot confirm this, for example when a process survives SIGKILL, the broker simply deletes the pod. A failed cleanup therefore costs only one extra pod start, and a dirty environment is never handed to the next job.

These rules target internal jobs and isolate callers from one another. The first job of the same caller's next session may still see the previous session's state; workloads that need kernel-level isolation must add microVMs.

The cost of isolation depends mainly on where the retirement boundary is drawn. The cleanup itself is cheap: a job that leaves no extra files or processes takes only 9.3 milliseconds, and the worst case the authors built takes 1.67 seconds. With the session tag as a retirement boundary, only about half of multi-step jobs stayed on warm pods. Turning the tag into a routing hint (prefer returning to the same pod, but do not retire it), together with event-driven dispatch, raised the share of jobs on warm pods to 0.99 and cut new pods per benchmark repeat from 28.9 to 2.5. The authors conclude that the time cost of isolation lies mainly in the policy of how often pods are replaced, and the cleanup itself is negligible.

Zero-VRAM standby: no twenty-second wait when a pod must change

The rules above leave a gap. When the next job comes from another caller, or crosses a session tag, the old pod has to be held or retired. On a capacity-capped pool, that job waits for a new pod to be built, which the paper puts at 22 to 200 seconds. Illustration: on one GPU, pod-7 has just run agent-A's last step when a job from agent-B arrives. pod-7 is still in agent-A's owner hold, so agent-B cannot use it.

The simple answer is to prepare an empty pod on another GPU, but that GPU sits idle while waiting. Firepool places the successor pod on the same physical GPU. Using fractional shares from a GPU-sharing scheduler (HAMi with Volcano on the H200 clusters), it puts 1 active pod and up to 3 standby pods on one GPU. A standby pod is a pod started in advance, queued to take over.

The share here is only a slot in the scheduler's books, which lets the standby pod be placed on this GPU. A standby pod runs nothing but the idle daemon, creates no CUDA context and allocates no GPU memory, which is why the paper calls it a zero-VRAM standby. Because the standby pod starts while the current job is still running, the twenty-plus seconds of pod startup overlap with the current job's run time.

A reader might ask whether the running job can then use only a fraction of the GPU memory. Illustration: suppose each pod is nominally given a quarter of a GPU, so 141 GB on an H200 divided by 4 is about 35 GB. When a job starts on the active pod, the daemon raises the memory limit to the full 141 GB only in the job process's environment, leaving the pod spec unchanged. The job can use the whole GPU, while the 3 standby pods keep their 35 GB limit, which does not matter because they use no memory at all.

During the handover, only one pod per GPU may run a job at a time. Only after agent-A's job process is confirmed to have exited and the GPU driver has freed its memory does a per-GPU gate promote the standby pod claimed by agent-B's job to the active pod and start the job. pod-7 is held or retired in the background without blocking agent-B. The scaling controller then adds standbys back to this GPU, and each one has to be created from scratch (see the next paragraph). For accounting, this group of up to 4 pods counts as one physical GPU.

Standby pods can also run out. Illustration: 4 jobs from different callers arrive in a row, each running 2 seconds, each needing a pod change. Once the 3 standby pods on this GPU are used up, the replacements still need twenty-plus seconds to start, and the fourth job has to wait. Standby is a buffer, and it refills at the speed of pod creation.

The authors measured this handover directly. One caller ran 30 jobs in sequence on a 2-GPU H200 pool, each job with a new session tag, and the whole set was repeated 10 times. Under the second rule, a job with a new tag can run once on the previous pod, which is then retired, so jobs alternate between "reuse the previous pod" and "find no idle pod and switch"; the latter are called handover jobs. With standby on, 101 of 117 handovers across the 10 repeats (86%) were served by a standby pod. Compared with standby off, the median handover wait fell from 51.0 seconds to 0.94 seconds and the whole run from 1,023 seconds to 337 seconds, 3.0× faster. This shows that once a standby pod can take over, switching pods across sessions mostly no longer waits for a new pod to start, and standby was faster in every one of the 10 repeats, so it was not a few lucky runs.

The GPU cost and limits of warm pods

To decide how many pods to keep, the obvious approach is to scale pod count by GPU utilization. But warm pods are idle by design, so their utilization is low, and scaling by utilization would shrink the pool exactly when it should be ready for a burst. Firepool scales by the queue instead: the controller reconciles periodically (every 30 seconds by default, counting demand against existing pods) and computes a deficit each round, where the floor is the number of pods kept even when demand is zero:

deficit = max(floor − physical GPUs, demand − eligible supply, 0)

The number actually created is the smallest of four values: the deficit, the headroom below the ceiling (ceiling minus physical GPUs), the per-round creation limit, and the remaining global budget set per GPU model. Here demand is the number of pods needed by queued, assigned and running jobs; eligible supply is the number of starting, idle, assigned and running pods; physical GPUs add to eligible supply the pods in owner hold and the pods that occupy GPUs but cannot take warm jobs (pods from an old template and single-use pods). An active pod and its standbys count as one GPU, and standbys are never counted as supply.

Counting starting pods as supply prevents repeated over-creation. Illustration: 5 jobs are queued, this round creates 5 pods, and by the next round they have not finished starting. If starting pods were not supply, demand 5 minus supply 0 would create 5 more, and 5 more the round after. With them counted, the deficit is 0 and nothing extra is created.

Owner-held pods count toward the ceiling but not as supply for other callers, which prevents starving them. Illustration: the ceiling is 5, agent-A has 4 pods in owner hold, and agent-B submits 1 job. If those 4 counted as supply, the deficit would be 0 and agent-B would wait for pods it cannot use. Without them, the deficit is 1 and the headroom below the ceiling is 5 − 4 = 1 (the 4 are counted in physical GPUs), so exactly 1 pod is created. If the 4 were not counted in physical GPUs either, the headroom would be overestimated as 5. With a deficit of 1 this example would still be fine, but with a larger deficit the new pods would push the total past the ceiling.

Operators tune four knobs: the floor (how many pods to keep even when demand is zero), the idle cap (how many idle pods to keep after keep-alive expires), the ceiling (the most GPUs to occupy) and keep-alive (how long a pod may stay idle before it can be reclaimed). Setting the floor and idle cap to 0 with a keep-alive gives the TTL (time-to-live) retention common in serverless, which reclaims a pod after an idle timeout. Setting the floor to peak and keep-alive to infinite gives a static cluster. One single-use pod per job is a Kubernetes Job. These common designs all become special cases of these settings, which let the authors compare them within one system.

For the GPU cost of warm pods, the authors use the held capacity mentioned earlier: how much GPU or CPU time the pool's pods hold from binding to deletion, busy or idle, divided by the time a pool kept always on at peak concurrency holds under the same load.

The trade-off between held capacity and start latency, both relative to an always-on pool sized for peak, with lower-left better: the left panel is measured on a local cluster, the right is simulated, the shaded band is 60 to 120 s keep-alive, and a 900 s keep-alive holds more than the always-on pool in both
The trade-off between held capacity and start latency, both relative to an always-on pool sized for peak, with lower-left better: the left panel is measured on a local cluster, the right is simulated, the shaded band is 60 to 120 s keep-alive, and a 900 s keep-alive holds more than the always-on pool in both

In a simulation using the production 22-second pod start, a 60 to 120 second keep-alive holds 0.48 to 0.68× the capacity of the always-on pool, and the median start latency is only 0.01 to 0.27 seconds above it. This suggests that as long as keep-alive covers the pauses in the loop, about half the capacity buys nearly all of the latency benefit. In the local measurement, a 900-second keep-alive held 1.15×, more than the always-on pool; the authors explain that a long keep-alive keeps the pods over-created during a burst until they time out. Keep-alive should therefore be set by the loop's pauses, and longer is not safer.

Measured on H200, reuse held 12.3 GPU-seconds per job, against 10.7 GPU-seconds when each job used a new session tag, 15% more. The extra is mostly the time a pod is held for its caller between agent steps, and it buys a 3.6× faster loop. For short agent loops like these, the GPU cost of holding pods is small: 15% more GPU time cuts the run time to under a third.

In benchmarks, reuse made loops 2.2 to 13 times faster

The benchmarks all ran on production H200 nodes (plus a CPU comparison), with the GPU pool capped at 2 GPUs and fixed-duration jobs, so differences come only from how pods are obtained. In the agent loop, each job takes 0.5 to 1.6 seconds, with LLM pauses in between. The RL loop runs 8 rollouts, 2 evaluations and 1 policy update per round, each job taking 1.5 to 4 seconds, with barriers between phases, so the slowest pod start in each phase holds up the whole round.

There are three control groups. "Fresh pod per job" is closest to a Kubernetes Job: each job gets a single-use pod that is deleted afterwards. "New tag per job" removes only the same-caller reuse: a job can still run once on the previous pod, and the standby pods on the H200 pool are still there. "Standby off" is the experiment described in the standby section. In the first two comparisons the treatment group is full Firepool reuse (a stable tag per caller); in the standby comparison both groups use a fresh tag per job and differ only in whether standby is on.

One round of two agents' tool loops on production H200 nodes with a two-GPU cap; each bar is one job's wait and run, colored by how its pod was obtained. Top: new tag per job, 185 s for the round. Bottom: Firepool reusing warm pods, 53 s
One round of two agents' tool loops on production H200 nodes with a two-GPU cap; each bar is one job's wait and run, colored by how its pod was obtained. Top: new tag per job, 185 s for the round. Bottom: Firepool reusing warm pods, 53 s
ComparisonWorkloadLoop time (treatment / control, s)SpeedupSubmit to start p50 (treatment / control, s)
Reuse vs fresh pod per job, H2004 agents119.3 / 1,554.213.0×0.61 / 89.5
Reuse vs fresh pod per job, H200RL163.0 / 1,761.910.8×3.92 / 115.8
Reuse vs new tag per job, H2002 agents51.8 / 186.43.6×1.91 / 2.29
Reuse vs new tag per job, CPURL116 / 5164.5×2.83 / 52.6
Reuse vs new tag per job, CPU4 agents186 / 4032.2×1.78 / 27.5
Standby on vs off, H20030 sequential jobs337 / 1,0233.0×1.66 / 28.2

The first two rows answer how much faster reuse is than a fresh pod per job. With a two-GPU cap, every job waits for the previous pod to be deleted and a new one to start, a median queue of about 90 seconds, and a multi-step agent loop grows from 0.6 to 2.5 minutes to 22 to 32 minutes. In short-job loops on tightly capped GPUs, each job's creation wait accumulates into the whole run. The 13× figure holds for short jobs under a two-GPU cap; it would not be as large for long jobs.

The third row has to be read separately from the first two. The two groups' median starts differ by only 0.4 seconds (1.91 vs 2.29), and the confidence interval of the paired difference spans zero. The reason is that 137 of the 190 jobs in the new-tag group landed on standby or recycled pods and did not wait for a new pod. Yet the whole run was 3.6× slower: 20 jobs did wait for new pods, 82 seconds at the median, and the jobs queued behind them waited too. This group's p90 start was 59.1 seconds, against 3.52 seconds for the reuse group. Looking only at the median start would miss this difference entirely; warm pools should be judged by the tail and the total run time.

Start latency distributions, dots are p50 and triangles p90. (a)(b) On CPU, a new tag per job shifts most jobs right by one pod start. (c) On H200 the two groups' medians are close and the gap is in the tail of jobs waiting for new pods; (d) is the standby benchmark
Start latency distributions, dots are p50 and triangles p90. (a)(b) On CPU, a new tag per job shifts most jobs right by one pod start. (c) On H200 the two groups' medians are close and the gap is in the tail of jobs waiting for new pods; (d) is the standby benchmark

The CPU pools in rows four and five have no standby, so with a new tag per job most jobs pay one pod start. The RL loop has phase barriers, and a straggler holds up the whole round, so it slows by 4.5×, more than the 2.2× of the agent loop.

These are all fixed-duration benchmarks; the production data answer whether warm pods suffice under real traffic. In the 14-hour sample mentioned at the start, 98.6% of requests on the single-GPU H200 pool got a warm pod or a standby pod. During a 3-hour burst within it, about 2,500 single-GPU jobs used up the idle pods, and 187 of the 222 jobs that found no idle pod (84.2%) were taken over by standby. Standby thus caught most of the jobs that found no idle pod in a real burst as well; the other 35 jobs got new pods.

Over a week of data, once a job got a warm or standby pod, its code started in 0.3 to 0.5 seconds at the median, with p90 no more than 0.7 seconds. Yet during the burst, warm-reuse jobs took 53 seconds at the median from submission to start, because callers had used up their GPU quota and new jobs queued behind their own earlier jobs.

Under a burst, jobs wait for quota; pods are no longer the bottleneck.

To cut those 53 seconds further, the thing to tune is quota and budget; further optimizing pod startup would not help.

Warm pods do not pay when jobs are long or pauses too long

As jobs get longer, the benefit shrinks. Illustration: each job saves 22 seconds. With a 2-second job body, total time including the wait is (2 + 22) / 2 = 12 times the work time; with a 200-second body it is only (200 + 22) / 200 ≈ 1.1 times. The paper's simulation (calibrated on CPU) points the same way, which means the gain is a short-job effect: beyond job bodies of about 200 seconds for RL and about 100 seconds for agents, the loop speedup falls below 1.1×.

When pauses are too long, the benefit also disappears. Illustration: keep-alive and owner hold are both set to 60 seconds, and agent-A waits 2 minutes for the LLM at each step; the pod is reclaimed before the next step arrives, so agent-A still waits for a new pod every step. In the paper's simulation, once the time spent waiting for the LLM between steps (the paper calls it think time) exceeds the keep-alive, the speedup is gone.

Standby also does not yet cover multi-GPU job groups: using it for them requires promoting the whole group of standby pods together, which the authors list as next work.

Back to agent-A's numbers: with a new pod every step, the run takes 99 seconds (illustrative). If all 4 steps get warm pods and each waits about 1 second (illustrative), it takes 4 × (1 + 2) + 3 = 15 seconds. The 99 − 15 = 84 seconds saved do not come from faster pod startup; a new pod still takes twenty-plus seconds. Firepool moves that time to when no job is waiting on it.

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Related papersMore from Paper Daily

  1. Infra · 2026-11-12SHADOWPREFILL: Keeping Your Next Model Warm for Per-Step Model Routing in Agentic LLM ServingRouting an agent per step, with easy steps on a small model and hard ones on a large model, saves money. But every model switch carries an extra charge. A KV cache belongs to one model, so the model being switched to lacks the recent context and must compute it before emitting its first token. On long-context workloads such as coding agents, the switching step can take up to 3.2 times as long as…
  2. Infra · 2026-11-03HiState: Hierarchical Prefix Cache Management for Hybrid LLM Serving in Agentic WorkloadsThe recurrent layers of a hybrid model compress the whole prefix into a fixed-size snapshot, and it cannot be cut back to an earlier position. So even when the KV is still cached, if the snapshot at the match point was never saved or has already been evicted, a long stretch must be recomputed from an earlier snapshot, or from scratch. HiState exploits an asymmetry: a cache hit reads every matchin…
  3. Personalization · 2026-10-25PAIR: Preference-Aligned Intent Retrieval for Premium People Recommendation at LinkedInOn LinkedIn, a software engineer who starts looking for a job usually wants to open the profiles of recruiters, hiring managers and HR, not other engineers. In professional networking, who should be recommended is often complementary, not similar. Yet most recommender systems assume that the more alike two profiles are, the more one should be recommended, so they keep showing him his peers. PAIR…
  4. Infra · 2026-11-03Matching Workloads to Blackwell: Power-Resolved Benchmarking of NVIDIA B200 and B300 across HPC and AI WorkloadsThe B300 is the upgrade to the B200, with more memory, but it is not faster on every task. The University of Chicago's Research Computing Center measured both on two eight-GPU clusters, and on convolutional network inference the B200 was up to 5.6 times faster than the B300. Per-GPU power shows why: on those tasks the B300 drew only 36% to 53% of its power limit, and the time went to the cost of…

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…