Learn / What are GPU spot instances?
What are GPU spot instances?
A GPU spot instance is spare cloud capacity rented at a steep discount, with the catch that the provider can reclaim it at short notice. Spot GPUs are typically 40 to 80% cheaper than the same hardware on-demand (a single H100 can dip below $1/hr on spot against roughly $2/hr on-demand), which makes them the cheapest way to run GPU work, as long as your workload can survive being interrupted.
Spot vs on-demand
| Spot / interruptible | On-demand | |
|---|---|---|
| Price | 40 to 80% cheaper | Full rate; the price ceiling |
| Can be reclaimed? | Yes, at short notice | No, it holds until you stop it |
| Good for | Checkpointed training, batch jobs, fault-tolerant serving | Production endpoints that must stay up |
| Risk | Preemption mid-job | Cost |
On-demand is both the price ceiling and the safety net: it costs more, but it cannot be preempted, and an on-demand renter always wins a machine over any spot bidder. That relationship is what makes spot pricing a live market rather than a fixed discount.
How spot pricing works
Providers sell idle GPUs on spot because an empty card earns nothing. The discount clears that inventory, and the reclaim right lets them take the capacity back when a full-price customer wants it. On some clouds the spot price is set for you; on GPU marketplaces it is a continuous auction where you name a bid, pay it per hour while you hold the machine, and lose the machine the moment someone outbids you. Prices move constantly with supply and demand (the cheapest H100 tiers can change price more than a dozen times an hour), which is why a live H100 & H200 price tracker is more useful than a static rate card.
There is a subtle trap in the auction version: once you hold a machine, the only price the market reports for it is your own bid, so a rival bidding up underneath you is invisible until your instance is taken. We took that problem apart with real data in bidding blind in GPU spot auctions, and found that the naive "defend my margin" strategy loses to simply renting on-demand, while a single risk-priced bid plus a timer wins.
Preemption: the catch
Preemption is when the provider reclaims your spot instance. There are two flavors, and they cost very differently:
- Outbid, disk kept. Your processes stop but the volume survives, so when capacity frees up again the same machine resumes in place in about a minute. Cheap, and common on marketplaces.
- Evicted, machine gone. An on-demand renter takes the hardware and you cannot get it back until they leave. You restart elsewhere and re-download your container image and model weights, tens of gigabytes, which means both downtime and ingress cost.
Preemptions also tend to arrive in bursts rather than at random, so one interruption is a signal that another may be seconds behind. Designing for that is the difference between spot being cheap and spot being unusable.
When to use spot for AI workloads
- Fine-tuning and training that checkpoints regularly: a preemption costs you the work since the last checkpoint, nothing more. See fine-tuning memory requirements to size the GPU first.
- Batch and offline inference: large jobs with no latency SLA can absorb interruptions by simply resuming.
- Fault-tolerant serving, but only with the right architecture: health-checked replicas behind a router, pre-staged warm disks so recovery takes a minute instead of fifteen, and an on-demand fallback for when spot capacity dries up.
The workload to keep off bare spot is a single production endpoint with no fallback: the one time it is reclaimed, it drops requests. For that case, weigh spot against reserved or on-demand capacity, and against owning the hardware, in how to reduce LLM inference cost and hosted vs self-hosted inference.
Frequently asked questions
What is a GPU spot instance?
A spot or interruptible instance is spare cloud capacity sold at a steep discount with the catch that the provider can reclaim it at short notice. Spot GPUs are typically 40 to 80% cheaper than on-demand and suit fault-tolerant work like checkpointed fine-tuning or batch inference, but not production endpoints that must stay up.
What happens when a spot instance is preempted?
Your running processes are stopped, usually with little or no warning. On many marketplaces the disk survives, so if capacity frees up the same machine can resume in place in about a minute; otherwise you restart elsewhere and re-download your image and weights, which is the expensive failure mode.
Can I run a production inference endpoint on spot GPUs?
Only with an architecture that tolerates preemption: health-checked replicas behind a router, pre-staged warm disks for fast recovery, and a fallback to on-demand capacity. A single spot instance with no fallback will eventually drop requests when it is reclaimed.
Is it cheaper to use spot GPUs or buy the hardware?
An H100 card costs roughly $25,000 to $30,000 before the server, power, and networking around it. At a rented on-demand rate near $2 per hour you would run about 15,000 hours, around two years of continuous use, before matching the purchase price. Renting wins unless you can keep the card busy around the clock for years.