Before you run 70B LLM GPU workloads, do the memory arithmetic. A 70-billion-parameter model is the point where a single consumer card stops being enough and the choice between one large GPU and several smaller ones becomes real. This article walks through the VRAM math in general terms, compares the three common layouts (two 48 GB cards, one 80 to 96 GB card, four 24 GB cards), and shows how to find a matching machine on EU-listed hosts with the DE, NL, FR and SE country codes.
For European locations in general, start with the GPU cloud in Europe guide.
The VRAM math, in plain terms
Memory for inference has two main parts.
1. Weights. Parameters times bytes per parameter:
| Precision | Bytes per parameter | 70B weights, roughly |
|---|---|---|
| 16-bit (FP16/BF16) | 2 | about 140 GB |
| 8-bit | 1 | about 70 GB |
| 4-bit | about 0.5 | about 35 to 40 GB |
Quantization formats add a little overhead, so treat these as floors, not exact sizes.
2. KV cache. Every token in every active conversation stores keys and values for every layer. For a typical 70B architecture with grouped-query attention, a common estimate at 16-bit is on the order of a third of a megabyte per token. That means one 8,000-token conversation needs a few GB of cache, and ten of them at once need tens of GB. Long context and many concurrent users can need as much memory as the weights.
Add a few GB for the runtime, CUDA context and activations. A practical rule: weights + KV cache for your target context and concurrency + 10 to 20 percent headroom.
Three layouts compared
| Layout | Total VRAM | 4-bit 70B | 8-bit 70B | Notes |
|---|---|---|---|---|
| 2x RTX A6000, 48 GB each | 96 GB | Comfortable, room for cache | Tight; little cache | Needs tensor parallelism across two cards |
| 1x H100 SXM, 80 GB | 80 GB | Comfortable on one card | Does not leave useful cache room | No splitting; simplest software |
| 1x RTX PRO 6000, 96 GB | 96 GB | Very comfortable | Fits with modest cache | One card, more memory than an H100 |
| 4x RTX 4090, 24 GB each | 96 GB | Works with tensor parallelism | Tight | More communication between cards; consumer class |
One big card is the simplest: no model splitting, fewer moving parts, and usually the smoothest latency. Two 48 GB cards split the model with tensor parallelism and give you similar total memory. Four 24 GB cards reach the same total from consumer hardware, at the cost of more traffic between GPUs, which on consumer cards usually goes over PCIe. That overhead can reduce throughput, so test before committing.
Account level matters here. The RTX 4090 is open to new accounts. The RTX A6000, H100 and RTX PRO 6000 are workstation or datacenter cards and need a higher account level; the console tells you when a choice is not available.
For more on when several GPUs beat one, see single vs multi GPU cloud. If you need even more memory per card, the H200 rental guide covers the 141 GB class.
Finding a matching machine in Europe
On Cloudj, the GPU count is part of the listing: 1x, 2x, 4x or 8x, with all GPUs in the same machine. Open the deploy page with the RTX A6000, set the GPU count to 2x, and enter:
DE, NL, FR, SEEach row shows the country at country level, plus reliability, max duration and bandwidth price. Multi-GPU machines are fewer than single-GPU ones, and stock changes during the day. If nothing fits:
- Try the other layouts: 1x H100 SXM, 1x RTX PRO 6000, or 4x RTX 4090.
- Use the minimum VRAM filter instead of a model, and check which cards appear.
- Add more EU codes that suit your policy.
- For batch jobs without an EU rule, widen to all countries; the RTX A6000 in the USA guide covers the
USsearch for the same card.
Disk matters too: 4-bit 70B weights are tens of GB, and you may keep more than one format. Disk cannot be resized after deploy, so size it with a margin.
Serving across two or four GPUs
Install your serving stack over SSH. With vLLM, split the model across GPUs in one machine with tensor parallelism:
pip install vllm
vllm serve <hf-model-id> --tensor-parallel-size 2 --max-model-len 8192 --host 127.0.0.1 --port 8000Use --tensor-parallel-size 4 on a 4x machine. While it serves a realistic load, check that every GPU holds part of the model and stays busy:
nvidia-smi --query-gpu=index,memory.used,utilization.gpu --format=csv -l 5The vLLM documentation explains quantized checkpoints and memory flags such as the GPU memory fraction.
Practical cautions
- Licences. Most 70B open models have licences with conditions. Read them before commercial use.
- Backups. Machines are independent hosts with no hardware SLA, and there is no automatic move if one goes offline. Keep weights and configuration where you can redeploy quickly.
- Availability window. Check the "available until" date before a long service.
- Data location. Restricting to EU codes keeps work on EU-listed hosts, but Cloudj makes no compliance claims; check your own obligations.
FAQ
Can a 70B model run on one 24 GB card?
Not usefully. Even at 4-bit, the weights alone exceed 24 GB. Use 96 GB of total VRAM or one 80 GB card.
Is one H100 better than two RTX A6000s?
One card is simpler and avoids splitting. Two A6000s give more total memory for cache. Test both with your context length.
Do the GPUs in a 4x listing sit in the same machine?
Yes. All GPUs in an instance are in the same host.
Where are prices?
Live on GPU cloud and each model page.
Next step
Work out your weights plus cache, then open the deploy page with the RTX A6000, set 2x and DE, NL, FR, SE. Compare with a single H100 or RTX PRO 6000, test under real load, and keep backups of everything you deploy.