Retrieval-augmented generation looks like one system, but it is really four jobs with very different hardware needs. Putting all of them on one big GPU wastes money; putting them in random countries can break the promises you made about where data lives. This guide shows how to design an embedding model GPU server EU setup: which parts need a GPU, which run happily on a CPU VPS, how to keep everything in European countries you chose, and how to survive a GPU host disappearing.
For choosing European countries in general, see the GPU cloud Europe guide.
The four jobs in a RAG system
| Job | What it does | Needs a GPU? |
|---|---|---|
| Ingestion and chunking | Parse documents, split into chunks | No, CPU work |
| Embedding | Turn chunks and queries into vectors | Helpful for bulk ingestion; small models can run on CPU for queries |
| Vector search | Find the nearest chunks to a query | No, a CPU database (for example PostgreSQL with pgvector) |
| Reranking and generation | Reorder results, write the answer with an LLM | Yes, especially the LLM |
The practical conclusion: the GPU is for the embedding model during bulk ingestion, the reranker and the LLM. Everything stateful (documents, vectors, metadata, users) belongs on a stable server that does not disappear. The GPU instance vs VPS for AI article explains this split in more general terms.
Reference architecture in Europe
- VPS in Frankfurt or Amsterdam: API, authentication, ingestion workers, PostgreSQL with pgvector, document storage and backups. A Cloudj VPS in Frankfurt or Amsterdam keeps all stored data in one known country.
- GPU instance in an EU country you allow: an embedding server, a reranker and an LLM server, all listening on localhost. It keeps no data you cannot recreate; documents pass through it only while being processed.
- Link between them: an SSH tunnel from the VPS to the GPU, so the model ports are never exposed publicly.
The PostgreSQL VPS setup guide covers installing the database side; add the pgvector extension for vector search.
Keeping data in your chosen countries
GPU location on Cloudj is a country filter over a live list of host machines. If you only enter EU codes, every machine you can deploy runs in one of those countries:
DE, NL, FRAdd more codes your rules allow (BE, AT, SE, PL and so on) to widen the choice. Results show the country only, never a city.
Be clear about what this gives you: control over which countries your data passes through. It does not make anything compliant. Cloudj does not claim hosts meet GDPR or any other data law, so check your own obligations and those of your clients. The EU data location for GPU cloud article walks through this reasoning. A few habits help: keep logs on the VPS, not the GPU; do not cache documents on the GPU disk longer than needed; and keep backups in the same countries.
Sizing the GPU
Embedding and reranker models are small compared to LLMs; most need a few gigabytes of VRAM. The LLM decides the card:
- L4, 24 GB class: an efficient datacenter card for embeddings, reranking and a small quantized LLM (7B to 8B class). It needs a higher account level.
- RTX 4090, 24 GB: faster for generation, same memory size, available at the starting account level.
- L40S, 48 GB class: room for a larger LLM (around 30B in 4-bit) alongside the embedding model and reranker.
Open the L4 deploy page, set your EU codes and see what is listed. If no L4 is available, search by minimum VRAM instead. Stock changes during the day; check the console. Live "from" prices are on GPU cloud.
Serving the models
Deploy with the Ubuntu + CUDA template and install your servers over SSH. A typical layout on one GPU puts embeddings on port 8081, the reranker on 8082 and the LLM on 8000, all bound to 127.0.0.1:
ssh -p <port> root@<host>
nvidia-smiHugging Face's Text Embeddings Inference serves embedding and reranker models; vLLM or a similar server handles the LLM. Pick a multilingual embedding model if your documents mix languages, and keep the same embedding model for ingestion and queries; changing it means re-embedding everything.
On the VPS, open one tunnel that forwards all three ports:
ssh -N -L 8081:127.0.0.1:8081 -L 8082:127.0.0.1:8082 -L 8000:127.0.0.1:8000 -p <port> root@<host>Run it under systemd so it restarts automatically.
Planning for a GPU host change
Each GPU machine is an independent host with no hardware SLA and an "available until" date. Design for replacement:
- Keep a setup script that installs the servers and downloads the models.
- Store nothing irreplaceable on the GPU; vectors and documents live on the VPS.
- Health-check each service from the VPS and alert on failure.
- When a host goes, deploy a new one in your EU codes, run the script and repoint the tunnel.
Because embeddings are stored on the VPS, a new GPU only needs the same model files to continue serving queries.
FAQ
Can the vector database run on the GPU instance?
It can, but it should not. The GPU host can disappear, and its disk with it. Keep the database on a VPS.
Does filtering by EU codes make my RAG system GDPR-ready?
No claim like that can be made by a filter. It keeps machines in countries you choose; your legal obligations are yours to check.
Do I need a GPU for embeddings?
For bulk ingestion of many documents, a GPU is much faster. For embedding single queries with a small model, a CPU can be enough.
Which card is the best starting point?
An RTX 4090 at the starting account level, or an L4 if your account allows datacenter GPUs. Move to a 48 GB class card for larger LLMs.
Build it in Europe
Open the L4 deploy page, enter your allowed EU codes and see what is listed now. Compare live prices on GPU cloud.