An Ollama GPU server is the quickest way to try open models on a real GPU: one install command, one pull, and you have a local API. The catch is security. Ollama's API has no built-in authentication, so a careless setup leaves a free GPU open to the internet. This guide installs Ollama on a Cloudj GPU instance near Asian users, picks a model that fits the card's VRAM, keeps the API bound to localhost, and shows two safe ways to reach it: an SSH tunnel for yourself, or a VPS reverse proxy with authentication for an app.
For choosing among Asian locations first, see the GPU cloud in Asia guide.
Step 1: deploy a GPU near your users
Ollama does not come preinstalled on Cloudj. You install it yourself in the Ubuntu 22.04 + CUDA (SSH) template, which takes a couple of minutes.
Open the deploy page with the RTX 3090, or pick another card, and enter SG, JP, HK in the Country field. If a machine is listed, check reliability, max duration and download speed; model pulls are large, so fast downloads help. Stock changes during the day; if nothing fits, add KR or TW, or switch to a card with similar VRAM.
Size the disk for the models you plan to keep, plus a margin. Disk cannot be resized after deploy.
Step 2: pick a model by VRAM
Ollama pulls quantized models by default, which is what makes them fit on consumer cards. Use the card's memory as the main guide, and leave headroom for context:
| Card | VRAM | Comfortable range (quantized, hedged) |
|---|---|---|
| RTX 3090 | 24 GB | Small and mid-size models, up to the low 30B class at 4-bit with modest context |
| RTX 4090 | 24 GB | Same sizes as the 3090, with faster generation |
| RTX 5090 | 32 GB | Low 30B class with more context, or several small models loaded together |
If a model does not fully fit, Ollama may split it between GPU and CPU, which is much slower. Check with ollama ps, which shows how much of the model sits on the GPU. All three cards are consumer GPUs, available to new accounts. Live prices are on GPU cloud.
Step 3: install and run Ollama
Connect with the SSH command from the instance's Connection tab. The SSH guide for GPU instances covers keys and ports if this is your first time.
apt-get update && apt-get install -y curl tmux
curl -fsSL https://ollama.com/install.sh | shIf the installer asks for another package, install it with apt-get and run it again. Containers usually have no systemd, so start the server yourself inside tmux:
tmux new -s ollama
ollama serveDetach with Ctrl-b d, then pull and test a model:
ollama pull <model>
ollama run <model> "Say hello in Vietnamese and Japanese."
nvidia-smiBy default Ollama listens on 127.0.0.1:11434. Keep it that way. Do not set OLLAMA_HOST=0.0.0.0 on a GPU instance: the API has no login, and anyone who finds the port could use your GPU at your expense. The Ollama documentation lists available models and their sizes.
Option A: SSH tunnel for personal use
If only you, or a few teammates, need the API, forward the port over SSH from your laptop:
ssh -N -L 11434:localhost:11434 -p <port> root@<host>Now http://localhost:11434 on your laptop reaches Ollama on the GPU. Desktop chat apps and code editors that support Ollama work against that address. Nothing is exposed publicly.
Option B: VPS reverse proxy with authentication
For an app or a team, put a stable Singapore VPS in front. The VPS keeps an SSH tunnel to the GPU, and Nginx adds TLS and a password:
server {
listen 443 ssl;
server_name llm.example.com;
ssl_certificate /etc/letsencrypt/live/llm.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;
location / {
auth_basic "LLM";
auth_basic_user_file /etc/nginx/.htpasswd;
proxy_pass http://127.0.0.1:11434;
proxy_read_timeout 300s;
proxy_buffering off;
}
}proxy_buffering off lets streamed tokens reach the client as they are generated. When the GPU host changes, because its "available until" date arrives or you move to a bigger card, you point the tunnel at the new host and the URL stays the same. The VPS in front of a GPU server guide covers the tunnel as a systemd service.
Keep it tidy
- Models live under
~/.ollama/models. Remove ones you no longer use withollama rm. - Machines are independent hosts with no hardware SLA. Keep your Modelfiles and prompts in a repository so you can rebuild anywhere.
- Respect each model's licence, especially for commercial use.
For Japanese-language models specifically, the Japanese LLM GPU server guide covers model choice and token budgets.
FAQ
Does Cloudj ship Ollama preinstalled?
No. Install it with the official script in the Ubuntu + CUDA template; it takes a few minutes.
Why is my model slow?
It may not fit fully in VRAM. Run ollama ps; if part of it runs on the CPU, choose a smaller model or a larger card.
Can I open port 11434 to the internet?
You should not. Ollama has no authentication. Use an SSH tunnel or a VPS proxy with a password.
Which country should I pick in Asia?
The one closest to your users with a listed machine. Try SG, JP, HK and measure.
Get started
Open the deploy page with the RTX 3090, enter SG, JP, HK, install Ollama in a few commands and keep it on localhost. Use a tunnel for yourself or a Singapore VPS with authentication for your app.