GPU Rental: How Much VRAM Does a Local LLM Need?

The most common mistake in GPU rental is picking a graphics card by its name instead of by the model you plan to run. When you want to host a large language model (LLM) yourself, gaming benchmarks barely matter. What decides everything is VRAM: if the model does not fit in graphics memory, it either refuses to load or spills into system RAM and crawls along at a few words per second. This guide shows you how to estimate the VRAM a model needs, how quantization and context length change that number, which class of card makes sense for which job, and how to get a rented GPU server up and running step by step.

Three concepts to understand before GPU rental

A VRAM estimate has three parts. Once you understand each of them, you can size almost any model in a couple of minutes.

• Parameter count: The "8B", "14B" or "70B" in a model name is the number of weights in billions. These weights take up most of the memory.
• Quantization (precision): How many bits each weight is stored in. FP16 needs 2 bytes per weight, 8-bit needs 1 byte, and 4-bit roughly half a byte. The same model can differ in size by a factor of four depending on quantization.
• Context and KV cache: The amount of text the model keeps "in mind" at once. For every token it stores key and value vectors in each layer, and that cache grows with context length.

On top of that, the CUDA runtime, the inference engine's own buffers and concurrent requests need anywhere from a few hundred MB to a few GB of headroom.

How much VRAM does a local LLM need? A practical formula

A rough but reliable formula looks like this:

Required VRAM ≈ (Parameters × Bytes per weight) + KV cache + 10-20% headroom

Example: 14B model, 4-bit (≈ 0.55 bytes per weight)
  Weights  : 14 × 0.55 ≈ 7.7 GB
  KV cache : ≈ 1.5 GB for 8K context
  Headroom : ≈ 1 GB
  Total    : ≈ 10-11 GB  → a 12 GB card is tight, 16 GB is comfortable

We use 0.55-0.6 rather than 0.5 for 4-bit because popular formats such as Q4_K_M in GGUF keep some layers at higher precision. Real file sizes usually land in that range.

Approximate VRAM by model size

The figures below cover the weights only and are approximate; architectures vary slightly. The last column is the card memory we suggest for comfortable use with 4K to 8K tokens of context.

Sizing the context window (KV cache)

If you plan to summarize long documents, ask questions over a codebase or run RAG (retrieval plus generation), calculate the KV cache separately:

KV cache (bytes) = 2 × layers × KV heads × head dim × context (tokens) × bytes

Example: 8B-class model (32 layers, 8 KV heads, dim 128, FP16 = 2 bytes)
  Per token    : 2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KB
  8K context   : ≈ 1 GB
  32K context  : ≈ 4 GB
  128K context : ≈ 16 GB

You can find the layer and head counts in the model's config.json on Hugging Face. The takeaway: even though an 8B model is only about 5 GB at 4-bit, opening a 128K context can fill a 16 GB card. Most inference engines let you store the KV cache in 8-bit, which roughly halves context memory.

If you serve several users at once, every concurrent request needs its own KV cache. Five users with 8K of context each means setting aside around 5 GB on top of the weights in the example above.

Which card class for which workload?

Thinking in VRAM classes makes a GPU rental decision much simpler:

Look at the architecture, not just the memory size. Older data center cards such as the 24 GB NVIDIA Tesla P40 are attractive for fitting a big model into memory and work fine for quantized inference with llama.cpp, but their FP16 compute is limited compared to newer cards and some current frameworks may not support older architectures. Newer cards like the RTX 5090 with 32 GB of GDDR7 are far ahead in memory bandwidth and token generation speed. For card-specific details, see our Tesla P40 server rental guide and our RTX 5090 server rental guide.

What drives speed?

Once a model fits in VRAM, token generation speed is mostly set by memory bandwidth, because nearly all weights are read from memory for every token. Of two cards with the same VRAM, the one with higher bandwidth feels noticeably snappier in chat-style use. In batch work, where many documents are processed at once, raw compute starts to matter as well.

GPU VDS or physical GPU server?

There are two main routes when you rent a graphics card. These questions help you decide:

• Is the load continuous? If you experiment a few hours a day, a GPU VDS is enough. If you run a customer support bot or internal search around the clock, a physical server gives more predictable performance.
• How many cards do you need? 70B-class models or heavy fine-tuning require several GPUs, which usually means a physical server.
• How much system RAM and disk? Model files can be tens of gigabytes. If you keep several models, plan enough NVMe space and enough system RAM in case part of a model spills over (at least twice the VRAM is a sensible rule).
• Do you need hardware control? If you want to manage the driver, CUDA version and operating system entirely yourself, a physical server gives you that flexibility.

We cover how a GPU VDS works in our guide to GPU VDS. For RAID, remote console and bandwidth on the physical side, see our checklist for renting a dedicated server.

First setup on a rented GPU server

The steps below assume Ubuntu 24.04 LTS. The goal is to confirm the card is visible, download and run a model, and measure real VRAM usage.

1. Check the card and driver

nvidia-smi
# You should see the card model, total/used memory and driver version.
# If the command is missing, install the recommended driver:
sudo apt update
sudo ubuntu-drivers install
sudo reboot

2. Quick start with Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1:8b        # pulls a 4-bit build by default
# In another terminal:
ollama ps                      # shows whether the model runs on GPU or CPU
nvidia-smi                     # shows actual VRAM usage

You want to see "100% GPU" in the ollama ps output. A split such as "CPU/GPU" means the model did not fit and part of it moved to system RAM; choose a smaller quantization or a shorter context.

3. Fine control with llama.cpp

If you want to set the context length, the number of layers offloaded to the GPU and the KV cache precision yourself, the llama.cpp server is a good fit:

# -ngl 99 : offload all layers to the GPU
# -c 16384: 16K context
./llama-server -m models/model-Q4_K_M.gguf -ngl 99 -c 16384 \
  --host 127.0.0.1 --port 8080

If the model does not fit, lower -ngl to keep some layers on the CPU, keeping in mind that every layer you move slows generation down.

4. Measure real usage and write it down

nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv -l 5

Watch this while running a typical workload, such as summarizing a long document. Keeping peak usage below 90% of total VRAM is a good threshold for avoiding out-of-memory errors under load.

Security: keep your local LLM off the open internet

One of the biggest benefits of a self-hosted model is that your data stays on your own server. That benefit disappears if the inference service is left unprotected.

• Ollama listens only on 127.0.0.1:11434 by default and has no authentication. Think twice before exposing it with OLLAMA_HOST=0.0.0.0.
• If you need remote access, put an authenticating reverse proxy (Nginx with basic auth or SSO) in front of it and use HTTPS.
• Allow only your own office IP addresses in the firewall:

sudo ufw default deny incoming
sudo ufw allow 22/tcp
sudo ufw allow from 203.0.113.10 to any port 443 proto tcp
sudo ufw enable

• Use SSH keys instead of passwords and disable root login. Our Linux server security guide walks through it.
• Back up model configuration regularly. Base model files can be downloaded again; your fine-tuned weights and RAG database cannot.

Data protection and a self-hosted AI server

When you feed internal documents, customer conversations or employee records into an AI model, that data may well include personal data. Running the model on a server you rent in Türkiye keeps prompts away from third-party AI services and avoids transferring them abroad, which reduces risk. It does not make you compliant on its own: access authorization, processing logs, encryption, retention periods and transparency duties remain the organization's responsibility, and the server is only one part of the technical measures. For the broader picture, read our guide to data protection and in-country hosting. This section is general information, not legal advice.

Five common VRAM sizing mistakes

1. Looking only at the file size: A 9 GB model file does not mean a 10 GB card is enough. Add the KV cache, runtime and concurrent requests and you are quickly at 12-13 GB.
2. Maxing out the context window: Enabling the longest context a model supports reserves gigabytes you may never use. Measure how many tokens your job really needs and set the context accordingly.
3. Assuming FP16 is required: For inference, 8-bit or a good 4-bit quantization is usually enough. FP16 weights are mainly needed for fine-tuning or quality comparisons.
4. Accepting partial CPU offload: Even a few layers left in system RAM slow generation down noticeably. In production, aim for the whole model to run in VRAM.
5. Forgetting the other models: If the same card also hosts an embedding, image generation or speech recognition model next to the LLM, add each one's memory to the total.

GPU rental checklist

Answer these questions before you order:

1. What is the largest model I will run, and at which quantization does it still give acceptable quality?
2. How much context do I need (4-8K for chat, 32K or more for document work)?
3. How many users or requests will run at the same time?
4. Have I added at least 15-20% headroom on top of the VRAM the formula gave me?
5. Is the card's architecture supported by my framework (Ollama, llama.cpp, vLLM, PyTorch)?
6. Is there enough system RAM and NVMe space for the model files and possible spill-over?
7. Is the load continuous, and does a GPU VDS or a physical GPU server suit it better?
8. How will the service be exposed, and are authentication and firewall rules ready?
9. Does my data contain personal data, and do the location and access controls meet my requirements?

Once you have filled in this list, the VRAM class you need is usually obvious. If you are torn between two, testing one class down and measuring real usage is smarter than renting a bigger card than you need.

Related Guides

• What is a GPU VDS and what is it used for?
• NVIDIA RTX 5060 Ti server rental for AI inference
• Installing Docker Compose on an Ubuntu server
• Dedicated server plans
• VDS plans

Frequently Asked Questions