Turkey (Türkçe) 
Running your own language model on your own server is the most direct way to do chat, summarization and code assistance without sending your data to a third-party AI service. The software you need is no longer complicated: Ollama downloads and runs the models, and Open WebUI provides a chat interface you can use from the browser. The hard part is taking the right steps in order: the driver, Docker, the service settings and, above all, safe access. In this guide we start from scratch on a GPU server running Ubuntu 24.04 LTS and build a working local AI environment without leaving a door open to the internet.
The setup has three parts. The first is the NVIDIA driver; the card cannot compute without it. The second is the Ollama service, installed directly on the server; it manages model files and uses the graphics card to generate answers. The third is Open WebUI running in a Docker container; it connects to Ollama over the local address and gives you a browser interface. We expose none of it to the internet: Ollama listens only on the local address, and you reach Open WebUI only through an SSH tunnel, behind the firewall.
Which card you use does not change the examples; only the models that fit your VRAM change. For the calculation, see our guide on how much VRAM a local LLM needs.
Connect to the server and update the packages first. Then set up a firewall and allow SSH only. To make SSH access key-based, follow our Linux server security guide.
sudo apt update && sudo apt -y upgrade
sudo apt -y install ufw curl ca-certificates
sudo ufw allow OpenSSH
sudo ufw enableAt this point we open neither the Ollama port nor the Open WebUI port; that is a deliberate choice.
Ubuntu recommends the driver package that suits your card. First see which package is recommended, then install it and reboot:
sudo ubuntu-drivers devices
sudo ubuntu-drivers install
sudo rebootWhen the server is back, you should see the card with this command:
nvidia-smiThe output shows the card name, the amount of VRAM and the driver version. Two points matter. Newer generation cards such as the RTX 50 series need a sufficiently recent driver and, in most cases, its open kernel module branch; so rather than picking a version by hand, use the recommendation in the ubuntu-drivers devices output. For older generation cards such as the Tesla P40 the opposite applies: the driver branch may one day stop supporting that generation, so pin the working version with sudo apt-mark hold and update deliberately. If nvidia-smi fails, see our Ubuntu 24.04 guide and the driver troubleshooting article in this series.
We only need Docker for Open WebUI. Ubuntu's own package is enough:
sudo apt -y install docker.io
sudo systemctl enable --now docker
sudo docker run --rm hello-worldIf you plan to build multi-container setups with Docker Compose, our Docker Compose setup guide will help. If you also want to run Ollama inside Docker, you would need the NVIDIA Container Toolkit; we avoid that extra layer by installing Ollama directly on the server.
The official install script installs Ollama and starts it as a systemd service. By default the service listens only on the local address:
curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama --no-pager
ollama pull llama3.1:8b
ollama run llama3.1:8bThe model name is an example; choose a current model from the Ollama library that fits your VRAM. While the model is running, run these in another terminal:
ollama ps
nvidia-smiThe PROCESSOR column in the ollama ps output tells you where the model runs. If it runs entirely on the GPU you see 100 percent GPU; if you see a CPU and GPU mix, the model did not fully fit in VRAM and has slowed down. In that case choose a smaller or more heavily quantized model. A local API is ready for programmatic access:
curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"Hello, introduce yourself.","stream":false}'Open WebUI is a full-featured chat interface built for Ollama. Two details matter here. First, because Ollama listens only on 127.0.0.1, an ordinary container on the bridge network cannot reach it; that is why we run the container on the host's network (--network=host). Second, ports that Docker publishes, such as -p 3000:8080, bypass ufw rules and can expose the container to the internet without you noticing. A container on the host network follows the normal firewall rules instead; since only SSH is open in ufw, the interface cannot be reached from outside:
sudo docker run -d --name open-webui --restart always \
--network=host \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:mainOnce the container starts, the interface listens on port 8080 on the server. Verify it is running with the following commands, and separately check that it is not visible from outside:
sudo docker ps
curl -I http://127.0.0.1:8080
sudo ufw statusTo reach the interface from your own computer, use an SSH tunnel. While the connection stays open, localhost:3000 on your computer is forwarded to Open WebUI on port 8080 of the server:
ssh -L 3000:127.0.0.1:8080 user@server-addressOpen http://localhost:3000 in your browser. The first account you create becomes the administrator, so create it yourself right away. To stop others from signing up, recreate the container with the -e ENABLE_SIGNUP=false variable. If you will share it with a team, prefer a reverse proxy that provides HTTPS and adds authentication instead of a tunnel; an AI interface with an open door and a weak password is an attractive target for attackers.
Build your own AI server in Istanbul
On GPU servers delivered with a physical GPU, full root access and Gen4/Gen5 M.2 NVMe storage, you can have Ollama and Open WebUI running in minutes.
Browse GPU server packagesYou change Ollama's behaviour by adding environment variables to its systemd service. If you will change the model directory, create it first and give it to the Ollama user, then open the override file:
sudo mkdir -p /data/ollama-models
sudo chown ollama:ollama /data/ollama-models
sudo systemctl edit ollamaIn the editor that opens, write for example the following, save, and restart the service:
[Service]
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_MODELS=/data/ollama-models"sudo systemctl daemon-reload
sudo systemctl restart ollamaOLLAMA_KEEP_ALIVE sets how long a model stays in memory after the last request, which stops a frequently used model from being reloaded every time. OLLAMA_NUM_PARALLEL limits the number of concurrent requests. OLLAMA_MODELS tells Ollama where to keep model files; a separate directory on your large NVMe disk keeps things tidy.
Use VRAM as your reference when picking a model. As a rough guide, models of about 8 billion parameters quantized to 4 bits fit comfortably in 8 GB; models of 14 billion parameters suit 16 GB; models of around 30 billion parameters fit 24 GB, and 32-billion-parameter models sit more comfortably in 32 GB once you allow room for context. Memory use grows as you raise the context length, so leave headroom in these numbers. To raise the context you can use a PARAMETER num_ctx 8192 line in a Modelfile.
The quality and speed of a locally running model depend largely on graphics card memory. The table below gives a rough starting point; real values change with the model, the quantization level and the context length.
| Card memory | Suitable model class | Note |
|---|---|---|
| 8 GB | About 7-8 billion parameters, 4-bit | Enough for chat and summarization; leave headroom for long contexts |
| 16 GB | About 14 billion parameters, 4-bit | A balanced starting point for daily use |
| 24 GB | About 30 billion parameters, 4-bit | Stronger reasoning and coding ability |
| 32 GB | About 32 billion parameters, higher quality | Runs comfortably with context headroom |
The same model comes in different quantization versions: a lower bit count reduces memory but costs a little quality. Start with the 4-bit version in your first trial and move up a level if quality is not enough. You can also read about which card fits you in the inference section of our RTX 5060 Ti guide.
Three scenarios make this environment genuinely useful. The first is an in-house chat assistant: employees draft text, summarize emails and fix wording without company information leaving the building. The second is document summarization and question answering: with Open WebUI's document upload you can ask questions about a contract, a report or a technical document, and the documents stay on your server. The third is a coding helper: with editor extensions that connect to Ollama's local API you get code completion and explanations. The shared benefit in all three is the same: data does not leave your own server, and there is no per-use fee to an outside service.
ufw is enabled and allows only the ports you need.ufw is bypassed for ports published by Docker, you run the container on the host network or bound only to 127.0.0.1.In a production environment the services must come back on their own when the server is restarted for maintenance. Ollama is a systemd service enabled by the install script, and the Open WebUI container starts with --restart always, so it returns by itself when Docker starts. It is a good habit to test this once on purpose: restart the server, and a few minutes later confirm with nvidia-smi, systemctl is-active ollama and sudo docker ps that everything is in place. The first request will be a little slow because the model has to be loaded into memory again; that is normal.
ollama ps to confirm the model has spilled to the CPU; try a smaller model or a lower context.journalctl -u ollama -e shows the error line. If the driver is not loaded, fix nvidia-smi first.--network=host and that the connection address in the Open WebUI settings is http://127.0.0.1:11434.ollama list and delete the ones you do not use with ollama rm model-name.dkms status that the module was built and reinstall the driver package if needed.Open WebUI chats and settings live in the Docker volume named open-webui; Ollama models can be downloaded again, so backing them up is not essential. What matters is your own settings and, if you use one, your knowledge base documents. Archive the volume regularly and keep the archive off the server. To update, stop the container, pull the new image and start it again with the same volume; the data stays in the volume, so your chats are preserved. Do kernel and driver updates in a maintenance window, then confirm with nvidia-smi that the card is visible.
Need a bigger card for several users?
More VRAM means larger models and more concurrent users. Choose the card and package that match your needs, or ask for a quote.
Go to the GPU server quote formHow do I know Ollama is using the GPU?
While a model is running, run the ollama ps command. If the PROCESSOR column shows 100 percent GPU, the model runs entirely on the graphics card. A CPU/GPU mix means the model did not fully fit in VRAM.
Do I need Docker for Ollama?
No. Ollama can be installed directly on the server and uses the GPU on its own. We use Docker only to run the Open WebUI interface in isolation, and that container does not need the GPU.
How do I reach Open WebUI safely from outside?
Allow only SSH in the firewall and connect to the interface through an SSH tunnel. If you need a public address, put it behind a reverse proxy that provides HTTPS, turn off sign-ups and set a strong administrator password.
Can several users use Ollama at the same time?
Yes, but every concurrent request needs extra memory and compute. You can cap concurrent requests with the OLLAMA_NUM_PARALLEL setting, and choose a card with more VRAM for heavy use.
Which models fit in 8 GB of VRAM?
Models of about 7 to 8 billion parameters, quantized to 4 bits, fit comfortably in 8 GB of VRAM. Memory use grows as you raise the context length, so leave some headroom.
What if the GPU disappears after a driver update?
After a kernel update the module may need to be rebuilt. Check the output of dkms status, confirm the linux-headers package is installed and, if needed, reinstall the driver package and reboot the server.
Can I move the models to another disk?
Yes. Set the OLLAMA_MODELS environment variable to a new directory and restart the service, and models will download there. On servers with a large NVMe disk this keeps things tidy.
