Turkey (Türkçe) 
Almost every problem with a GPU server starts with the nvidia-smi command. If it shows your card and its memory, the basics are working; if it throws an error, something is wrong with the driver, the kernel, the power or the hardware. The trouble is that the error messages are short and their causes vary widely: the same sentence is sometimes just a side effect of a kernel update and sometimes the first sign of a hardware failure. In this guide we cover the six most common problems on GPU servers (a driver communication error, a card that does not appear, Xid errors, thermal throttling, full VRAM and GPU in Docker), showing which command to use to look at what. At the end you will also find what information to send to the support team.
Before you start debugging, knowing what a healthy output looks like makes the job easier. When you run the command you see the driver and CUDA versions at the top, then one row per card. The fields in that row mean this:
To watch continuously, nvidia-smi -l 2 refreshes the output every two seconds. For a more readable live view you can install the nvtop package (sudo apt install nvtop).
This message means nvidia-smi cannot talk to the driver module. The most common cause is a kernel update: when Ubuntu installs a new kernel, the NVIDIA module has to be rebuilt for it, and if the build fails or the kernel headers are missing, the module does not load. The second common cause is Secure Boot: if it is on, an unsigned module will not load. Third, the open source nouveau driver may have claimed the card. Look at these in order:
uname -r
lsmod | grep -i nvidia
dkms status
sudo dmesg -T | grep -i -E "nvrm|nvidia" | tail -20
mokutil --sb-stateuname -r shows the running kernel, and dkms status shows which kernel the NVIDIA module was built for. If there is no "installed" line for the running kernel, rebuild the module:
sudo apt -y install linux-headers-$(uname -r)
sudo dkms autoinstall
sudo modprobe nvidia
nvidia-smiIf it still does not work, reinstall the driver package and reboot the server. If mokutil --sb-state says "SecureBoot enabled", the unsigned module may have been blocked; in that case you need to sign the module or turn off Secure Boot. To change that setting on your server you may need the hardware management console; see our HTML5 KVM console guide.
This message says the driver is working but found no card. Start by checking whether the card appears in the system at all:
lspci | grep -i nvidia
lspci -nn | grep -i -E "vga|3d"If the card is in this list but nvidia-smi cannot find it, the driver and the card do not match; review the driver version and apply the ubuntu-drivers devices recommendation. If the card does not show up in lspci at all, the problem is not in software: it may be the PCIe connection, the power supply or the card itself. In that case do not waste time on reinstalling; collect the output and open a support ticket.
A working server suddenly losing its GPU one day usually comes from automatic updates that run at night. The kernel is updated, the NVIDIA module cannot be built for the new kernel, and when the server reboots the card is gone. To prevent this on a production server, pin the working driver and kernel packages and do updates deliberately in a maintenance window:
dpkg -l | grep -E "nvidia-driver|linux-image|linux-headers"
sudo apt-mark hold linux-image-generic linux-headers-generic
sudo apt-mark hold nvidia-driver-XXXReplace nvidia-driver-XXX here with your own version number. When you want to update, remove the lock with sudo apt-mark unhold, update, reboot and verify with nvidia-smi. On older generation cards such as the Tesla P40, this habit matters even more, because new driver branches may stop supporting that generation.
The driver writes problems on the card to the kernel log with "Xid" codes. Search for them with this command:
sudo dmesg -T | grep -i xidThe meaning of the codes is detailed by driver version and documented in NVIDIA's Xid error catalogue; still, two groups are worth knowing. Application-caused errors (for example a program's invalid memory access) usually affect only that program; updating it or trying a different version is enough. Hardware or bus-caused errors are more serious. The best known is Xid 79: "GPU has fallen off the bus", meaning the card has dropped off the bus. Communication with the card was cut because of power, heat or the PCIe connection, and it often does not recover without a restart. If you see Xid 79, do not fiddle with software; open a support ticket right away together with the dmesg output.
If your card works but is slower than expected, it may have entered throttling. When the card gets too hot or hits its power limit, it lowers its own speed. You can ask the driver for the reason:
nvidia-smi -q -d PERFORMANCE
nvidia-smi -q -d TEMPERATURE,POWERThe "Clocks Event Reasons" section (called "Clocks Throttle Reasons" in older versions) in the output of the first command shows which limit is active: "HW Thermal Slowdown" and "SW Thermal Slowdown" are heat related, while "SW Power Cap" is the power limit. A power limit is sometimes a deliberate setting and is normal. If thermal slowdown keeps showing up, there may be a problem with cooling or airflow; that is a hardware matter and should be reported to the support team. To record the temperature under load and look at it over time, use the method in the monitoring section below.
If memory use looks high and a new job fails with "CUDA out of memory", a process left in the background is most likely holding the memory. To see which process uses how much memory:
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
ps -fp 12345
sudo fuser -v /dev/nvidia*The first command lists the processes using GPU memory. In the second, write the process number you got from the list instead of 12345 to see what the process is. If it is an unneeded process, end it with kill; if it does not respond, use kill -9 as a last resort. A Jupyter notebook or a half-finished training script are the typical culprits. If you use containers, the process may appear under another name on the host; find the relevant container with docker ps.
Reach your GPU server's hardware remotely
On physical GPU servers, full root access and a browser-based IPMI/KVM console let you step in on your server even when SSH drops.
Browse GPU server packagesIf docker run --gpus all gives the error "could not select device driver with capabilities: [[gpu]]" when you want to use the GPU inside a container, Docker does not know the NVIDIA runtime. The fix is to install the NVIDIA Container Toolkit and register it with Docker. After installing the package from NVIDIA's official repository:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo docker run --rm --gpus all ubuntu nvidia-smiThe last command starts a container and runs nvidia-smi inside it; if you can see the card, the setup is complete. If you still get an error, make sure nvidia-smi works on the host; Docker cannot carry a broken driver into the container. If you build multi-container setups with Docker, our Docker Compose guide will help.
The best troubleshooting is collecting data before a problem occurs. A simple method is to write the output of an nvidia-smi query to a file at regular intervals. That way you can answer the question "what happened at three in the morning?":
nvidia-smi --query-gpu=timestamp,name,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw --format=csv -l 30 >> ~/gpu-monitor.csvYou can run this command inside tmux or turn it into a systemd service. You can open the file in a spreadsheet and look at how temperature, utilization and power change over time. For more advanced monitoring you can feed this data into a Prometheus and Grafana setup; NVIDIA's DCGM-based exporter is common for this, but some metrics may not be available on GeForce class cards. For a simple alarm, a script can write a record to the system log when a temperature threshold is exceeded.
Staying calm and following an order saves time when a problem hits. First run nvidia-smi. If it works and sees the card, the problem is most likely application or settings related: look at VRAM use, processes and throttling. If it throws an error, check with lspci whether the card appears on the bus. If the card appears, the problem is on the driver or kernel side: the dkms status and dmesg output will point the way. If the card does not appear at all, it is hardware or power, and fiddling with software is pointless. This three-branch logic puts the large majority of problems into the right category in a few minutes. Apply only the section for the branch you are on; changing everything at once hides the cause.
Some problems come not from the graphics card but from the resources around it. When the disk fills, driver and application logs cannot be written, package updates stop halfway and the server behaves oddly; that is why you should always look at the df -h output. When system RAM runs out, the kernel may kill processes, and it looks as if your GPU job crashed on its own; free -h and the "Out of memory" records in dmesg show this. Remember that loading a large model may temporarily need a lot of system memory. In that case, adding RAM or lowering the loading settings is more effective than changing the graphics card.
When you cannot solve the problem yourself, the support team needs the right information to understand it quickly. The commands below gather everything needed into a single file:
{
date
uname -a
lspci -nn | grep -i -E "nvidia|vga|3d"
nvidia-smi
nvidia-smi -q
dkms status
sudo dmesg -T | grep -i -E "nvrm|xid|nvidia" | tail -100
sudo journalctl -b -p err --no-pager | tail -100
} > ~/gpu-report.txt 2>&1Attach the resulting gpu-report.txt file to your ticket. Also write when the problem started, what you did before it (an update, new software, a restart) and which job was running. This information turns a half-hour exchange into a single message. If you cannot reach the server over SSH, you can connect through the IPMI/KVM console and run the same commands there.
nvidia-smi after an update.tmux or as a service.Not solved? Reach expert support
Write to the support team, reachable around the clock for GPU driver, CUDA and operating system topics, with your outputs attached.
Contact the support teamWhy does the nvidia-smi 'couldn't communicate with the NVIDIA driver' error appear?
It mostly happens because the NVIDIA module was not rebuilt after a kernel update, or because Secure Boot blocks the unsigned module. Looking at the dkms status and lsmod output and rebuilding the module usually fixes it.
What should I do if nvidia-smi says 'No devices were found'?
First check with lspci | grep -i nvidia whether the card appears on the PCIe bus. If the card is not listed at all, the problem is not software but hardware or power; open a support ticket and attach the output.
What does an Xid 79 error mean?
Xid 79 usually indicates that the card has dropped off the bus, meaning the system lost communication with it. Hardware causes such as power, heat or the PCIe connection are possible; if you see it, open a support ticket together with the dmesg output.
What GPU temperature is normal?
A high temperature at full load is normal, but performance drops if the card enters thermal throttling. Look at the throttle reasons in the nvidia-smi -q -d PERFORMANCE output; if you see constant heat-related throttling, tell the support team.
How do I fix the Docker 'could not select device driver' error?
The NVIDIA Container Toolkit must be installed and registered with Docker. Configure the Docker runtime with the nvidia-ctk runtime configure command, restart Docker and then test with docker run --gpus all.
Why does VRAM look full when nothing is running?
Usually a process left in the background is holding the memory. Look at the process list at the bottom of the nvidia-smi output or the compute-apps query and end the unneeded process.
What can I do so a kernel update does not break the driver?
You can pin the working driver and kernel packages with apt-mark hold and do updates deliberately in a maintenance window. Excluding kernel packages from automatic updates is also a common method.
