- Shell executor
- Docker executor
- Docker Machine executor
- Kubernetes executor
- Validate that GPUs are enabled
Using Graphical Processing Units (GPUs)
- Introduced in GitLab Runner 13.9.
GitLab Runner supports the use of Graphical Processing Units (GPUs). The following section describes the required configuration to enable GPUs for various executors.
Shell executor
No runner configuration is needed.
Docker executor
Use thegpusconfiguration option in therunners.dockersection.
For example:
[runners.docker]
gpus="all"
Docker Machine executor
See thedocumentation for the GitLab fork of Docker Machine.
Kubernetes executor
No runner configuration should be needed. Be sure to check that the node selector chooses a node with GPU support.
GitLab Runner has beentested on Amazon Elastic Kubernetes Service withGPU-enabled instances.
Validate that GPUs are enabled
You can use runners with NVIDIA GPUs. For NVIDIA GPUs, one
way to ensure that a GPU is enabled for a CI job is to runnvidia-smi
at the beginning of the script. For example:
train:
script:
-nvidia-smi
If GPUs are enabled, the output ofnvidia-smidisplays the available devices. In
the following example, a single NVIDIA Tesla P4 is enabled:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla P4 Off | 00000000:00:04.0 Off | 0 |
| N/A 43C P0 22W / 75W | 0MiB / 7611MiB | 3% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
If the hardware does not support a GPU,nvidia-smishould fail either because
it’s missing or because it can’t communicate with the driver:
modprobe: ERROR: could not insert'nvidia':No such device
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.