写在前方：ubuntu系统，显卡重启后驱动失效，显卡不可用。网上冲浪之后得以有效解决，以下是解决方案
- 查看显卡nvidia-smi；驱动失效消息：
(base) root@node:~# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
-
驱动失效原因：
系统内核升级，与原驱动信息不匹配 -
解决办法：
不建议重新安装驱动，可通过DKMS(Dynamic Kernel Module Support)修复，它能够维护内核外的驱动程序，并且在内核版本变化后自动生成新的模块。
１、下载dkms，apt-get install dkms：
(base) root@node:~# apt-get install dkms
２、查看驱动版本信息ls /usr/src |grep nvidia：
(base) root@node:~# ls /usr/src |grep nvidia
nvidia-550.90.07
３、使用dkms修复：
(base) root@node:~# dkms install -m nvidia -v 550.90.07
４、检查驱动是否可用：nvidia-smi
(base) root@node:~# nvidia-smi
Fri Jul 12 06:00:52 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A800 80GB PCIe Off | 00000000:4B:00.0 Off | 0 |
| N/A 41C P0 68W / 300W | 1MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA A800 80GB PCIe Off | 00000000:65:00.0 Off | 0 |
| N/A 43C P0 68W / 300W | 1MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA A800 80GB PCIe Off | 00000000:B1:00.0 Off | 0 |
| N/A 42C P0 71W / 300W | 1MiB / 81920MiB | 3% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA A800 80GB PCIe Off | 00000000:E3:00.0 Off | 0 |
| N/A 48C P0 74W / 300W | 1MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------++-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
参考资料：
https://blog.csdn.net/trainingVIP/article/details/137789875