NVIDIA GPU error 확인

Xid Error Listing

XID Errors :: GPU Deployment and Management Documentation

The following table lists the Xid errors along with the potential causes for each.

XIDFailureCauses  HW ErrorDriver ErrorUser App ErrorSystem Memory CorruptionBus ErrorThermal IssueFB Corruption

 

추가적인 부연설명은 표 아래

XID              Failure      Causes  

                                 HW Error  Driver Error      User App Error

                                                                                   System Memory Corruption

                                                                                                     Bus Error

                                                                                                                    Thermal Issue FB Corruption

1	Invalid or corrupted push buffer stream		X		X	X		X
2	Invalid or corrupted push buffer stream		X		X	X		X
3	Invalid or corrupted push buffer stream		X		X	X		X
4	Invalid or corrupted push buffer stream		X		X	X		X
	GPU semaphore timeout		X	X	X	X		X
5	Unused
6	Invalid or corrupted push buffer stream		X		X	X		X
7	Invalid or corrupted push buffer address		X			X		X
8	GPU stopped processing		X	X		X	X
9	Driver error programming GPU		X
10	Unused
11	Invalid or corrupted push buffer stream		X		X	X		X
12	Driver error handling GPU exception		X
13	Graphics Engine Exception		X	X	X	X	X	X
14	Unused
15	Unused
16	Display engine hung		X
17	Unused
18	Bus mastering disabled in PCI Config Space		X
19	Display Engine error		X
20	Invalid or corrupted Mpeg push buffer		X		X	X		X
21	Invalid or corrupted Motion Estimation push buffer		X		X	X		X
22	Invalid or corrupted Video Processor push buffer		X		X	X		X
23	Unused
24	GPU semaphore timeout		X	X	X	X	X	X
25	Invalid or illegal push buffer stream		X	X	X	X		X
26	Framebuffer timeout		X
27	Video processor exception		X
28	Video processor exception		X
29	Video processor exception		X
30	GPU semaphore access error		X
31	GPU memory page fault		X	X
32	Invalid or corrupted push buffer stream		X		X	X	X	X
33	Internal micro-controller error		X
34	Video processor exception		X
35	Video processor exception		X
36	Video processor exception		X
37	Driver firmware error		X		X	X
38	Driver firmware error		X
39	Unused
40	Unused
41	Unused
42	Video processor exception		X
43	GPU stopped processing		X	X
44	Graphics Engine fault during context switch		X
45	Preemptive cleanup, due to previous errors -- Most likely to see when running multiple cuda applications and hitting a DBE		X
46	GPU stopped processing		X
47	Video processor exception		X
48	Double Bit ECC Error	X
49	Unused
50	Unused
51	Unused
52	Unused
53	Unused
54	Auxiliary power is not connected to the GPU board
55	Unused
56	Display Engine error	X	X
57	Error programming video memory interface	X	X					X
58	Unstable video memory interface detected	X	X
	EDC error – clarified in printout	X
59	Internal micro-controller error (older drivers)		X
60	Video processor exception		X
61	Internal micro-controller breakpoint/warning (newer drivers)
62	Internal micro-controller halt (newer drivers)	X	X				X
63	ECC page retirement recording event	X	X					X
64	ECC page retirement recording failure	X	X
65	Video processor exception	X	X
66	Illegal access by driver		X	X
67	Illegal access by driver		X	X
68	Video processor exception	X	X
69	Graphics Engine class error	X	X
70	CE3: Unknown Error	X	X
71	CE4: Unknown Error	X	X
72	CE5: Unknown Error	X	X
73	NVENC2 Error	X	X
74	NVLINK Error	X	X			X
75	Reserved
76	Reserved
77	Reserved
78	vGPU Start Error		X
79	GPU has fallen off the bus	X	X		X	X	X
80	Corrupted data sent to GPU	X	X		X	X		X
81	VGA Subsystem Error	X
82	Reserved
83	Reserved
84	Reserved
85	Reserved
86	Reserved
87	Reserved
88	Reserved
89	Reserved
90	Reserved
91	Reserved
92	High single-bit ECC error rate	X	X

 

nvidia gpu는 불량 판단이 굉장히 까다롭고 애매한 부분이 있다.

nvidia-smi 명령어를 사용했을 경우

 

 

위와 같이 Pwr ERR!가 떠 있다.

 

# nvidia-smi -q

    Power Readings
        Power Management            : Supported
        Power Draw                  : 54.16 W
        Power Limit                 : 250.00 W
        Default Power Limit         : 250.00 W
        Enforced Power Limit        : 250.00 W
        Min Power Limit             : 100.00 W
        Max Power Limit             : 280.00 W

 

위와 같이 nvidia-smi -q를 사용하였을 때 power reading이 지금은 정상적이지만 이상이 생길 때는 N/A로 많이

바뀌어 있을 것이다.

 

 

이런경우 몇 가지 경우가 있는데 물론 GPU를 바로 교체하면 되겠지만 생각보다 nvidia에서 불량을 판단하는데 까다롭다. 시스템을 재부팅할 경우 GPU가 정상적으로 돌아오기 때문이다.

 

하지만 /var/log/messages에 아래와 같은 log가 남아있다면 의심해 볼 여지는 충분한다.

 

Dec  4 19:50:58 srp40-0009 kernel: NVRM: GPU at PCI:0000:0c:00: GPU-158895f5-b658-e8d2-6b21-825840d08d4a
Dec  4 19:50:58 srp40-0009 kernel: NVRM: GPU Board Serial Number: 0324817091877
Dec  4 19:50:58 srp40-0009 kernel: NVRM: Xid (PCI:0000:0c:00): 79, GPU has fallen off the bus.
Dec  4 19:50:58 srp40-0009 kernel: NVRM: GPU at 00000000:0c:00.0 has fallen off the bus.

Xid 79 - 5개의 원인에 해당이 된다.

h/w, driver, user app, bus error, thermal 등

보통 nvrm xid error가 뜨면 gpu를 교체하는게 맞다.

 

이 외에도 nvidia-smi시 gpu drop이 되는 경우도 있고 여러 가지가 있다.

확실한 error시 messages쪽을 확인해 볼 수 있도록 하자.

 

하지만 nvidia에서는 사실 NVIDIA Field Diagnostic를 통해 rma신청이 되며 저 툴에서 error가 뜨지 않으면 gpu 불량으로 보지 않는다. 참 그렇다....ㅋㅋ