How-To Manually Test Probes

Cubert · Post by **Cubert** » Thu Feb 02, 2023 4:01 pm

As for the batter showing a 5, that maybe a script issue not capturing state correctly for that item. I will need to go in to python script and see how they are extracting that information.

mspguyoi · Post by **mspguyoi** » Fri Feb 03, 2023 5:16 pm

I checked the CIM data but there is no such element called "HP Smart Array P440ar RAID Controller : Embedded : HPSA1" in the CIM data window.

: Screenshot 2023-02-03 121546.png (68.4 KiB) Viewed 12844 times

I sorted by Hardware item

Cubert · Post by **Cubert** » Tue Feb 07, 2023 2:40 pm

interesting....

I have run your raw data through the Automat script parser and this is what I get back.

( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Management Module 6:1', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Board 7:1', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Board 7:2', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Add-in Card 11:1', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Add-in Card 11:2', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Add-in Card 11:3', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Add-in Card 11:4', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Add-in Card 11:5', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Add-in Card 11:6', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Internal Expansion Board 16:1', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Internal Expansion Board 16:2', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Internal Expansion Board 16:3', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Hardware Management Controller (Node 0)', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'HP Smart Array P440ar RAID Controller : Embedded : HPSA1', '20')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Board 1 PwrMeter Output', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Board 1 Power Meter', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 2 PS 2 Output', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 2 Power Supply 2: Failure status', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 2 Power Supply 2: Power Supply AC lost', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 1 PS 1 Output', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 1 Power Supply 1: Failure status', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 1 Power Supply 1: Power Supply AC lost', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 7 Fan 7 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 7 Fan 7', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 6 Fan 6 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 6 Fan 6', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 5 Fan 5 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 5 Fan 5', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 4 Fan 4 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 4 Fan 4', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 3 Fan 3 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 3 Fan 3', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 3 37-Fuse', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Battery 2 36-Storage Batt', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Other 4 35-I/O Zone', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Add-in Card 6 34-HD Cntlr Zone', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Internal Expansion Board 2 32-PCI 2 Zone', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Internal Expansion Board 1 31-PCI 1 Zone', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Management Module 1 30-iLO Zone', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Battery 1 29-Battery Zone', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Other 3 28-P/S 2 Zone', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Other 2 27-Front Ambient', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Add-in Card 4 24-HD Controller', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 2 20-PS 2 Internal', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 1 19-PS 1 Internal', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Module (DC-to-DC) 4 16-VR P1 Mem', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Module (DC-to-DC) 3 15-VR P1 Mem', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Module (DC-to-DC) 1 13-VR P1', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 2 12-PS 2 Inlet', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 1 11-PS 1 Inlet', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Other 1 10-Chipset', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 1 08-HD Max', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Memory Device 2 05-P1 DIMM 7-12', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Memory Device 1 04-P1 DIMM 1-6', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Other 1 02-CPU 1', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Other 1 01-Inlet Ambient', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Proc 1 Level-2 Cache', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Proc 1 Level-3 Cache', '0')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Memory', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Proc 1', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'IPMI SEL', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Board 2 Memory Status: Uncorrectable ECC', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Board 2 Memory Status: Correctable ECC logging limit reached', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Battery 1 Megacell Status: Failed', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan Device 8 Fans', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 3 Power Supplies', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 7 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 7', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 6 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 6', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 5 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 5', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 4 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 4', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 3 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Fan 3', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 1', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Power Supply 2', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk 1 on HPSA1 : Port 1I Box 1 Bay 1 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk 2 on HPSA1 : Port 1I Box 1 Bay 2 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk 3 on HPSA1 : Port 1I Box 1 Bay 3 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk 4 on HPSA1 : Port 1I Box 1 Bay 4 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk 5 on HPSA1 : Port 2I Box 1 Bay 5 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk 6 on HPSA1 : Port 2I Box 1 Bay 6 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk 7 on HPSA1 : Port 2I Box 1 Bay 7 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Disk 8 on HPSA1 : Port 2I Box 1 Bay 8 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'HP Smart Array P440ar RAID Controller : Embedded : HPSA1', '20')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Logical Volume 1 on HPSA1 : RAID 1 + 0 : 3353GB : Disk 1,2,3,4,5,6,7,8', '5')
,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'Battery on HPSA1', '0')

As you see above we have a line for the failed array going into your database as a query.

,( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'HP Smart Array P440ar RAID Controller : Embedded : HPSA1', '20')

So for that not to be there in the table when you go to read it out is a scratch you head moment.

But,
I suspect that its the "Ignore" in our SQL statement that would allow record to be discarded if it did not meet table requirements.

Code: Select all

INSERT IGNORE INTO `plugin_p4l_vmware_healthmon_cimdata` (`ClientID`,`LocationID`,`ProbeID`,`ESXHost`,`CIM_Item`,`CIM_Value`) VALUES @CIMSQLDATA@ ON DUPLICATE KEY UPDATE CIM_Value=VALUES(CIM_Value)

So lets try this, Place the following code block into your SQL Yog and execute the sql query, see if it yells at you for any reason. If not, go check you CIM data in the plugin now for this item to be available. otherwise I suspect you will receive an error for some reason,

Code: Select all

INSERT INTO `plugin_p4l_vmware_healthmon_cimdata` (`ClientID`,`LocationID`,`ProbeID`,`ESXHost`,`CIM_Item`,`CIM_Value`) VALUES 
( %clientid% , %locationid% , %computerid% , '@sqlESXHost@' , 'System Management Module 6:1', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Board 7:1', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Board 7:2', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Add-in Card 11:1', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Add-in Card 11:2', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Add-in Card 11:3', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Add-in Card 11:4', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Add-in Card 11:5', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Add-in Card 11:6', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Internal Expansion Board 16:1', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Internal Expansion Board 16:2', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Internal Expansion Board 16:3', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Hardware Management Controller (Node 0)', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'HP Smart Array P440ar RAID Controller : Embedded : HPSA1', '20')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Board 1 PwrMeter Output', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Board 1 Power Meter', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 2 PS 2 Output', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 2 Power Supply 2: Failure status', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 2 Power Supply 2: Power Supply AC lost', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 1 PS 1 Output', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 1 Power Supply 1: Failure status', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 1 Power Supply 1: Power Supply AC lost', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 7 Fan 7 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 7 Fan 7', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 6 Fan 6 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 6 Fan 6', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 5 Fan 5 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 5 Fan 5', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 4 Fan 4 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 4 Fan 4', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 3 Fan 3 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 3 Fan 3', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 3 37-Fuse', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Battery 2 36-Storage Batt', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Other 4 35-I/O Zone', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Add-in Card 6 34-HD Cntlr Zone', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Internal Expansion Board 2 32-PCI 2 Zone', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Internal Expansion Board 1 31-PCI 1 Zone', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Management Module 1 30-iLO Zone', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Battery 1 29-Battery Zone', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Other 3 28-P/S 2 Zone', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Other 2 27-Front Ambient', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Add-in Card 4 24-HD Controller', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 2 20-PS 2 Internal', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 1 19-PS 1 Internal', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Module (DC-to-DC) 4 16-VR P1 Mem', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Module (DC-to-DC) 3 15-VR P1 Mem', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Module (DC-to-DC) 1 13-VR P1', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 2 12-PS 2 Inlet', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 1 11-PS 1 Inlet', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Other 1 10-Chipset', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 1 08-HD Max', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Memory Device 2 05-P1 DIMM 7-12', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Memory Device 1 04-P1 DIMM 1-6', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Other 1 02-CPU 1', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Other 1 01-Inlet Ambient', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Proc 1 Level-2 Cache', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Proc 1 Level-3 Cache', '0')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Memory', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Proc 1', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'IPMI SEL', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 8 C1 P2I Bay 8: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 7 C1 P2I Bay 7: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 6 C1 P2I Bay 6: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 5 C1 P2I Bay 5: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 4 C1 P1I Bay 4: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 3 C1 P1I Bay 3: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 2 C1 P1I Bay 2: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Drive Fault', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Predictive Failure', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Parity Check In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: In Critical Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: In Failed Array', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Rebuild In Progress', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk or Disk Bay 1 C1 P1I Bay 1: Rebuild Aborted', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Board 2 Memory Status: Uncorrectable ECC', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'System Board 2 Memory Status: Correctable ECC logging limit reached', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Battery 1 Megacell Status: Failed', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan Device 8 Fans', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 3 Power Supplies', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 7 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 7', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 6 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 6', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 5 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 5', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 4 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 4', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 3 DutyCycle', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Fan 3', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 1', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Power Supply 2', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk 1 on HPSA1 : Port 1I Box 1 Bay 1 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk 2 on HPSA1 : Port 1I Box 1 Bay 2 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk 3 on HPSA1 : Port 1I Box 1 Bay 3 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk 4 on HPSA1 : Port 1I Box 1 Bay 4 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk 5 on HPSA1 : Port 2I Box 1 Bay 5 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk 6 on HPSA1 : Port 2I Box 1 Bay 6 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk 7 on HPSA1 : Port 2I Box 1 Bay 7 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Disk 8 on HPSA1 : Port 2I Box 1 Bay 8 : 838GB : Data Disk', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'HP Smart Array P440ar RAID Controller : Embedded : HPSA1', '20')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Logical Volume 1 on HPSA1 : RAID 1 + 0 : 3353GB : Disk 1,2,3,4,5,6,7,8', '5')
,( %clientid% , %locationid% , %computerid% ,  '@sqlESXHost@' , 'Battery on HPSA1', '0')
 ON DUPLICATE KEY UPDATE CIM_Value=VALUES(CIM_Value)

You will need to copy the above query into a text editor and globally replace the values with real actual values of your client, location, computerid and esxhost. Esxhost must be in single ticks but clientID, LocationID and ComputerID(Probe) can be without.

%clientid% , %locationid% , %computerid% , '@sqlESXHost@'

mspguyoi · Post by **mspguyoi** » Tue Feb 07, 2023 3:59 pm

I tried to run the SQL query and that worked without any errors.

The CIM data now shows the RAID controller failure.

: Screenshot 2023-02-07 105749.png (17.65 KiB) Viewed 12800 times

The overal device status shows okay still but I guess that is due to the fact we manually inserted the data into the table.

mspguyoi · Post by **mspguyoi** » Wed Feb 08, 2023 3:15 pm

I noticed this morning that now the same server is showing this:

: Screenshot 2023-02-08 101426.png (40.94 KiB) Viewed 12787 times

Any idea why that is?

Cubert · Post by **Cubert** » Thu Feb 09, 2023 2:03 pm

That's a head scratcher..

Make sure the dates of the data are changing as probe updates... so we know its fresh data.. Monitor it a few days to see if stops updating.

mspguyoi · Post by **mspguyoi** » Thu Feb 09, 2023 5:30 pm

I've checked the data and it is updating for all the 4 servers.

I also found out that 3 out of the 4 servers listed in my previous snippet have battery issues and but only the one esxi101, that we have been working on, is showing it.

The 4th server has a battery status of unknown
I verified that by running the check_esxi_hardware.py script but what I noticed is that there are 2 items for battery

20230209 11:56:10 Element Name = Battery 1 Megacell Status: Failed
20230209 11:56:10 Element HealthState = 5
20230209 11:56:10 Element Name = Battery 1 Megacell Status: Presence Detected
20230209 11:56:11 Check classe VMware_Battery
20230209 11:56:11 Element Name = Battery on HPSA1
20230209 11:56:11 Element HealthState = 0

The HP battery status I don't see in the CIM data for that server and I check SQL CIM data table and I don't see it there either.
Is that HP data not used?

Cubert · Post by **Cubert** » Tue Feb 14, 2023 4:52 pm

This is where it gets me head scratching...

It should be reporting all items with a Element Name and a Element HealthState value. When we run the script and view its returns we are seeing this data. If we do exactly what the Automate script does (cut and paste), we get correct data in CIM view.

So not sure why it wouldn't be there when script runs successfully. ( no failures ).

You are using the same probe, same python script and same commands. Best I can offer currently is to walk through each server as we did with the first and force a population update and see if that clears the issues out.

I'll keep poking at it here to see if I can duplicate the issue.

mspguyoi · Post by **mspguyoi** » Wed Feb 15, 2023 10:21 pm

They all use the same probe but I can try to use a different probe as maybe it is something on the probe that's causing it.
You thinks that's worth a try?

Cubert · Post by **Cubert** » Fri Feb 17, 2023 8:08 pm

Actually try deleting on the probe the python directory on c:, allow probe to rerun and it will reinstall python and update the esx_health_check.py script during that time. It maybe that your script was out of date on probe.

Support Forum

How-To Manually Test Probes

Re: How-To Manually Test Probes

Re: How-To Manually Test Probes

Re: How-To Manually Test Probes

Re: How-To Manually Test Probes

Re: How-To Manually Test Probes

Re: How-To Manually Test Probes

Re: How-To Manually Test Probes

Re: How-To Manually Test Probes

Re: How-To Manually Test Probes

Re: How-To Manually Test Probes