Troubleshooting on DataStores

Troubleshooting On VMWare DataStores

ESX hosts store all files necessary for virtual machine operation in a VMware Virtual Machine File System (VMFS). Proper capacity management is critical because this file system is required for any operations related to a virtual machine. ​This document gives more insight on VMFS and its troubleshooting through putty and GUI (Graphical user interface) First steps you need to check free space on VMFS (DS) where VM's actually saved by system. ​ Define some threshold limits in your system under VMFS (DS) to avoid last minute surprise. If you have e.g 100GB DS, define threshold limit on 85 percentage so that you can get alerts proactively by system. ​ In GUI, Select DataStore where you need to define threshold parameters e.g screen below.

Configure the alarms that associated with VMFS DS, keep 80% as "information", 90% "warning", 95% "critical". these alerts helps you in daily operations.​

Let's understand more on troubleshooting piece like

1. VMFS Maintenance

2. Performance issues

3. Virtual volumes (vVols)

4. Unable to mount datastore

5. Storage and host disconnects​

Storage and Host Disconnects

​Below are few possible causes of disconnects. ​

• Storage connectivity

• Mis-configuration

• Hardware failure

• Power outage

• Path failover

• Poor array performance​

Few important error codes in vobd log

Scsi event : H:0x0 D:0x0 P:0x0 Valid sense data: 0x0 0x0 0x0

​– H: - Host (HBA)

– D: - Device

– P: - Plugin.

Important sense codes

– H:0x5 – DID_Busy

– H:0x2 – DID_Bus_Busy

– H:0x8 – DID_Abort

Note: With the above sense codes NMP will not failover the paths to next available path. – H:0x1 – DID_No_connect Note: 0x1 indicates that the ESXi host is not able to see the device.

Logs events under /var/log/vobd.log

​[root@esxi02:/vmfs/volumes/5ea5d455-08c35095-b133-000c2957a10c/log] less vobd.log

[root@esxi02:/vmfs/volumes/5ea5d455-08c35095-b133-000c2957a10c/log] less vmkernel.log

Tools and Logs

Tools

• vSphere Performance tabs.

• Esxtop

• VscsiStats

this is a great tool for virtual machine disk I/O workload characterization. Have you ever wondered about the most common I/O size generated by the Guest OS? What about the latency of those I/Os? What about checking to see the I/O generated by a Guest OS when it is in a so-called ‘idle’ state? vscsiStats can help with all of these queries, as well as providing some excellent troubleshooting options.​

vscsiStats: Helps with ioLength, seekDistance, outstandingIOs, latency

Logs

• Vobd.log

• Syslog​

Configure syslog parameters under vSphere environment. this will collect the information from esxi host and save it in centralize location for analysis.