How to troubleshoot when LUNs are missing from ESXi Server Environment
Audio : Listen to This Blog.
LUNs are reported as missing in ESXi Server Storage Adapter.
Common symptoms encountered are:
- CHAP Secret Mismatch
- Access Control List(ACL) was not properly set to LUN
- Masking the LUN Path from ESXi Host through HBA Adapter. This may prevent ESXi Host from accessing storage devices
We have to undertake some preliminary, pre-requisite troubleshooting steps to ensure that LUN missing is not due to the above mentioned three scenarios:
Link Connectivity b/w ESXi Host and Storage Subsystem
- Link Connectivity verification using vmkping command
- Available of ESXi HBA Adapter [ in online state] using esxcli iscsi adapter list command
Note: Ensure that the HOST ID on the array for the LUN (on ESX it shows up under LUN ID) is less than 255. Any LUN that has a HOST ID greater than 255 may not be shown under ESXi Host Storage Adapters.
i. Available device list from ESXi HBA Adapter [using esxcli storage core device list command – This should not go beyond 254+2[ local device and CD Drive ]
Now that we’re done with the preliminary steps, we can move on troubleshooting in case of the scenarios as mentioned above.
Case 1: CHAP Secret Mismatch
Check for CHAP Secret configuration from Storage Sub System and ensure that the same settings are configured from ESXi/ESX Hosts too. In case it differs, LUN Discover operation will fail, leading to LUN Missing issue.
Trouble Shooting Steps:
a. tail -f /var/log/syslog.log
Note: If you see the message like “Login failed” or “Login Authentication failed”, please go ahead and verify the CHAP Secret Settings on both Storage Subsystem and ESXi Host Storage Adapter.
Chap Secret Verification CLI Commands on ESXi Host:
b.esxcli iscsi adapter auth chap get –adapter vmhba37
c. esxcli iscsi adapter auth chap set –adapter vmhba37 –direction=uni –authname=”chap” –level=required –secret=”password”
Chap Secret Verification through vCenter Client on ESXi Host:
- Navigate to: Configuration tab -> ‘Storage adapters’
- Select the software adapter
- Click property and click chap button
Case 2: Access Control List (ACL) was not properly set to LUN
On the Storage Subsystem, please ensure that the LUN IQNs and access control list (ACL) set to allow the ESXi/ESX Host HBAs to access the iSCSI Targets.
Trouble Shooting Steps:
1.Run the following commands
a. vol –list (ClientName cmdlet) //This may vary among hardware vendors
b. Vol –info test1 //This may vary among hardware vendors
1.‘*’ in Initiator Group means unrestricted access
2.Specific IQN value of Initiator must be added into the LUN iSCSI Initiator Group Creation:
Adding Initiator to the Initiator Group:
ACL Update on LUN
Case 3: Masking the LUN Path from ESXi Host through HBA Adapter
Note: Ensure that there is no claim rule set for ESXi Host, which affects LUN Mapping.
Trouble Shooting Steps
1.Run the following commands in the ESXi Host
a) esxcli storage core claimrule list
b) esxcli storage core claimrule remove –rule 101
Note: Remove unwanted rules using above command
Troubleshooting the network issue using tcpdump with the below mentioned arguments:
- “tcpdump-uw -w /tmp/t.cap port 3260 -i vmk1”