Virtual Products: Working with VMware Logs

Scope

This article explains how to gather VMWare logs on vSphere 5.x and 6.0 hosts that use either the vSphere Client or vSphere Web Client. It also provides a description of VM logs, and outlines which to use for various issues.

Procedure

vmkernel.log

VMWare hosts store VM logs in the //var/run/log directory. The vmkernel.log file records storage and networking device and driver events, as well as VMs starting. You can use this log to troubleshoot issues between a Datto appliance and the hypervisor, such as ESXi screenshots or virtualizations. You can also use it to investigate hypervisor-involved backup issues. If all of the virtual agents on a Datto appliance are having the same issue or identical errors, this should be one of the first logs you check.

vmware-<xx>.log

You can also find VM-specific logs on the hypervisor, in the storage location of the agent's .vmx file. The This location is often /vmfs/volumes/xxxxxxxx-xxxxxxx-xxxx-xxxxxxxxxxxx/<VM name>/, but may be different. The log files will be called vmware.log and will be incrementally numbered, such as vmware-23.log.

Other VMWare Logs

The//var/run/log directory on the hypervisor houses several other logs that can be useful in troubleshooting. Here are some, not all, of the logs that may be found in those locations:

  • //var/run/log/auth.log: ESXi Shell authentication success and failure. Could point to issues with the device connecting to the hypervisor.
  • //var/run/log/dhclient.log: DHCP client service, including discovery, address lease requests and renewals. Unlikely this will be necessary to use unless a Virtual product is being set up with DHCP.
  • //var/run/log/esxupdate.log: ESXi patch and update installation logs. May house information such as if a patch was applied to correct the CBT bug in 5.1 and 6.0.
  • //var/run/log/hostd.log: Host management service logs, including virtual machine and host Task and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections. Could cover a variety of issues.
  • //var/run/log/shell.log: ESXi Shell usage logs, including enable/disable and every command entered. For more information, see the Managing vSphere with Command-Line Interfaces section of the vSphere 5 Command Line documentation and Auditing ESXi Shell logins and commands in ESXi 5.x. Not likely needed.
  • //var/run/log/sysboot.log: Early VMkernel startup and module loading.
  • //var/run/log/syslog.log: Management service initialization, watchdogs, scheduled tasks and DCUI use. May cover multiple issues.
  • //var/run/log/usb.log: USB device arbitration events, such as discovery and pass-through to virtual machines. Very unlikely we would need this.
  • //var/run/log/vob.log: VMkernel Observation events, similar to vob.component.event. Covers events with SANs being used. Could possibly explain I/O issues.
  • //var/run/log/vmkernel.log: Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup. As stated before, this covers multiple things for host specific issues, and is a good place to look.
  • //var/run/log/vmkwarning.log: A summary of Warning and Alert log messages excerpted from the VMkernel logs. This is a good place to start as it removes all the informational entries.
  • //var/run/log/vmksummary.log: A summary of ESXi host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption.

Hypervisors will periodically archive logs. Archived logs reside in /var/run/log.

Obtaining VMWare Logs

Obtaining Diagnostic Information for ESXi 5.x and 6.0 hosts using the vSphere Client

You can gather ESXi 5.x host diagnostic information by using the vSphere Client connected to the ESXi host or to vCenter Server. To gather diagnostic data using the VMware vSphere Client:

  • Open the vSphere Client, and connect to vCenter Server, or directly to an ESXi 5.x host.
  • Log in using an account with administrative privileges or with the Global.Diagnostics permission.
  • Select an ESXi host, cluster, or datacenter in the inventory.
  • Click File ? Export ? Export System Logs.
  • If a group of ESXi hosts are available in the selected context, select the host or group of hosts from the Source list.
  • Click Next.
  • In the System Logs pane, select the components for which the diagnostic information must be obtained. To collect diagnostic information for all the components, click Select All.

IMPORTANT  Confirm that HungVM is not selected, as this may cause a virtual machine failure.

  • Click Next.
  • In the Download Location pane, click Browse, and select a location on the client's disk to save the support bundle.
  • Click Next.
  • In the Ready to Complete pane, review the summary, and click Finish. The Downloading System Logs Bundles dialog will appear and report the progress of the support bundle creation from each source. You will also see the hypervisor create a Generate system logs bundles task.

Obtaining Diagnostic Information for ESXi 5.x and 6.x hosts using the vSphere Web Client

You can gather ESXi 5.x host diagnostic information by using the vSphere Web Client connected to the ESXi host or to vCenter Server. To gather diagnostic data using the VMware vSphere Web Client:

  • Open the vSphere Web Client.
  • Log in using an account with administrative privileges or with the Global.Diagnostics permission.
  • Select Hosts and Clusters from the Home tab.
  • Select an ESXi host or vCenter in the inventory.
  • Click Actions.
  • Select All vCenter Actions ? Export System Logs...
  • If a group of ESXi hosts are available in the selected context, select the host or group of hosts from the Source list.
  • Click Next.
  • In the System Logs pane, select the components for which the diagnostic information must be obtained. To collect diagnostic information for all the components, click Select All.
  • If required, select the Gather performance data option, and specify a duration and interval.
  • Click Generate Log Bundle. The host or vCenter Server will generate .zip bundles containing the log files. The Recent Tasks panel shows the Generate diagnostic bundles task in progress.
  • Click Download Log Bundle.

After the download completes, click Finish, or generate another log bundle.

You can also use the Events Log to troubleshoot issues, particularly when trying to determine a timestamp.

To export the Events Log:

  • Select an inventory object.
  • Click the Monitor tab, and click Events.
  • Click the Export icon.
  • In the Export Events window, specify what types of event information you want to export.
  • Click Generate CSV Report, and click Save.
  • Specify a file name and location, and save the file.

Obtaining Diagnostic Information for ESXi 5.x and 6.0 hosts using SSH

The filesystem on vSphere hypervisors is Unix-based. If you are unable to access the vSphere Client, you may be able to work with the partner to establish an SSH session on the host.

Understanding the logs

Each log can contain a large variety of events. See the Additional Resources section of this article for troubleshooting resources. You can also query search engines for specific logged events to determine their meanings.

Additional Resources

Networking

Pairing Failures

VMWare Tools

Versioning

iSCSI