vSANTraces are generally used to troubleshoot and debug critical performance issues in vSAN(OSA and ESA) and also in some cases they are used to troubleshoot vSAN data-path related problems such as Stuck-IO, Congestion, slow/no de-stage etcetera. vSANTraces are not require to troubleshoot/debug all kinds of issues however is required for certain cases.
In any vSAN enabled host, you might have noticed that the default location/path where the traces are collected is “/vsantraces”. Depending on the requirement, users may however change the path for vSANTraces to external Storage or a local datastore and change the default settings of “Number Of Files To Rotate:” and “Maximum Trace File Size:” for vSANTraces using esxcli commands. Having said that VMware doesnt recommend/allow customers to set the path for vSANTraces to vSAN-Datastore directly because it will impact performance of the virtual machines running on vSANDatastore as vsanTraces are write heavy (every subsystem of vSAN data-path writes to traces for bunch IO) which helps engineering to debug issues.
vsantraces are very chatty and time period coverage/retention period is very short worst case less than 10mins. When you delay support bundle collection after an issue in your vSAN environment there is high probability that vsantraces have already rolled over. In such cases you might have to reproduce the issue if possible if you understand how to manifest it or wait for the issue to occur again and collect traces/support bundles immediately after the issue, so that VMware support and engineering can debug certain performance issues, congestion issues..etc.
The problem is that you might not know when such issue can happen again and it will be too difficult to manually monitor such issues and collect data for support and engineering.VMware introduced this new feature called “Native vSAN Trace object” for vSAN 8.0 U1 which helps to alleviate the problems which I have explained above.
What does “Native vSAN Trace object” feature offer?
- Creates a NameSpace Object in vSAN Datastore (is hidden folder “.vsan.trace” ) with storage policy of FTT-1/RAID-1 on both OSA and ESA architecture.
- Automatically creates multiple sub-directories under the namespace folder for each host with their respective vSAN-Host-UUIDs
- Under each host-UUID folder it creates additional sub-directories with date format (YYYY-MM-DD)
- Periodically dumps vsantraces from the default path /vsantraces under respective date sub-directory.
- Persists/retains traces files under each dated-sub-directories for upto 6 days and deletes older files.
- They can use upto 512GB or 1% of vSAN Datastore at most. Usage of “Native Trace Object” can be viewed under “Capacity” section for vSAN. The Native trace Object is also displayed under “Virtual Objects” view with type “other” just like vSAN Performance Object.
- This feature is enabled only if the trace log location remains under the default path/setting ( /vsantraces)
- Offers a command to create a support-bundle by dumping traces from native vsan trace object, which can be run from each host respectively.
- In a stretched cluster setup native trace object feature only collects traces from data nodes and witness traces are excluded.
Pre-Requisite
- VMware vCenter server hosting a given vSAN cluster should be on build version 8.0 U1 or higher.
- All hosts in the cluster should be on build version 8.0 U1 or higher.
- Default setting for vsantrace should not be changed from /vsantraces for this feature to work.
Native Trace Object in UI
Native trace Object CLI
Check if default vsantraces settings:
Identify native trace object namespace folder:
Check the storage Policy for the native trace object since its not shown in virtual Object View:
Viewing folder structure for each host inside Native Trace Object
When collecting regular support bundle trace files from Native trace object is not collected.
vm-support command with specific manifest needs to be run on each host to collect traces from Native trace object for each host in the cluster.
This feature is intentionally implemented this way to make sure support bundles are not very large and we do not want vsantraces from native trace object file is not required for every other support ticket.
Syntax: vm-support -a Storage:VSANTracesExtended -w <localdatastore path or /scratch/>
Current Limitation
- Deleting old files from vsan trace object folder will not recover space on vSAN Datastore as it doesnot support unmap. vSAN native trace object keeps saving latest trace files and deletes the old ones. It prefers to request new disk space while adding the new files until reaching the double size of the object. After that, the object will reuse the old space if only no old space is available.
VMConAWS users will also see native trace object being created as part of future upcoming VMC releases, however VMC users will not have direct access to this object or vSAN traces as its part of SaaS and VMware will manage this.
Conclusion
Native vSAN Trace object feature definitely helps to RCA a lot of unknown performance issues/congestion issues which cannot be reproduced, it is highly recommended to ensure this is fully functional after you upgrade to vSAN 8.0 U1 or higher version.
Default vsantraces will be still available when a support bundle is collected. When VMware support requests older vsantraces while troubleshooting certain issues, then you may follow the steps described in this blog to collect and upload additional trace files from Native vSAN Trace Object.
If you want to read other related topics head to Home-Page, thanks for taking time to read this article on “Native vSAN Trace Object”.