Large Write Buffer for vSAN 8 OSA Architecture

Introduction

In vSAN 8.0,  VMware vSAN introduced support for “Large Write Buffer”(LWB) cache disk size of upto 1.6TB can be used with OSA architecture. Although larger disks were supported in the previous releases for use in caching tier for vSAN OSA disk groups, the write buffer usage was limited to 600GB even when cache disks are larger, using larger disks in the previous releases (7.0 U3 and lower) was only preferred to increase endurance life of Cache Disk SSD.

Large Write Buffer feature (LWB) is targeted to support modern applications which have started to use large block size for writes (8k thru 256k), where such applications tends to perform bursty writes to vSAN which can easily fill up existing default write buffer of 600GB in Cache disk, which leads to sudden drop in write throughput and increased latency due to Congestion mechanism which is built into the system. Having write buffer of size upto 1.6TB can help absorb such bursty IO pattern better while providing consistent performance to all VMs in the cluster.

This Large Write Buffer (LWB) feature support is not enabled by default on vSAN 8.0 and higher version OSA clusters by default, allows users to decide to enable this feature. A host level advanced config option is used to enable/disable LWB, additionally we need to recreate every diskgroup post enabling/disabling the configuration flag for LWB to function. Please refer to official VMware KB for more details.

Pros:

• Provides significant performance improvement for Large block bursty workload which might easily fill up default 600GB write buffer. This will have better and improved sustained high throughput at Cache disks when using LWB feature.
• LWB feature keeps a lot of data set in the cache tier when there steady state workload, thereby reducing write amplification for IOs when it receives read modified writes to blocks which are yet to be destaged to capacity tier section.
• Reduces load on the Capacity tier section because LWB handles most of the overwrites in the cache tier and de-stage to Capacity disks is not triggered too often, throttled destage is only triggered when 30% of write buffer is filled up, unthrottled destage is triggered only when nearing max write buffer usage.
• Better coalescing and full stripe writes to the capacity tier disks with LWB since more overwrites are handled in Cache disks, reduces write amplification while destaging to Capacity Tier.
• Large Write Buffer also significantly improves read performance since most of the reads will be served directly from the cache disk without needing to read from capacity disks.
• Significantly improves resynchronization performance and ETA (IOPS/Throughput) because of LWB can accept more resync-writes while accepting writes from guests.
• Significantly improves reboot times ~50% faster as LWB allows most/all of the in-memory metadata to be dumped in to the Cache disk, which can be replayed quickly during reboots directly instead of sequentially reading it from Cache/Capacity disks.

Cons:

• Increased memory requirement –

Every vSAN disk group with large cache tier will incur an additional memory cost of 5GB. With the max number of supported disk groups per host (which is 5), the max additional memory cost increase is 25GB per host. Deployments that are tightly provisioned in terms of memory might experience issues due to this increased memory usage from vSAN.

• Unpredictable Performance with heterogeneous Setup – While the full benefits of the large cache tier may not be realized until all disk groups in a cluster are using the feature, there is no performance or functional regression from having a mix of small and large cache drives. Having mixed/heterogeneous sized Cache disks within a cluster with this feature enabled might lead to unpredictable performance.
• Requires all diskgroups to be recreated post enabling the feature, you will need to perform this task when there is minimal/steady state workload running to reduce the impact to guest performance due to resync traffic caused by Diskgroup recreation (Recommended to use full Data evacuation when you have enough free space in the cluster or Ensure Accessibility method recreation when there is space constraint in the vSAN Datastore)

How to enable/disable Large Write Buffer (LWB)

• Command to check the current setting:
  

  esxcfg-advcfg -g /LSOM/enableLargeWb

• Command to enable the feature:
  

  esxcfg-advcfg -s 1 /LSOM/enableLargeWb

Please follow official VMware KB-89485, where steps are documented clearly. 

Prerequisite

• Read and understand Pros and Cons explained above. Ensure that each host has enough additional memory (5GB per DG) before enabling/recreating diskgroups and also ensure that you have enough memory for HA restarts for VMs when there are host failures.
• vCenter Server should be version 8.0 or higher.
• All hosts in the cluster should be 8.0 or higher, make sure that all hosts are on the same build version before enabling the feature. (i.e not recommended to enable the feature with hosts in mix-mode some running 8.0.x and others 8.1.x)

I highly recommend you to read thru VMware Official Blog on Large Write Buffer for additional information.

If you wish to read other related topics head to Home-Page, thanks for taking time to read this article on “vSAN 8 Large Write buffer (LWB) for OSA Architecture”.