WSFC with Native Shared Disks on vSAN

With introduction of vSAN 6.7 U3 / vSphere 6.7 U3 we now have native support for SCSI-3 Persistent reservation on vSAN objects. This means that we no longer need vSAN iSCSI targets for creating application level clustering services like Windows Server Failover Clusters (WSFC) which needs to share a disk in order to determine ownership across the application cluster.

This new feature allows for VMs running WSFC to use a shared VMDK on a vSAN datastore by creating SCSI controllers in physical Mode and then create and share vmdks using independent-persistent presentation mode. This drastically reduces the time in deployment and configuration on WSFC when compared to previous vSAN releases which requires us to configure thru vSAN iSCSI targets and also comes with its overhead .

Configuring vSAN native WSFC

• Create a new virtual SCSI Controller using the “physical” option set for the SCSI Bus Sharing setting.  This is the virtual SCSI controller that will be used by the shared VMDK.

• For the given VMDK that will be shared, assign it to the virtual SCSI Controller created above, and ensure the disk mode is set to “Independent – Persistent”

Go to other VMs and add them using “Add Existing Disk option” on which you would want to add the shared drives, make sure you select your new SCSI controller which was created and set the drives to independent persistent mode. See below GIF for complete setup

• Once the drives are shared you can go to the guest and claim the drives , following Demo-GIF only shows that the drives can be claimed and used within guest

Recommended Practices

• Once the drives are shared between the two VMs , it is highly recommended that you setup VM to VM anti-affinity rule so that both clustered VMs are not moved by DRS automatically to the same host to avoid simultaneous failures.
• For Stretched cluster implementation :
  1. Double-check the VM to VM affinity rules and VM to Host affinity rules and set them up correctly.
  2. Its important to note that the Objects cannot be owned by two hosts at the same time , which means that if we create shared disks on VM-1 and power it on on host-1 the ownership stays with host-1 for the respective shared vmdk and secondary VM-2 which shares the disk with VM-1 will send all its IOPS over the vSAN VMkernel traffic over to the object owner to commit its IOPS and this might use good amount of vSAN bandwidth between the sites when implemented in stretched clusters .
  3. If all the heavy lifting is done on the primary object owner we will see greater response times for the IOPS and when site fails all shared disk object ownership will move to the secondary VM
  4. Depending on the practicality and requirements you can chose to setup storage policy to tolerate failures across and within sites “PFTT/SFTT”

Example Object owner comparison 

[root@host1:~] esxcli vsan debug object list --vm-name=Node1-WSFC  
 Object listed below are truncated  namespace,vmswap and first OS disk objects are not listed , showing only shared disks from VM-1

 Object UUID: 9e1beb5d-87b9-1d24-5a3d-ecf4bbec65d8
    Version: 10
    Health: healthy
    Owner: host2.vsensei.com
    Size: 10.00 GB
    Used: 0.11 GB
    Policy:
       stripeWidth: 1
       cacheReservation: 0
       CSN: 3
       forceProvisioning: 0
       spbmProfileName: vSAN Default Storage Policy
       proportionalCapacity: 0
       spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
       hostFailuresToTolerate: 1
       spbmProfileGenerationNumber: 0
 Configuration:
   RAID_1      Component: 9e1beb5d-a78d-9524-231c-ecf4bbec65d8        Component State: ACTIVE,  Address Space(B): 10737418240 (10.00GB),  Disk UUID: 52d97584-3471-9660-934a-e2cafd267313,  Disk Name: naa.5002538c4044d6a3:2        Votes: 1,  Capacity Used(B): 62914560 (0.06GB),  Physical Capacity Used(B): 58720256 (0.05GB),  Host Name: host1.vsensei.com      Component: 9e1beb5d-16f9-9624-787d-ecf4bbec65d8        Component State: ACTIVE,  Address Space(B): 10737418240 (10.00GB),  Disk UUID: 52ece953-87e3-0fe4-8ab8-cbdd374aa0f3,  Disk Name: naa.5002538c4044d6ae:2        Votes: 1,  Capacity Used(B): 62914560 (0.06GB),  Physical Capacity Used(B): 58720256 (0.05GB),  Host Name: host3.vsensei.com   Witness: 9e1beb5d-c8ca-9724-8c77-ecf4bbec65d8     Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: 525452e3-e6c0-884a-a6ea-edc03570a490,  Disk Name: naa.5002538c4044d6ab:2     Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: host1.vsensei.com
Type: vdisk
Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/071aeb5d-21b3-664d-7ae0-ecf4bbec6050/Node1-WSFC_1.vmdk (Exists)
Group UUID: 071aeb5d-21b3-664d-7ae0-ecf4bbec6050
Directory Name: N/A

Object UUID: 8f1deb5d-abd5-f134-85ec-ecf4bbec65d8
Version: 10
Health: healthy
Owner: host2.vsensei.com
Size: 40.00 GB
Used: 0.20 GB
Policy:
stripeWidth: 1
cacheReservation: 0
CSN: 3
forceProvisioning: 0
spbmProfileName: vSAN Default Storage Policy
proportionalCapacity: 0
spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
hostFailuresToTolerate: 1
spbmProfileGenerationNumber: 0
Configuration:
  RAID_1      Component: 8f1deb5d-6bba-8535-5e36-ecf4bbec65d8        Component State: ACTIVE,  Address Space(B): 42949672960 (40.00GB),  Disk UUID: 5225900e-2911-3548-1c57-3fb38e948a00,  Disk Name: naa.5002538c4044d881:2        Votes: 1,  Capacity Used(B): 109051904 (0.10GB),  Physical Capacity Used(B): 104857600 (0.10GB),  Host Name: host1.vsensei.com      Component: 8f1deb5d-833a-8735-aa62-ecf4bbec65d8        Component State: ACTIVE,  Address Space(B): 42949672960 (40.00GB),  Disk UUID: 52cf328a-533c-5162-12ef-bff7ac01d103,  Disk Name: naa.5002538c4044d6ad:2        Votes: 1,  Capacity Used(B): 109051904 (0.10GB),  Physical Capacity Used(B): 104857600 (0.10GB),  Host Name: host1.vsensei.com   Witness: 8f1deb5d-fb06-8835-4f8c-ecf4bbec65d8     Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: 52208d4c-7f55-578a-cf89-21b65b58e3a3,  Disk Name: naa.5002538c4044d87f:2     Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: host3.vsensei.com
Type: vdisk
Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/071aeb5d-21b3-664d-7ae0-ecf4bbec6050/Node1-WSFC_2.vmdk (Exists)
Group UUID: 071aeb5d-21b3-664d-7ae0-ecf4bbec6050
Directory Name: N/A

When we look at debug object list output was looked from VM-2 which is sharing the disk from VM1 , it does not show the shared disk because the attached disks belong to a different namespace UUID and also for a fact that we cant have two object owners for the same object the ownership always stays with the first VM-1.

[root@host2:~] esxcli vsan debug object list --vm-name=Node2-WSFC
Object UUID: f119eb5d-f922-730b-4de1-ecf4bbec6050
Version: 10
Health: healthy
Owner: host3.vsensei.com
Size: 255.00 GB
Used: 0.79 GB
Policy:
cacheReservation: 0
spbmProfileGenerationNumber: 0
stripeWidth: 1
forceProvisioning: 0
spbmProfileName: vSAN Default Storage Policy
hostFailuresToTolerate: 1
spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
proportionalCapacity: [0, 100]
CSN: 3
Configuration:
  RAID_1      Component: f119eb5d-3031-090c-559b-ecf4bbec6050        Component State: ACTIVE,  Address Space(B): 273804165120 (255.00GB),  Disk UUID: 52cf328a-533c-5162-12ef-bff7ac01d103,  Disk Name: naa.5002538c4044d6ad:2        Votes: 1,  Capacity Used(B): 423624704 (0.39GB),  Physical Capacity Used(B): 419430400 (0.39GB),  Host Name: host1.vsensei.com      Component: f119eb5d-5169-0a0c-4104-ecf4bbec6050        Component State: ACTIVE,  Address Space(B): 273804165120 (255.00GB),  Disk UUID: 52ece953-87e3-0fe4-8ab8-cbdd374aa0f3,  Disk Name: naa.5002538c4044d6ae:2        Votes: 1,  Capacity Used(B): 423624704 (0.39GB),  Physical Capacity Used(B): 419430400 (0.39GB),  Host Name: host3.vsensei.com   Witness: f119eb5d-512f-0b0c-8869-ecf4bbec6050     Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: 52d97584-3471-9660-934a-e2cafd267313,  Disk Name: naa.5002538c4044d6a3:2     Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: host1.vsensei.com
Type: vmnamespace
 Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/Node2-WSFC (Exists)
Group UUID: f119eb5d-f922-730b-4de1-ecf4bbec6050
Directory Name: Node2-WSFC

Object UUID: f319eb5d-8ed8-cf3b-3494-ecf4bbec6050
Version: 10
Health: healthy
Owner: host3.vsensei.com
Size: 40.00 GB
Used: 19.39 GB
Policy:
cacheReservation: 0
spbmProfileGenerationNumber: 0
stripeWidth: 1
forceProvisioning: 0
spbmProfileName: vSAN Default Storage Policy
hostFailuresToTolerate: 1
spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
proportionalCapacity: 0
CSN: 3
Configuration:
  RAID_1      Component: f319eb5d-fe40-843c-ec0f-ecf4bbec6050        Component State: ACTIVE,  Address Space(B): 42949672960 (40.00GB),  Disk UUID: 525452e3-e6c0-884a-a6ea-edc03570a490,  Disk Name: naa.5002538c4044d6ab:2        Votes: 1,  Capacity Used(B): 10510925824 (9.79GB),  Physical Capacity Used(B): 10406068224 (9.69GB),  Host Name: host1.vsensei.com      Component: f319eb5d-5bdf-853c-ff7b-ecf4bbec6050        Component State: ACTIVE,  Address Space(B): 42949672960 (40.00GB),  Disk UUID: 52d97584-3471-9660-934a-e2cafd267313,  Disk Name: naa.5002538c4044d6a3:2        Votes: 1,  Capacity Used(B): 10510925824 (9.79GB),  Physical Capacity Used(B): 10406068224 (9.69GB),  Host Name: host1.vsensei.com   Witness: f319eb5d-7c0e-873c-ccb2-ecf4bbec6050     Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: 52208d4c-7f55-578a-cf89-21b65b58e3a3,  Disk Name: naa.5002538c4044d87f:2     Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: host3.vsensei.com
Type: vdisk
Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/f119eb5d-f922-730b-4de1-ecf4bbec6050/Node2-WSFC.vmdk (Exists)
Group UUID: f119eb5d-f922-730b-4de1-ecf4bbec6050
Directory Name: N/A

Object UUID: fa3aeb5d-f5eb-6ef0-f766-ecf4bbec91a8
Version: 10
Health: healthy
Owner: host3.vsensei.com
Size: 8.00 GB
Used: 0.01 GB
Policy:
cacheReservation: 0
spbmProfileGenerationNumber: 0
stripeWidth: 1
forceProvisioning: 1
spbmProfileName: vSAN Default Storage Policy
hostFailuresToTolerate: 1
spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
proportionalCapacity: 0
CSN: 3
Configuration:
  RAID_1      Component: fa3aeb5d-b08c-e5f0-ee09-ecf4bbec91a8        Component State: ACTIVE,  Address Space(B): 8589934592 (8.00GB),  Disk UUID: 52cf328a-533c-5162-12ef-bff7ac01d103,  Disk Name: naa.5002538c4044d6ad:2        Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: host1.vsensei.com      Component: fa3aeb5d-e2a4-e6f0-99f1-ecf4bbec91a8        Component State: ACTIVE,  Address Space(B): 8589934592 (8.00GB),  Disk UUID: 52d97584-3471-9660-934a-e2cafd267313,  Disk Name: naa.5002538c4044d6a3:2        Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: host1.vsensei.com   Witness: fa3aeb5d-ce5d-e7f0-b3a2-ecf4bbec91a8     Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: 52ece953-87e3-0fe4-8ab8-cbdd374aa0f3,  Disk Name: naa.5002538c4044d6ae:2     Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: host3.vsensei.com
Type: vmswap
 Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/f119eb5d-f922-730b-4de1-ecf4bbec6050/Node2-WSFC-3f77264a.vswp (Exists)
Group UUID: f119eb5d-f922-730b-4de1-ecf4bbec6050
Directory Name: N/A

For Additional information on Microsoft SQL clustering setup with SCSI-3 PR read VMware-Blogs-native-sql-cluster-setup-vSAN and VMware-Blogs-Native-WSFC-vSAN-Stretched-Clusters