vSAN Scrubber changes 7.0U1c

vSAN Scrubber is a background operation at DOM owner. Scrubber performs detection (scrub) and fixing (recover) of checksum and IO errors.

Mostly Checksum errors and I/O errors which are seen in vSAN environment is due to medium errors against the physical capacity disks where the data is persisted. These errors don’t surface until we try to read the object from both mirrors (in case of raid-1 objects). We might always be performing reads on a single mirror especially in case of Stretched cluster where site affinity is set to either preferred site or secondary site to avoid cross site traffic.

Maintenance tasks, hardware failures which results in a resync task , will trigger reads against the secondary/surviving mirrors which may discover and report checksum errors or I/O failures due to bad sectors backing the object’s component and resync would get stuck.

vSAN 7.0U1 and previous release had the advanced scrubber value “VSAN.ObjectScrubsPerYear” set to 1 per year against each objects, this setting is now changed to 6 per year on the latest vSAN 7.0U1c and later releases. This means that vSAN Scrubber will scrub every objects once every two months to make sure all the affected components with unreadable blocks, incorrect checksum are relocated to different sectors/disks by rebuilding from neighboring components/mirrors. See ESXI 7.01c release notes for additional information.

We can also look at some statistics against certain VMDK if necessary to know if there were any checksum errors against a specific object and ETA for scrubber completion (**Generally used by VMware support)

[root@localhost] esxcli vsan debug object list -u 19433d5f-9dee-9f58-9b67-ecf4bbec65d8


Object UUID: 19433d5f-9dee-9f58-9b67-ecf4bbec65d8
   Version: 13
   Health: healthy
   Owner: is-tse-d155.vsensei.local
   Size: 50.00 GB
   Used: 8.90 GB
   Policy:
      objectVersion: 13
      hostFailuresToTolerate: 1
      SCSN: 92
      CSN: 94

   Configuration:

      RAID_1
         Component: 19433d5f-e068-605c-faf8-ecf4bbec65d8
           Component State: ACTIVE,  Address Space(B): 53687091200 (50.00GB),  Disk UUID: 52de9507-470b-8a3c-1916-3dc6c696bc5f,  Disk Name: naa.5002538c4044d881:2
           Votes: 1,  Capacity Used(B): 4827643904 (4.50GB),  Physical Capacity Used(B): 4777312256 (4.45GB),  Host Name: is-tse-d156.vsensei.local
         Component: 19433d5f-6436-635c-631f-ecf4bbec65d8
           Component State: ACTIVE,  Address Space(B): 53687091200 (50.00GB),  Disk UUID: 52deab75-cff7-87a7-1273-5a505995c264,  Disk Name: naa.5002538c4044d87f:2
           Votes: 1,  Capacity Used(B): 4827643904 (4.50GB),  Physical Capacity Used(B): 4777312256 (4.45GB),  Host Name: is-tse-d157.vsensei.local
      Witness: 19433d5f-7f0b-645c-55c6-ecf4bbec65d8
        Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: 525a9779-affc-46ec-5f59-70a0b6ae28d2,  Disk Name: naa.5002538c4044d6ad:2
        Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: is-tse-d155.vsensei.local

   Type: vdisk
   Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/14433d5f-ce0e-0a0c-8d9f-ecf4bbec65d8/VC-50_12.vmdk (Exists)
   Group UUID: 14433d5f-ce0e-0a0c-8d9f-ecf4bbec65d8
   Directory Name: N/A


[root@localhost] vsish -e get /vmkModules/vsan/dom/owners/19433d5f-9dee-9f58-9b67-ecf4bbec65d8/scrubStats

DOM Owner Scrub Stats
   Total bytes scrubbed during the last round:56224055296
   Total bytes allocated for this object:9554624512
   Total size of all components including parity if any:107374182400
   Number of rounds of scrubbing completed:0
   Number of scrub ops issued:10
   Total number of LSEs/checksum errors detected:0
   Total number of faked checksum errors detected:0
   Total number of OOB LSEs/checksum errors detected:0
   Total number of OOB faked checksum errors detected:0
   Total number of LSEs/checksum errors recovered:0
   Total number of faked checksum errors recovered:0
   Total number of OOB LSEs/checksum errors recovered:0

   Total number of OOB faked checksum errors recovered:0
   Total number of LSEs/checksum errors that failed to recover:0
   Total number of faked checksum errors that failed to recover:0
   Total number of OOB LSEs/checksum errors that failed to recover:0
   Total number of OOB faked checksum errors that failed to recover:0
   Seconds since the last round of scrubbing started:11501811
   Total bytes scrubbed so far in current round:0
   Initial estimation of current scrub ETA (seconds):36432237
   Current estimation of current scrub ETA (seconds):558692354
   Maximum estimation of current scrub ETA (seconds):3071998125
   Maximum estimation of last scrub ETA round (seconds):0

Also take look at the new blog post on Advanced-Cross-vCenter Migration feature which is included with vSphere 7.0U1c.