Witnesses do not maintain VM data, but rather only component metadata.
It is important to remember that data is stored on vSAN in the form of objects. Objects are comprised of one or more components of items such as VM home, swap, disks, and snapshots.
Objects can be split into more than one component when the size is >255 GB, and/or a number of stripes (stripe width) policy is applied.
Additionally, the number of objects/components for a given Virtual Machine is multiplied when a number of failures to tolerate (FTT) policy is applied for data protection and availability.
The required bandwidth between the witness and each site is equal to ~1138 B x number of components / 5 s.: 1138 B x NumComp / 5 seconds
The 1138 B value comes from operations that occur when the preferred site goes offline, and the secondary site takes ownership of all the components.
When the primary site goes offline, the secondary site becomes the leader. The witness sends updates to the new leader, followed by the new leader replying to the witness as ownership is updated.
The 1138 B requirement for each component comes from a combination of a payload from the witness to the backup agent, followed by metadata indicating that the Preferred Site has failed.
In the event of a Preferred Site failure, the link must be large enough to allow for the cluster ownership to change, as well as ownership of all the components within five seconds.
Example workload
With a VM comprised of:
Approximately 100 VMs with the above configuration would require the witness to contain 600 components: (100 VMs * 3 components/VM * 2 (FTT+1) * 1 (stripe width))
To successfully satisfy the Witness bandwidth requirements for a total of 600 components on vSAN, use the following calculation:
Converting Bytes (B) to Bits (b), multiply by 8: B = 1138 B * 8 * 600 / 5 s = 1,092,480 Bits per second = 1.09 Mbps
VMware recommends adding a 10% safety margin and round up: B + 10% = 1.09 Mbps + 109 Kbps = 1.199 Mbps
With the 10% buffer included, a guideline can be stated that for every 6,000 components, 1.2 Mbps is appropriate.