The number of writes to replicated volumes cannot exceed the bandwidth of a single network path between the clusters. This is to allow for the possibility of one network path between clusters failing but still maintaining service levels for your application requirements.

Bandwidth within a replicating system

As previously described, replicated I/O is sent from the SDC to the SDR, after which there are subsequent I/O operations from the SDR to SDSs on the source system. The SDR first passes the volume I/O on to the associated SDS for processing, such as compression and committal to disk. The associated SDS will probably not be on the same node as the SDR, and bandwidth calculations must account for this possibility. Then, the SDR applies incoming writes to the journaling volume. Because the journal volume is like any other volume, the SDR sends I/O to the various SDSs backing the storage pool in which the journal volume resides. Journaling adds two I/O operations: The SDR first writes to the relevant primary SDS backing the journal volume, and the primary SDS sends a copy to the secondary SDS. Finally, the SDR makes an extra read from the journal volume before sending data to the remote site.

Therefore, write operations for replicated volumes require three times more bandwidth within the source cluster as write operations for nonreplicated volumes. Carefully consider the write profile of workloads that will run on replicated volumes; additional network capacity is needed to accommodate the additional write overhead. In replicating systems, therefore, we recommend using 4 x 25 GbE or 2 x 100 GbE networks to accommodate the back-end storage traffic.