IP load balancer

We recommend that you consider using a network IP load balancer. Because ECS is a multi-node object store and Veeam Backup & Replication can access only one end point, using an IP load balancer ensures that Veeam backup tasks use all ECS nodes.

Veeam v12 can be configured to have a Scale out Backup Repository contain multiple Object Repositories, one for each ECS node, and Veeam will balance consumption across them.  This is not discussed in this whitepaper.

We use ECS Connection Manager when testing our Veeam Backup & Replication solutions. ECS Connection Manager can be implemented as a hardware appliance or as a virtual machine.

Table 9.      ECS Connection Manager throughput

For more information, see ECS Connection Manager.

ECS replication considerations for Veeam

If you want to use ECS replication and Object Lock with Veeam Backup & Replication, you cannot use an ECS bucket that has Access During Outage (ADO) enabled, pre ECS 3.8.

Pre ECS 3.8, when ADO is enabled, if a temporary site outage (TSO) of one VDC occurs, object operations of read, create, update, and delete, as well as list buckets not owned by an online VDC, will fail.

With ECS 3.8 Object Lock and ADO can be enabled together in a bucket by users with system administrator privileges, when the data loss risks during a temporary site outage are well understood.  Access During Outage (ADO) is a behavior that allows data access during a temporary site outage (TSO). When Object Lock and ADO are enabled together in a bucket, there is a risk of losing locked versions during a TSO. As a result, for ADO buckets, setting Object Lock is denied by default. You can allow Object Lock and ADO to co-exist, when you have system administrator privileges, and you understand the risk of losing locked versions of data during a TSO.

For more information about ADO, see the ECS Administration Guide.

ECS garbage collection and Veeam Backup & Replication

The ECS storage engine uses the concept of “chunks” to store all types of data (metadata and user data). An ECS chunk is described as a logical container of contiguous space with the default size of 138 MB.

Objects are written to the chunk until it is filled up to 128 MB or after a set period of time. Chunks are written in an append-only pattern, which means that an application cannot modify or delete existing data within a chunk; instead, updated data is written to a new chunk. Thus, for both deletes and updates, space occupied by objects is no longer referenced and becomes a candidate for garbage collection.

The following figure shows the implementation workflow.

ECS garbage collection methods

ECS space reclamation can occur not only when a chunk is completely full of garbage but also when a chunk is partially filled with garbage. A chunk is eligible for garbage collection under these conditions:

• Normal garbage collection—When the entire chunk is garbage, reclaim space.
• Partial garbage collection by merge—When the chunk is two-thirds garbage, reclaim the chunk by merging the valid parts of the chunk with other partially filled chunks to a new chunk, and reclaim the space.
• Partial garbage collection by compaction—When the chunk is more than one-third but less than two-thirds garbage, the valid parts of the chunk are re-erasure coded to protect the valid data. This process includes treating the garbage ranges within the chunk as zeros for recalculating the erasure code of the valid data. Once the valid data has been re-erasure coded, the garbage ranges are then eligible to be reclaimed.

Veeam Backup & Replication and ECS garbage collection

As previously described, you can configure Veeam Backup & Replication to use block sizes from 256 KB to 8 MB. Regardless of the block size chosen, many blocks will be written into one 128 MB chunk.

Here are a few, somewhat simplistic backup scenarios to show how efficient ECS garbage collection might be.

Scenario 1: All backups expire after the same length of time

In this scenario, many backup jobs are running simultaneously, and ECS chunks are being written with objects from many different backup jobs. However, because all these backup jobs will expire simultaneously, the chunks will become 100 percent filled with garbage and normal garbage collection will free up 100 percent of the storage.

Scenario 2: 80 percent have 5-year retention times, and 20 percent have 6-month retention times

In this scenario, many ECS chunks are written with Veeam blocks; 80 percent of the chunks have 5 years before they expire, and 20 percent have a 6-month expiration. After 6 months, we will have ECS chunks that have 20 percent garbage in them. These chunks will not be treated by normal garbage collection (they are not 100 percent garbage). They will also not be treated by partial garbage collection by merge (they are not two-thirds garbage) or partial garbage collection by compaction (they are less than one-third garbage). Garbage collection will not touch them. 

Therefore, until another 4.5 years passes, there will be a 20 percent storage overhead due to garbage collection not being able to free up the storage used by these deleted objects.

These scenarios are illustrations only. Although real production backups will have more varied backup policies than depicted, these scenarios show that there is potential for a 30 percent storage overhead.

Conclusion

Predicting storage overhead due to garbage collection not being able to free storage from deleted objects is complicated but should be considered to avoid the need to assume a 30 percent storage overhead.

Tools are available to monitor the progress of ECS garbage collection and study its effect on your ECS system. For more information, see the ECS Info Hub.