Another threat that needs to be addressed, particularly at scale, is the looming specter of hardware induced corruption. For example, when CERN tested the data integrity of standard disk drives, they discovered some alarming findings. To do this, they built a simple write and verify application which they ran across a pool of three thousand servers, each with a hardware RAID controller. After five weeks of testing, they found in excess of five hundred instances of silent data corruption spread across seventeen percent of the nodes - after having previously thought everything was fine. Under the hood, the hardware RAID controller only detected a handful of the most blatant data errors and the rest passed unnoticed.
Suffice to say, this illustrates two inherent data protection requirements: First, the need for an effective, end-to-end data verification process to be integral to a storage device in order to detect and mitigate such instances of silent data corruption. Second, the requirement for regular and reliable backups is the linchpin of a well-founded data protection plan.