We are pleased to announce the certification of Dell EMC Isilon version OneFS 8.2 with HDP 3.1 and CDH 6.3.1 as an HDFS store and Dell EMC ECS v3.3.0 with HDP 3.1.4 and CDH 6.3.2 as a S3 object store via the Cloudera QATS program! See the Announcement.
Since the beginning of our respective partnership with Cloudera in 2015, Dell EMC has delivered differentiated consolidated data lake solutions for hundreds of Cloudera customers running Hortonworks Data Platform (HDP) and Cloudera Distribution of Hadoop (CDH) in a shared storage configuration with Dell EMC Isilon.
Dell EMC Isilon scale-out Network Attached Storage (NAS) has the ability to run HDFS natively and incorporates critical components of the HDFS software stack such as the name-node and data-node inside the OneFS software. Isilon OneFS provides complete name-node and data-node redundancy as each node in an Isilon cluster acts as a active name-node and data-node, there is no need to configure a local name-node or standby name-node when using Isilon as the HDFS store for Hadoop. Isilon significantly improves name-node and data-node resiliency and performance while rapidly serving petabyte scale data sets. Isilon OneFS natively implements erasure coding improving storage efficiency by 3x over legacy direct attached storage Hadoop deployments.
Dell EMC ECS, the leading object-storage platform from Dell EMC, has been engineered to support both traditional and next-generation workloads alike. Deployable in a software-defined model or as a turnkey appliance, ECS boasts unmatched scalability, manageability, resilience, and economics to meet the demands of modern business. ECS can be deployed with the reliability of private cloud infrastructure. Each service layer in ECS is independently scalable with no single points of failure. The data services layer of ECS provides S3 access, as well as access for HDFS, NFS, CAS, and SWIFT. The storage services layer stores, retrieves, protects, and replicates data. All data within ECS is common and sharable so data written over S3 can be accessed over HDFS, NFS, SMB, or SWIFT and vice versa. The fabric layer provides clustering, configuration management, and health monitoring. Everything done within ECS is done within the software which is containerized using docker. The storage solution can be federated to up to 8 locations and all the sites can be managed as a single resource. Federation also allows for a single global namespace with simple and secure access to content. Applications can read and write in an active-active or everywhere-active manner, a single URL can serve as single point of access for cloud based apps. ECS can also serve as secondary storage, which frees up access to primary storage. Policy based tiering is supported allowing integration with Isilon through cloudpools or with geodrive allowing windows based users access to ECS via SMB.
Dell EMC Isilon and ECS are enterprise grade storage solutions supporting quotas, snapshots, multitenancy, data at rest encryption and in-flight encryption using TLS, both solutions adheres to the security and exchange commission rule 17A-4F requirements and is also compliant with STIG hardening guidelines.
Cloudera’s new streamlined Quality Assurance Test Suite (QATS) certification process is designed to validate HDP and CDH on a variety of Cloud, Storage & Compute Platforms. The validation and certification of Dell EMC’s Isilon and Dell EMC's ECS storage solution is enabling us to deepen and expand our partnership and further support our joint customers.
What is the Cloudera QATS Program?
The QATS program is Cloudera's’ highest certification level, with rigorous testing across the full breadth of HDP and CDH services. In this case it focused on testing all the services running with HDP 3.1 and CDH 6.3.1 with Isilon OneFS, and it validated the features and functions of both HDP and CDH Hadoop distributions.
QATS is a product integration certification program designed to rigorously test Software, File System, Next-Gen Hardware and Containers with HDP and CDH. With dedicated Cloudera engineering resources to continuously and thoroughly test each release of HDP, QATS ensures that solutions are validated for a comprehensive suite of use cases and deliver high performance, under rigorous loads.
QATS functional testing was performed on the latest HDP 3.1 and CDH 6.3.1 versions with Isilon 8.2. In addition, Stress, High Availability, Reliability, Performance, Operational Readiness, and Integration Testing were executed to ensure all components accommodate system demands.
For the ECS v3.3 certification with HDP 3.1.4, more than 1500 functional tests were successfully carried out on the following components: Spark, MapReduce, Tez, Hive/Hive LLAP, Sqoop and Pig. For the ECS v 3.3 certification with CDH 6.3.2, more than 2400 functional tests were successfully carried out on the following components: Spark, Hive, Spark-Hive, MapReduce, Impala, Sqoop and Sentry.
This new certification enables Cloudera and Dell EMC to provide best-in-class support to our customers. Our teams will continue to collaborate at various levels – across engineering R&D as well as our go-to-market plans – to ensure continuous product compatibility and high-quality customer service. And we will continue to run QATS tests against future product versions including CDP.
Note: Cloudera has not released QATS software for CDP nor do they support any 3rd party storage solutions for CDP at this time. As soon as Cloudera releases QATS software for CDP, Dell EMC will certify CDP with Isilon and ECS accordingly. Until then, QATS certification is limited to HDP and CDH Hadoop distributions only.
QATS Test Coverage
The QATS certification covers an extensive range of tests including:
What does this mean for our joint customers?
QATS certification on Isilon provides existing customers with better support and reduced risk. Clusters that use CDH/HDP in conjunction with Cloudera Certified Technologies operate with lower risk and lower total cost of ownership (TCO). Cloudera Certified Technologies have been tested and validated to use supported APIs and to comply with Cloudera development guidelines for integration with Hadoop.
Isilon Certification includes the following components:
Specific Features within HDFS that are not support with Isilon HDFS:
Other features not support with Isilon:
Dell EMC Isilon 8.2 has now been certified with HDP 3.1 and CDH 6.3.1. The solution will be jointly supported by Cloudera and Dell EMC. We have a joint support process in place that involves triaging any issue that occurs with our solution, regardless of where it is discovered and directing issues to the appropriate teams, either in Cloudera or Dell EMC.
Cloudera and Dell EMC will also develop a joint reference architecture and solutions around Hadoop Tiered Storage. Hadoop Tiered Storage will enable customers to use existing Direct Attached Storage (DAS) clusters for hot data and Isilon for cold data within the same logical Hadoop cluster to simultaneously deliver extreme performance and economic scaling. QATS certification of the Hadoop Tiered Storage Solution is underway and will be finished by the end of May 2020.