The platform supports several different types of storage for ephemeral and persistent storage, and for modern data stack storage.
Home > Workload Solutions > Data Analytics > Guides > Design Guide—Modern Data Stack with Symcloud Platform and Delta Lake with Intel > Container platform implementation
The platform supports several different types of storage for ephemeral and persistent storage, and for modern data stack storage.
Symcloud Storage is a scalable, high-performance software defined storage system that is Kubernetes Container Storage Interface (CSI) compatible. It provides data resiliency through replication, and supports encryption, compression, and thin-provisioning. Symcloud Storage is application and infrastructure aware, allowing it to support data locality, snapshots, and backups for applications running on the platform. Symcloud Storage is hosted on nodes with the storage role. At installation time, it discovers available disks and pools them to provide storage to applications.
Symcloud Storage uses volumes as the unit of allocation. These volumes are analogous to Kubernetes PersistentVolume objects. The replication factor, encryption, and compression properties are specified when a volume is created. The storage class can also specify the preferred media type, either HDD, SSD, or NVMe.
Symcloud Storage is exposed to applications through the Kubernetes CSI using a Kubernetes StorageClass object. Symcloud Storage ships with three predefined StorageClasses:
robin
—The default StorageClass that has no advanced features and can be used for standard ReadWriteOnly (RWO) and ReadWriteMany (RWX) volumes. robin-repl-3
—A StorageClass that uses three replicas for data reliability. robin-immediate
—A StorageClass that creates a volume when a Persistent Volume Claim (PVC) is created. It does not wait for the first consumer of that volume.
The platform uses file collections to store application bundles, images, and collected logs. File collections are exposed through the Symcloud file server, which runs on the control plane nodes. Storage for each file collection is allocated from Symcloud managed storage as a volume.
The platform can use any storage system that has a CSI compatible driver. This storage can be used for ephemeral and persistent storage exactly like Symcloud Storage. However, the backup, snapshot, and migration capabilities of the platform are only available for applications using Symcloud Storage.
The platform has two types of modern data stack storage available, using:
These storage systems are managed and scaled independently from the core platform, providing a decoupled storage and compute architecture.
Applications can connect to either or both modern data stack storage options directly over the network from the application-level code:
hadoop-hdfs-client
) provide the hdfs:// protocol. hadoop-aws
) provide the s3a:// protocol. Depending on the application and its implementation, the images and application bundle may require that the necessary libraries be included. The application must handle authentication to any external storage.