The Dell Streaming Data Platform is an elastically scalable, open-source platform that is created by Dell Technologies that allows for the intake and storage of streaming data. It is designed to absorb, store, and analyze continuously streaming data in real time, using Pravega as its key engine.
The platform concurrently processes both real-time and collected historical data in the same application, which can include millions of data streams from multiple sources, while ensuring low latencies and high availability. SDP ingests and stores streaming data from a range of sources, including:
- IoT devices
- Web logs
- Industrial automation
- Financial data
- Live video
- Social media feeds
- Applications
- Event-based streams
SDP manages stream ingestion and storage, and it hosts the analytic applications that process the streams. It dynamically distributes data processing and analytical jobs over the available infrastructure, and automatically scales resources to satisfy processing requirements in real time as the workload changes.
SDP integrates many capabilities into a single software platform, including:
- The platform ingests all types of data, whether static or streaming, in real time. Even historical files of data become bounded streams of data when ingested.
- Elastic-tiered storage provides instant access to real-time data and infinite storage, and access to historical data. This loosely coupled, long-term storage enables an unbounded digital video recorder (DVR) for all streaming data sources.
- Real-time stream analysis happens with an embedded analytics engine. Analyzing historical and real-time streaming data is unified to simplify the application-development process.
- Real-time and historical unification
- Real-time and historical data is processed to create and store new streams, send notifications to enterprise alerting tools, and send output to third-party visualization tools.
- Integrated management provides data security, configuration, access control, resource management, an intuitive upgrade process, health and alerting support, and network topology oversight.
- A web portal allows users to configure stream properties, view stream metrics, run applications, and view job status.
- Application Programming Interfaces (APIs) are included in the distribution. The web portal supports application deployment and artifact storage.
The benefits of using SDP at the manufacturing edge include:
- Expanded on-premises storage to include durable long-term storage on VxRail.
- Pass-through high-performance/low-latency storage for higher scale.
- Expanded options for developers to use open-source tools for real-time analytics using data streams.
- Expanded view of the physical world to include ingestion and processing of data streams including video, X-ray, Lidar, IR, and audio together with OT feeds from Litmus and other IT data streams.
- A high-performance, long-term storage option that provides persistence, scale, and centralization.
- Unlimited playback of historical data for AI and ML use cases, with tiered modeling and analytics.
- Combined stream analysis and OT data correlation using the ingestion of parallel data streams from other sources and contextualization.
Within the DVD for Manufacturing Edge, SDP acts as a storage solution to persist and centralize data that is ingested at the edge, as showing in the following figure:
Figure 16. Streaming Data Platform overview
For more technical details and documentation, see the Additional Resources section.