SDP is an elastically scalable platform for ingesting, storing, and analyzing streaming data in real time. The platform concurrently processes both real-time and collected historical data in the same application. SDP ingests and stores streaming data from a range of sources, including IoT devices, web logs, industrial automation, financial data, live video, social media feeds, applications, and event-based streams. The platform can process millions of data streams from multiple sources while ensuring low latencies and high availability. The platform manages stream ingestion and storage and hosts the analytic applications that process the streams. SDP dynamically distributes data processing and analytical jobs over the available infrastructure. Also, it dynamically and automatically scales resources to satisfy processing requirements in real time as the workload changes.

SDP integrates the following capabilities into a single software platform:

• Stream ingestion—The platform ingests all types of data, whether static or streaming, in real time. Even historical files of data, when ingested, become bounded streams of data.
• Stream storage—Elastic tiered storage provides instant access to real-time data and infinite storage, as well as access to historical data. This loosely coupled long-term storage is what enables an unbounded digital video recorder (DVR) for all streaming data sources.
• Stream analytics—Real-time stream analysis is possible with an embedded analytics engine. Analyzing historical and real-time streaming data is unified to simplify the application development process.
• Real-time and historical unification—The platform can process real-time and historical data, create and store new streams, send notifications to enterprise alerting tools, and send output to third-party visualization tools.
• Platform management—Integrated management provides data security, configuration, access control, resource management, an intuitive upgrade process, health and alerting support, and network topology oversight.
• Run-time management—A web portal lets users configure stream properties, view stream metrics, run applications, and view job status.
• Application development—APIs are included in the distribution. The web portal supports application deployment and artifact storage. In summary, the platform enables persisting streaming data, analyzing that data in real time, and supports historical analysis on the stored stream.