Significant progress has been made in data consolidation that is associated with data lake initiatives. However, the ongoing creation of new systems hosting silos of information continues to complicate the efforts of both IT professionals and the data consumer communities. It is becoming increasingly clear that having a single, consolidated data analytics repository may not be a realistic goal. In order to enable a broad community of data consumers, organizations must provide tools that can handle the complexity of these complex data environments.
The proliferation of multiple data initiatives and new data silos appearing outside of any data lake boundary creates an environment where many potential data consumers cannot find or use critical information. Unfortunately, the systems that provide the best features for consolidating enterprise class data into data lakes are typically designed for professional data engineers who are comfortable with software development. The latest generation of tools available to transform, or
munge, data into consolidated lakes have greatly improved in scalability. However, they have not addressed ease-of-use limitations that prevent usage by many types of users with important, data-driven business use case needs.
In order to realize an attractive positive ROI from data consolidation investments by organizations, there must be a parallel effort to enable efficient data access for everyone with a demonstrable business need. This effort requires a differentiated set of tools for data discovery and access that must match the skills of diverse target users.
Most organizations today lack such an efficient process, where large groups of business users can uncover valuable data relationships between the information in a data lake and the many
satellite data sources. The typical approach today relies too heavily on investments from IT in order to provide data extracts to feed the needs of people familiar with the relevant business concepts. Those users lack the required source system access and the coding skills to explore the data on their own.