Matt Aslett's Analyst Perspectives

Ventana Research recently published the 2023 Analytic Data Platforms Value Index. As organizations strive to be more data-driven, increasing reliance on data as a fundamental factor in business decision-making, the importance of the analytic data platform has never been greater. In this post, I’ll share some of my observations about how the analytic data platforms market is evolving.

Our Value Index for Analytic Data Platforms represents technology vendors and products designed to store, manage, process and analyze data across on-premises, hybrid and multicloud environments, enabling organizations to maximize data for insights leading to greater operational efficiency. Analytic data platforms support applications used to analyze the business, including decision support, business intelligence, data science and artificial intelligence and machine learning. They include real-time analytics data engines, data warehouses and data lakes as well as the increasing convergence of data warehouse, data lake and data-streaming technologies.

The market for analytic data platforms has been dominated by the relational data model and relational database management systems since the 1980s. More recently, data processing frameworks, such as Apache Spark and Apache Hadoop, have been used to form the basis of a data platform by providing data persistence, data management, data processing and data query functionality. Early data lake projects — designed to economically store and process large volumes of raw data — were primarily based on Hadoop. Today, adoption of cloud-computing environments has led to the increasing use of object stores as an underlying data persistence layer for analytic data platforms, with separate cloud products and services providing the data management, processing and query functionality. Object storage has become the dominant platform for data lakes in recent years. Our Analytics and Data Benchmark Research shows that more than one-half of participants (53%) are using object stores in analytics efforts, and a further 18% plan to do so within two years.

While cloud-based object storage provides a low-cost environment for storing large volumes of data, it lacks structured data management and processing functionality to support multiple BI projects as well as data science and operational applications. Accelerating the analysis of data in data lake environments is a key trend driving the analytic data platform space. At Ventana Research, we call this trend hydroanalytics, and have identified two key architecture patterns utilized to deliver it. The data lakehouse concept is designed to incorporate data warehousing functionality — such as structured data management and processing functionality, including support for table formats, metadata management and transactional updates and deletes — into the data lake itself. I assert that by 2026, three-quarters of data lake users will adopt emerging table formats to support transactional consistency when inserting, updating and deleting data in data lake environments.

The other approach also utilizes a data lake for low-cost storage. However, an associated data warehouse provides the ability to persist curated subsets of structured data and apply predetermined schema, enabling users to take advantage of established data warehousing functionality for high-performance and high-concurrency query requirements.

Regardless of the approach taken, migration of analytic workloads to the cloud is a significant trend in the analytic data platform sector. Most analytic data platforms were traditionally deployed on-premises, but organizations are increasingly deploying analytic data platforms on cloud infrastructure or using analytic data platform functionality delivered as managed cloud services. Ventana Research’s Analytics and Data Benchmark research finds that two-thirds of organizations have a primary data platform for analytics in the cloud. That could be a data lake, a data warehouse or a combination of the two. One approach does not suit all use cases, and organizations use a variety of data platforms to fulfill the spectrum of requirements for a myriad of BI needs. Ventana Research’s Data Lakes Dynamics Insights research also shows that, to date, less than one-quarter of organizations have adopted a data lake to replace an existing data warehouse environment, and data lake and data warehouse environments coexist in almost three-quarters of organizations.

Another key trend in the data platform sector is the blurring of the lines between operational and analytic workloads. The development of intelligent applications infused with the results of analytic processes, such as personalization and AI-driven recommendations, provides a set of workloads that span traditional requirements. While this impacts the requirements for operational data platforms, it does not eradicate the need for analysis of data in a separate analytic data platform to support BI and data science projects.

At Ventana Research, we continue to believe that, for most use cases, there is a clear, functional requirement for either analytic or operational data platforms. The Analytic Data Platforms Value Index reflects this by assessing products positioned as analytic data platforms on the ability to serve the specific requirements of analytic use cases. From a Capability perspective, this was reflected by a greater emphasis in our assessment of analytic data platforms on their support for:

• Query optimization and query management.
• Native data visualization, graph and spatial analysis.
• Native machine learning and statistical analytics.
• Native data engineering functionality.
• Data virtualization and query federation.
• BI and ML vendor partnerships.

Organizations should monitor the evolution of functionality in the analytic data platform sector, including the arguments for, and against, in-database machine learning, as well as increased integration of data engineering and visualization functionality. The results of our analysis are reported in our 2023 Analytic Data Platforms Value Index. We encourage you to review the results and consider how each of these vendors can support the needs of your organization.

Regards,

Matt Aslett