Matt Aslett's Analyst Perspectives

As data continues to grow and evolve, organizations seek better tools and technologies to employ data faster and more efficiently. Finding and managing data remains a perennial challenge for most organizations, and is exacerbated by increasing volumes of data and an expanding array of data formats. At the same time, organizations must comply with a growing list of national and regional rules and regulations, such as General Data Protection Regulation and the California Consumer Privacy Act. While these regulations protect consumers, they increase complexity for governing and providing access to data.

I have previously written about the importance of data democratization as a key element of a data-driven agenda. Removing barriers that prevent or delay users from gaining access to data enables it to be treated as a product that is generated and consumed, either internally by employees or externally by partners and customers. This is particularly important for organizations adopting the data mesh approach to data ownership, access and governance. Data mesh is an organizational and cultural approach to data, rather than a technology platform. Nevertheless, multiple vendors are increasingly focused on providing products that facilitate adoption of data mesh and promote data democratization. Amazon Web Services is one such vendor, thanks to the recent launch of Amazon DataZone, one of the figurehead analytics and data announcements made during the company’s recent re:Invent customer event.

The market for data and analytics products is constantly evolving, with the emergence of new approaches to data persistence, data processing and analytics. This enables organizations to constantly adapt data analytics architecture in response to emerging functional capabilities and business requirements. It can, however, also be a challenge. Investments in data platforms cannot be constantly written-off as organizations adopt new products for new approaches. Too little change can lead to stagnation, but too much change can be chaotic, leading to silos of data and data integration complexity. This is one reason why there is growing interest in the concept of data fabric for managing and governing data across distributed environments. In addition to supporting hybrid and multi-cloud strategies, data fabric enables organizations to manage and generate insight from data spread across a combination of long-standing and new data platforms. Promethium focuses on automating data management and data governance across a distributed architecture with a combination of data fabric and self-service augmented analytics capabilities.

Data observability was a hot topic in 2022 and looks likely to be a continued area of focus for innovation in 2023 and beyond. As I have previously described, data observability software is designed to automate the monitoring of data platforms and data pipelines, as well as the detection and remediation of data quality and data reliability issues. There has been a Cambrian explosion of data observability software vendors in recent years, and while they have fundamental capabilities in common, there is also room for differentiation. One such vendor is Soda Data, which offers an open-source platform for self-service data observability that is focused on facilitating collaboration between business decision-makers and data teams responsible for generating and managing data to improve trust in data.

Organizations across various industries collect multiple types of data from disparate systems to answer key business questions and deliver personalized experiences for customers. The expanding volume of data increases complexity, and data management becomes a challenge if the process is manual and rules-based. There can be numerous siloed, incomplete and outdated data sources that result in inaccurate results. Organizations must also deal with concurrent errors – from customers to products to suppliers – to create a complete view of the data. Many vendors, including Tamr, have turned to artificial intelligence and machine learning to overcome the challenges associated with maintaining data quality amid the growing volume and variety of data. I assert that by 2026, more than three-quarters of organizations’ data management processes will be enhanced with artificial intelligence and machine learning to increase automation, accuracy, agility and speed.

Despite the emphasis on organizations being more data-driven and making an increasing proportion of business decisions based on data and analytics, it remains the case that some of the most fundamental questions about an organization are difficult to answer using data and analytics. Ostensibly simple questions such as, “how many customers does the organization have?” can be fiendishly difficult to answer, especially for organizations with multiple business entities, regions, departments and applications. Increasing volumes and sources of data can hinder, rather than help. Only 1 in 5 participants (20%) in Ventana Research’s Analytics and Data Benchmark research are very confident in their organization’s ability to analyze the overall quantity of data. This is a perennial issue that data and application integration vendors, such as SnapLogic, are aiming to address – increasingly through automation and products for business users as well as data management professionals.

Ventana Research recently announced its 2023 Market Agenda for Data, continuing the guidance we have offered for two decades to help organizations derive optimal value and improve business outcomes.

Data observability is a hot topic and trend. I have written about the importance of data observability for ensuring healthy data pipelines, and have covered multiple vendors with data observability capabilities, offered both as standalone and part of a larger data engineering system. Data observability software provides an environment that takes advantage of machine learning and DataOps to automate the monitoring of data quality and reliability. The term has been adopted by multiple vendors across the industry, and while they all have key functionality in common – including collecting and measuring metrics related to data quality and data lineage – there is also room for differentiation. A prime example is Acceldata, which takes a position that data observability requires monitoring not only data and data pipelines but also the underlying data processing compute infrastructure as well as data access and usage.

The shift from on-premises server infrastructure to cloud-based and software-as-a-service (SaaS) models has had a profound impact on the data and analytics architecture of many organizations in recent years. More than one-half of participants (59%) in Ventana Research’s Analytics and Data Benchmark research are deploying data and analytics workloads in the cloud, and a further 30% plan to do so. Customer demand for cloud-based consumption models has also had a significant impact on the products and services that are available from data and analytics vendors. Data platform providers, both operational and analytic, have had to adapt to changing customer demand. The initial response — making existing products available for deployment on cloud infrastructure — only scratched the surface in terms of responding to emerging expectations. We now see the next generation of products, designed specifically to deliver innovation by taking advantage of cloud-native architecture, being brought to market both by emerging startups, and established vendors, including InterSystems.

Earlier this year, I wrote about the increasing importance of data observability, an emerging product category that takes advantage of machine learning (ML) and Data Operations (DataOps) to automate the monitoring of data used for analytics projects to ensure its quality and lineage. Monitoring the quality and lineage of data is nothing new. Manual tools exist to ensure that it is complete, valid and consistent, as well as relevant and free from duplication. Data observability vendors, including Monte Carlo Data, have emerged in recent years with the goal of increasing the productivity of data teams and improving organizations’ trust in data using automation and artificial intelligence and machine learning (AI/ML).