Using our research, best practices and expertise, we help you understand how to optimize your business processes using applications, information and technology. We provide advisory, education, and assessment services to rapidly identify and prioritize areas for improvement and perform vendor selection
We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.
Services for Technology Vendors
We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.
I recently wrote about the development, testing and deployment of data pipelines as a fundamental accelerator of data-driven strategies. As I explained in the 2023 Data Orchestration Buyers Guide, today’s analytics environments require agile data pipelines that can traverse multiple data-processing locations and evolve with business needs.
Given the increasing complexity of evolving data sources and requirements, it’s essential to automate and coordinate the creation, scheduling and monitoring of data pipelines. This is the realm of data orchestration, which enables the flow of data across the organization via capabilities for pipeline monitoring, pipeline management and workflow management.
Traditional approaches to data management are rooted in point-to-point batch data processing, whereby data is extracted from its source, transformed for a specific purpose and loaded into a target environment for analysis. These approaches are unsuitable for the demands of today’s analytics environments, which require sequential or parallel execution of a complete set of tasks via data pipelines, typically based on directed acyclic graphs that represent the relationships and dependencies between the tasks. I assert that by 2027, more than one-half of enterprises will adopt data orchestration technologies to automate and coordinate data workflows and increase efficiency and agility in data and analytics projects.
At the highest level of abstraction, data orchestration covers three key capabilities: collection (including data ingestion, preparation and cleansing), transformation (additionally including integration and enrichment) and activation (making the results available to compute engines, analytics and data science tools or operational applications). Whether stand-alone or embedded in larger data-engineering platforms, data orchestration has the potential to drive improved efficiency and agility in data and analytics projects. Data orchestration addresses one of the most significant impediments to generating value from data. Participants in Ventana Research’s Analytics and Data Benchmark Research cite preparing data for analysis and reviewing data for quality and consistency issues as the two most time-consuming tasks in analyzing data.
The development and orchestration of agile data pipelines is an important aspect of Data Operations, which provides an overall approach to automate data monitoring and the continuous delivery of data into operational and analytical processes through the application of agile development, DevOps and lean manufacturing by data engineering professionals in support of data production. Data orchestration is also integral to the development and delivery of applications driven by artificial intelligence and generative AI.
Almost one-half (49%) of participants in ISG’s 2023 Application Development and Maintenance Study expect to AI-enable applications by embedding AI and ML models into current applications and processes. Data orchestration automates and accelerates the flow of data from multiple sources, including existing applications and data platforms with the output of large language models and vector databases, complementing MLOps, which serves the collection of artifacts and orchestration of processes necessary to deploy and maintain AI/ML models.
Agile and collaborative practices were a core component of the Capabilities criteria we used to assess data pipeline tools in the 2023 Data Orchestration Buyers Guide, alongside the functionality required to support pipeline monitoring, pipeline management and workflow management as well as integration with the wider ecosystem of DevOps, data management, DataOps and business intelligence and AI tools and applications.
The orchestration of data pipelines is just one aspect of improving the use of data within an enterprise. In addition to the development, testing, and deployment of data pipelines, DataOps also encompasses data observability, which I will explore in greater detail in a forthcoming Analyst Perspective. Nevertheless, I recommend that all enterprises explore how the orchestration of data pipelines can help increase the potential for improved data-driven decision-making as part of a broader evaluation of the people, processes, information and technology improvements required to deliver data-driven decision-making.
Regards,
Matt Aslett
Matt Aslett leads the software research and advisory for Analytics and Data at Ventana Research, now part of ISG, covering software that improves the utilization and value of information. His focus areas of expertise and market coverage include analytics, data intelligence, data operations, data platforms, and streaming and events.
Ventana Research’s Analyst Perspectives are fact-based analysis and guidance on business,
Each is prepared and reviewed in accordance with Ventana Research’s strict standards for accuracy and objectivity and reviewed to ensure it delivers reliable and actionable insights. It is reviewed and edited by research management and is approved by the Chief Research Officer; no individual or organization outside of Ventana Research reviews any Analyst Perspective before it is published. If you have any issue with an Analyst Perspective, please email them to ChiefResearchOfficer@ventanaresearch.com