Matt Aslett's Analyst Perspectives

I recently wrote about the various technologies used by organizations to process and analyze data in real time. I explained that while the terms streaming data and events and streaming analytics are often used interchangeably, they are separate disciplines that make use of common underlying concepts and technologies such as events, event brokers and event-driven architecture. Confluent’s acquisition of Immerok earlier this year provided a reminder of this fact. Confluent is one of the most prominent vendors in the streaming data and events sector, known for its event/stream processing and governance software, cloud services and expertise. The company acquired Immerok to boost its capabilities and expertise with real-time processing and analytics, enhancing its ability to address the full spectrum of stream processing use cases.

Confluent was founded in 2014 by the creators of the Apache Kafka open-source project developed at LinkedIn to store and process data related to member activity as well as logs and metrics. Based on a publish-and-subscribe messaging model for communicating events and event streams, Apache Kafka has become widely adopted by numerous companies to support real-time data processing and forms the core of the Confluent Platform software distribution and Confluent Cloud managed service.

The company reported total revenue of $586 million in fiscal year 2022, up 51% from $388 million the previous year, and forecasts revenue between $760 million and $765 million in fiscal 2023. Confluent has benefited from increased adoption of streaming data and event processing in general as well as expansion and greater maturity among existing adopters, both of which have been fueled by its investment in additional features that separate its commercial offerings from open-source Apache Kafka.

The Kora engine that underpins Confluent Cloud provides additional capabilities related to tiered storage, elastic scaling, high availability and performance, while the company has also invested in enterprise-grade security and governance capabilities. As I described last year, governance has been a key focus for the company, with its Stream Governance suite providing capabilities for schema management and data quality as well as self-service data discovery and classification and stream lineage. Confluent recently announced the addition of Data Quality Rules to Stream Governance suite to ensure the integrity of event data as it flows through an organization as well as stream sharing to enable the sharing of data from Confluent Cloud to other organizations (such as partners and suppliers) that are also using Apache Kafka.

The focus on operational enhancements is understandable, given that a higher proportion of participants in Ventana Research’s Analytics and Data Benchmark Research currently use streaming data in operational processes (48%) more than analytics processes (44%). Greater growth in the use of streaming data in analytic processes is anticipated, however, supporting Confluent’s recent move to expand its stream processing and analytics capabilities through the acquisition of Immerok, one of the primary companies behind the Apache Flink stream processing engine.

Immerok was founded in 2022 by a distributed team of Apache Flink experts to build a cloud-based service around that software. Apache Flink is an open-source stream-processing engine that emerged in 2014 from Stratosphere, a research project collaboration involving the Technical University Berlin, Humboldt University and the Hasso Plattner Institute. It complements Apache Kafka by providing a distributed processing engine for performing stateful computations over unbounded and bounded streams of events, enabling users to create applications for SQL-based streaming and batch analytics of event data.

There is already heavy overlap amongst users of Apache Kafka and Apache Flink: a large percentage of Apache Flink users are also using Apache Kafka. While fewer Apache Kafka users are taking advantage of Apache Flink today, this reflects the relative age and maturity of Apache Flink compared to Apache Kafka and indicates an opportunity for Confluent to provide additional value to its customer base by facilitating the expansion of event-processing initiatives to address streaming analytics. We assert that by 2025, one-half of organizations will incorporate streaming analytics into business processes, enabling faster response to opportunities and threats.

Immerok was attractive to Confluent both for the expertise of its workers in relation to stream processing and analytics and also for its managed Flink cloud service. The latter is being added to Confluent Cloud, initially as a serverless SQL API endpoint, the early access program for which was launched in May. The purchase of Immerok is designed to accelerate Confluent’s capabilities for real-time processing but is not the company’s first investment in stream processing and analytics. It is also partially complementary to Confluent’s existing offerings. Rather than a standalone stream-processing engine, Kafka Streams is a client library embedded in applications to enable the processing of streams of Kafka event messages in real time. Meanwhile, ksqlDB is a streaming database built on Kafka Streams that provides a SQL query layer for developing event streaming applications. Confluent does not have plans to deliver an on-premises offering based on Apache Flink, so ksqlDB’s availability both on-premises and in Confluent Cloud means that it will continue to play a role in supporting on-premises stream processing requirements, as well as facilitating migration to Flink on Confluent Cloud.

Work remains to integrate Apache Flink into Confluent’s product portfolio and articulate how customers should determine when to use Apache Flink rather than Kafka Streams or ksqlDB, but the acquisition of Immerok by Confluent will better enable the company to address the full spectrum of event and streaming use cases, both operational and analytic. I recommend that any organization considering investing in streaming data and event processing, including streaming analytics, include Confluent in the evaluations.

Regards,

Matt Aslett