[DISCUSS] ASF Project Spotlight article for Apache Iggy

[justinmclean]

Now that 0.9.0 vote is over, I would like to put Iggy forward for an ASF Project Spotlight.

The Spotlight is an interview-format article on the ASF blog. The questions cover what the project is, why it was built, how the community works and where it is heading. The Iceberg one is a good example.

I will write it up and share it here for comments.

[justinmclean]

ASF Project Spotlight: Apache Iggy

Piotr Gankiewicz is the original creator of Apache Iggy, a member of the Apache Iggy PMC, and co-founder and CTO of LaserData. He previously worked as a software architect in fintech, trading systems and cybersecurity, was a Microsoft MVP for his open source work, and co-founded DevMentors.io.

Kranti Parisa is Chair of the Apache Iggy PMC and co-founder and CEO of LaserData, and led Iggy's journey into the ASF. He previously led large-scale search and personalisation platforms at Apple and Comcast and headed product engineering at Dialpad. A longtime Apache contributor to Lucene, Solr and Camel, he co-authored the books Apache Solr Enterprise Search Server and O'Reilly’s award-winning Reliable Machine Learning.

Apache Iggy is one of the ASF's newest Top-Level Projects. In this Spotlight, Piotr and Kranti talk about the engineering choices behind it, the long road to clustering, and the community that has grown around it.

What Is Apache Iggy and Why It Exists

Q: Can you tell us a bit about Apache Iggy?
A: Apache Iggy is a persistent message streaming platform written in Rust. It stores messages in streams, topics, and partitions as an append-only log, while consumers read independently or through consumer groups. The server is distributed as a single binary with no external broker or database dependency. It supports TCP, QUIC, WebSocket, and HTTP, and provides SDKs for Rust, C#, C++, Java, Go, Python, Node.js, and PHP. The project also includes a CLI, Web UI, benchmarking tooling, an MCP server, and a connector runtime with sources and sinks for systems including PostgreSQL, Elasticsearch, MongoDB, Apache Iceberg, ClickHouse, Delta Lake, and Apache Doris and more. Iggy became a Top-Level Project on 19 August 2026 and released 0.9.0, its first version with replication, 21st September 2026.

Q: Apache Kafka and other streaming platforms already existed. Why build another one?
A: Previous generations of streaming systems proved the importance of durable, ordered event logs. But many were designed around the hardware, runtimes, and application architectures of an earlier era. Iggy began with a different question: if we designed a persistent streaming engine from scratch today, what would we do differently? The project uses Rust and modern systems techniques including thread-per-core execution, shared-nothing architecture, Linux io_uring, zero-copy data paths, and Direct I/O. The objective is not to reproduce Kafka with a different implementation language. It is to rethink the streaming engine around predictable low tail latency, high throughput, resource efficiency, a smaller operational footprint, and a simpler path from application code to durable streams.

Streaming Beyond Server-to-Server

Q: Why is multi-protocol support a core part of Iggy’s design?
A: Real-time systems are no longer composed only of backend services speaking one broker protocol. Producers and consumers increasingly include backend services, browsers, developer tools, edge applications, gateways, data platforms, and AI-oriented systems. Iggy therefore treats protocol choice as part of the developer experience. TCP provides a direct high-performance path, QUIC brings a modern transport option, WebSocket makes persistent bidirectional streaming natural for browser and interactive applications, and HTTP lowers the integration barrier for environments where a dedicated streaming client is impractical. These protocols expose the same durable streaming model rather than creating separate products for separate application surfaces.

Q: How does that change the kinds of applications a streaming engine can serve?
A: A conventional broker is often deployed primarily as infrastructure between backend services. Iggy’s protocol surface is intended to make the durable log directly useful across a wider range of application boundaries. The same underlying stream can participate in service-to-service eventing, telemetry and observability, real-time user experiences, data ingestion, CDC, AI and agent event flows, video or IoT pipelines, and movement of data into databases, search systems, analytic engines, and data lakes. The architectural idea is simple: durable streaming should be a reusable systems primitive, not an integration pattern limited to one class of client.

Developer Experience as a Systems Feature

Q: Why invest in native SDKs across so many languages?
A: A streaming platform is only as usable as the path from an application to the stream. Iggy maintains SDKs across major application ecosystems including Rust, C#, C++, Java, Go, Python, Node.js, and PHP. The goal is to let teams use the project from the language and runtime they already operate, with idiomatic APIs and runnable examples, rather than forcing every team through a single language binding or a bespoke integration layer. Community members helped drive this breadth early in the project by contributing SDKs for the languages they used themselves.

Q: Where do connectors fit into that strategy?
A: Protocols and SDKs solve application connectivity; connectors solve system connectivity. Iggy’s connector runtime links streams to operational databases, search engines, analytics systems, and lakehouse storage. Together, the protocol, SDK, and connector layers are intended to reduce the amount of glue code required to move real-time data through a heterogeneous architecture.
Designed Around Modern Systems Engineering

Q: What are the key architectural choices behind Iggy?
A: Iggy uses a thread-per-core, shared-nothing design. Each CPU core runs its own thread without shared hot-path state, and Linux io_uring is used for disk and network I/O. Because the server is written in Rust, it has no garbage collector. The project has also invested in zero-copy data paths and Direct I/O. These choices are aimed at reducing scheduling, synchronization, memory-management, and I/O overhead that can show up as latency variance under load.

Q: Why focus so much on tail latency and efficiency?
A: Average latency can hide the behavior users actually experience under load. Streaming infrastructure sits on the path of many downstream systems, so slow outliers can propagate through an architecture. Iggy’s engineering work therefore emphasizes predictable tail behavior as well as throughput. The project’s published benchmark shows a single-node producer P99 latency of about 1 ms, and the project provides iggy-bench so users can measure performance on their own hardware rather than relying only on published numbers.

Q: What changed with clustering?
A: Clustering was one of the project’s most requested capabilities. An early experiment used Raft, but the team later moved to Viewstamped Replication Revisited (VSR) based consensus, chosen for the project’s high-throughput replication goals. The groundwork landed in 0.7.0, and 0.9.0 is the first release designed to run as a multi-node cluster. Single-node operation remains available for simpler deployments.

Real-World Use and Interoperability

Q: What kinds of workloads is the community targeting?
A: Apache Iggy is designed for real-time, data-intensive workloads where predictable latency, throughput, and efficiency matter. Current use cases and integrations span event streaming, logs and observability, real-time analytics, CDC and data pipelines, AI and agent data streams, video and IoT ingestion, and high-volume movement between operational systems and downstream stores. Apache Camel includes a camel-iggy component, LakeSail pairs Iggy with its Sail compute engine, and AWeber’s Imbi platform engineering tool uses Iggy for message streaming. The project also maintains a Powered By page where adopters can share their use cases.

Q: Does adopting Iggy require an all-or-nothing migration?
A: No. Interoperability is an important direction for the project. The community is developing a Kafka protocol gateway so existing Kafka applications can work with Iggy, while the connector ecosystem provides paths into databases, search engines, analytic systems, and lakehouse technologies. This lets teams evaluate Iggy at a specific boundary or workload rather than treating adoption as a wholesale replacement exercise.

From an Experiment to an Apache Top-Level Project

Q: When was Iggy started and how did it reach the ASF?
A: Piotr Gankiewicz started Iggy in March 2023 as an exploration of Rust and the internals of message streaming. What began as a basic append-only log server attracted contributors who expanded the engine and built SDKs for their own languages. In early 2025, contributors chose a path intended to make the project independent of any one person or company. Iggy entered the Apache Incubator in February 2025 and became an Apache Top-Level Project on 19 August 2026.

Q: How has the community grown?
A: The first major wave of contributors arrived after technical posts about building a message streaming system in Rust reached Reddit and Hacker News. Today the project reports more than 180 contributors, nearly 3,000 pull requests, more than 4,900 GitHub stars, and more than 700 Discord members. Around 60 people contributed to the 0.9.0 release. Just as importantly, contributors have progressed into committer and PMC roles as the project has matured.

The Apache Way: Community Over Code

Q: Why was Apache governance important for a systems project like Iggy?
A: Streaming infrastructure can become a long-lived dependency at the center of an architecture. Users need confidence not only in performance and features, but also in the project’s governance and continuity. At the ASF, technical and project decisions are made in the open on the project mailing list, contributors can earn greater responsibility through sustained participation, and the project is not governed according to one company’s product strategy. That neutrality is especially important for infrastructure intended to connect many languages, protocols, and data systems.

Q: How does Iggy embody “community over code”?
A: Iggy began as one person’s project and is now governed by its community. Regular contributors have become committers and PMC members, discussions and decisions take place openly, and the community is creating more opportunities for users to share what they are building, including regular meetups. Code is only one form of contribution: documentation, examples, issue reports, integrations, user stories, technical discussions, and helping other adopters all strengthen the project.
The Next Generation of Streaming Workloads

Q: What is changing in the streaming landscape?
A: The role of streaming is broadening. Real-time data increasingly crosses boundaries between services, interactive applications, operational databases, analytical systems, data lakes, and AI-oriented workloads. That creates pressure for streaming infrastructure to be efficient enough to sit on more data paths, accessible from more environments, and interoperable with a wider ecosystem. Iggy’s response is to combine a durable append-only log with modern systems engineering, multiple native protocols, broad language support, and connectors rather than optimizing only for a single server-to-server integration model.

Q: What does the future hold for Apache Iggy?
A: The immediate focus is hardening clustering on the path to 1.0. A Kafka protocol gateway is in progress, and the community is exploring multi-leader replication per partition, tiered storage, a message registry, embedded key-value storage, and additional connectors. The broader direction remains consistent with the project’s original question: how should durable streaming evolve when modern hardware, application architectures, and real-time workloads look very different from the environment in which the previous generation of streaming platforms was designed?

Getting Started and Getting Involved

Q: What is the best way to try Iggy?
A: Start with the official documentation and run the server with Docker or from source. Build a producer and consumer using the SDK that matches your application stack, then use iggy-bench to test the workload on your own hardware. The published benchmark site provides a reference point, while the project’s Discord and dev mailing list are the places to ask questions and discuss designs with the community.

Q: How can people contribute beyond code?
A: Try Iggy on a real workload and report what you learn. Improve documentation and examples. Help answer questions. Share an integration or operational pattern. Add your organization and use case to the Powered By page. Present at a community meetup. For code contributions, the project’s good first issues provide an accessible starting point.

Apache Iggy Resources

Website: https://iggy.apache.org
Documentation: https://iggy.apache.org/docs/
GitHub: https://github.com/apache/iggy
Benchmarks: https://benchmarks.iggy.apache.org
Mailing lists: https://iggy.apache.org/community/mailing-lists/
Discord: https://discord.gg/apache-iggy
Powered By: https://iggy.apache.org/powered-by/

[kparisa]

looks great. +1

[krishvishal]

Looks good. +1

[hubcio]

lgtm +1