Tag: karafka

Karafka 2.6 and Web UI 1.0: Laying the Groundwork for Kafka Queues

August 5, 2026 / Maciej Mensfeld

I'm happy to announce that Karafka 2.6 and Karafka Web UI 1.0 have just been released.

For those new here: Karafka is a Ruby and Rails multi-threaded, efficient Kafka processing framework, and its Web UI is a monitoring and management dashboard that ships alongside it. As with every release in the 2.x line, this is a continuation rather than a rewrite - you upgrade, apply a couple of small changes, and keep going.

On the surface, 2.6 is a focused set of features - redesigned Declarative Topics, dynamic worker pool scaling, a new low-level offsets API, and lag compensation for paused partitions. Underneath, it is the largest internal reorganization the framework has seen in years. Almost none of that groundwork is directly visible to you today, and that is the point: it is the foundation for where Karafka is going next.

This article covers the most significant changes rather than every one. For the full list, the Karafka changelog and Web UI changelog are the source of truth.

The Bigger Picture: Kafka Queues (KIP-932)

Let me start with the direction, because it explains most of this release.

While ago Kafka received a fundamentally new way to consume data. KIP-932 - "Queues for Kafka" - introduced Share Groups, a cooperative model that sits alongside the classic consumer group. Instead of partitions being exclusively assigned to a single consumer, share groups let multiple consumers cooperatively pull from the same partitions and acknowledge individual records. In practice, this brings queue-like semantics to Kafka: work-queue fan-out, per-message acknowledgement, and consumer counts no longer capped by partition count.

The rest of the stack is now catching up to the broker. librdkafka is gaining Share Group support, and I'm building the rdkafka-ruby bindings for it as we speak - the layer Karafka sits on top of. Bringing this all the way up into Karafka is a multi-step journey across the whole stack, and 2.6 is where the framework's part of it begins.

For a framework like Karafka, this is not a small bolt-on. Consumer groups are woven into the fabric of the processing, routing, connection, and instrumentation layers. Share groups need their own parallel strategies, coordinators, jobs, and callbacks - and layering a second, coexisting group type on top cleanly is only possible once the consumer-group-specific code is isolated in its own namespace.

Share Groups are not in 2.6. What is in 2.6 is the mandatory first step: Karafka reorganizes all of those layers into consistent ConsumerGroups namespaces, introduces group-type-agnostic routing accessors, and threads a parallel group / group_id vocabulary through instrumentation payloads. It is deliberately invisible plumbing - and it is what makes Kafka Queues in Karafka tractable rather than a rewrite.

Sponsorship and Community Support

Karafka's progress continues to be powered by the people and companies who fund it, report issues, review pull requests, and run it in production at a scale I could never reproduce alone. To everyone who sponsors the project, contributes code, files detailed bug reports, or helps another user in Slack: thank you. The scope here - dozens of fixes, a major internal reorganization, and the beginning of the Share Groups journey - is only sustainable because Karafka Pro and Enterprise customers let me treat this as serious, ongoing engineering. The more successful the commercial side becomes, the more I can give back to OSS.

Karafka Framework

Redesigned Declarative Topics

The Declarative Topics system now lives in a standalone declaratives.draw DSL, independent of routing.

This resolves a mismatch that grew as adoption spread: many teams use Karafka as the single source of truth for their entire topic infrastructure, not just the topics they consume. Previously, declarations were embedded in routing - so managing a topic (another team's service, a shared audit log, a produce-only sink) meant adding it to your routing, implying consumption intent and dragging all the consumer machinery along. Separating them makes each purpose explicit: routing describes what you consume and how, declaratives describe what topics exist and how they're configured.

class KarafkaApp < Karafka::App
  declaratives.draw do
    defaults do
      replication_factor 3
    end

    topic :orders do
      partitions 6
      config('retention.ms': 86_400_000)
    end

    # Produce-only or owned elsewhere - no routing entry needed
    topic :audit_log do
      partitions 3
      config('cleanup.policy': 'compact')
    end
  end

  routes.draw do
    topic :orders do
      consumer OrdersConsumer
    end
  end
end

The old routing-based config() approach is deprecated but still works in 2.6, so there is no forced migration.

Dynamic Worker Pool Scaling

The worker thread pool can now be scaled at runtime without restarting - handy for time-of-day load, external signals, or ramping up after a deploy.

Karafka::Server.workers.scale(10) # add threads immediately (synchronous)
Karafka::Server.workers.scale(3)  # drain down gracefully (asynchronous)
Karafka::Server.workers.size

Scaling up is synchronous; scaling down lets workers exit as they finish in-flight jobs. Both directions emit worker.scaling.up / worker.scaling.down events. config.concurrency still sets the initial pool size at boot.

Lag Compensation for Long-Paused Partitions

librdkafka refreshes watermark offsets and lag only from fetch responses, so a long-paused partition reports frozen lag in statistics.emitted - and in everything built on it, including the Web UI. Karafka Pro can now compensate: when enabled, it periodically refreshes the watermarks and lags of long-paused partitions through the running connection and overlays them onto the emitted statistics, handing back to live stats on resume. It's opt-in and off by default for now:

config.internal.statistics.consumer_groups.lag_compensation.interval = 30_000
config.internal.statistics.consumer_groups.lag_compensation.pause_age = 30_000

More Robust Error Handling

Two behavioral changes landed in the consumption error path.

Non-StandardError exceptions (like ScriptError) are no longer silently skipped - they flow through the normal retry / pause / DLQ path.
Process-critical errors (SystemExit, SignalException, NoMemoryError) are recorded, keep the partition paused, and trigger a graceful shutdown via the auto-subscribed Instrumentation::CriticalErrorsListener rather than being retried or dispatched to a DLQ.

If your consumers can raise non-StandardError exceptions, the outcome now differs from 2.5.

Performance and Ractors

Several internal admin operations that issued N sequential per-partition calls are now single batched calls - watermark reads resolve in two calls regardless of partition count. This is why 2.6 requires karafka-rdkafka >= 0.28.0, which exposes list_offsets and rebuilds consumer #lag on top of it.

Ractor-based parallel deserialization is implemented and works, but I've deliberately held it back from 2.6. This release already carries a large volume of internal change, and stacking Ractors on top would make it much harder to reason about anything that surfaces in production. They'll ship once the 2.6 internals settle - I'd rather isolate risk than bundle two big unknowns into one version.

Karafka Web UI 1.0

After a long run of production-hardened 0.x releases, the Web UI graduates to 1.0.

Why 1.0, and Why Now

The 0.x numbering was always a conservative signal, not a reflection of stability. The Web UI has been production-ready and free of major breaking changes for a very long time, so 1.0 is, first and foremost, an honest version number. There are only small breaking changes in the configuration between 0.11.7 and 1.0. From 3.0 onward, Web UI versioning will align with Karafka's major version and move in lockstep.

Modernized CSRF Protection

The old token-based CSRF approach (route_csrf) is replaced with header-based protection using the browser-enforced Sec-Fetch-Site header (sec_fetch_site_csrf). Modern browsers always send it and it can't be forged cross-origin, so protection is simpler and more robust - with no tokens to thread through your views. For virtually everyone this is transparent; only non-browser clients hitting unsafe methods directly must send Sec-Fetch-Site: same-origin.

Better Monitoring and a More Consistent Interface

Poll interval monitoring: consumer reporting now tracks poll_interval (max.poll.interval.ms) per subscription group, so you can catch a slow consumer before Kafka evicts it. (Consumer schema bumped to 1.7.0.)
Runtime-aware worker count: the UI reads the live count from Karafka::Server.workers.size, reflecting dynamic pool scaling accurately.
Consistency and polish:
- a standardized empty-state component across every list view
- topic/partition/offset coordinates in the Explorer and Errors views now link straight to the relevant message
- Pro gating hardened so a misclick no longer navigates away to an upsell page; rel="noopener noreferrer" on every external link
- and a whole class of overflow bugs from long topic names fixed.

Dozens of Bug Fixes

Beyond the headline features, this release is genuinely fix-heavy - roughly 30 fixes in Karafka 2.6 and around 25 more in Web UI 1.0. A representative sample from the framework:

Reset the per-partition retry counter on revocation, so a message reclaimed after a rebalance isn't wrongly treated as retry-exhausted and dispatched to the DLQ early.
Return false from #mark_as_consumed / #commit_offsets! when the partition was lost (previously could return true in some cases).
Reset seek_offset only after a successful #seek, so a raising seek no longer skips the rest of a batch.
Leave tombstone records untouched on encrypted produce/consume instead of crashing, keeping them valid for log compaction (Pro).
Reset ActiveJob CurrentAttributes in an ensure, so a failed job's attributes no longer leak into the next job.

Upgrade Notes

The 2.5 → 2.6 upgrade is intentionally small for most applications, but there are a few breaking changes, behavioral shifts, and internal namespace moves worth knowing about before you deploy. Web UI 1.0 requires Karafka 2.6, so upgrade them together. Rather than repeat it all here, read the Karafka 2.6 upgrade guide and the Web UI 1.0 upgrade guide - they walk through every required action step by step.

Karafka Pro

Much of the deepest work here - lag compensation, batched Pro iterator resolution, granular backoff correctness, virtual-partition DLQ ordering fixes - lives in Karafka Pro. Pro is what funds this pace of development and how I prioritize the flood of questions and edge cases the community brings. If Karafka is load-bearing infrastructure for you, Karafka Pro pays for itself in features and support - and directly funds the OSS work, including the Share Groups journey ahead.

Summary

Karafka 2.6 is a foundation release. Its visible features - declarative topics, dynamic worker scaling, the offsets API, Pro lag compensation - are useful on their own, but its most important work is the internal reorganization that clears the path for Kafka Queues / Share Groups (KIP-932). Pair that with a 1.0 Web UI that finally carries a version number matching its maturity, plus over fifty bug fixes across both projects, and this release makes the whole ecosystem more solid while setting up what's next.

Thank you to everyone who makes that possible.

References

Want to follow the Share Groups work as it lands? Join us in the Karafka Slack channel.

One Thread to Poll Them All: How a Single Pipe Made WaterDrop 50% Faster

February 19, 2026 / Maciej Mensfeld

This is Part 2 of the "Karafka to Async Journey" series. Part 1 covered WaterDrop's integration with Ruby's async ecosystem and how fibers can yield during Kafka dispatches. This article covers another improvement in this area: migration of the producer polling engine to file descriptor-based polling.

When I released WaterDrop's async/fiber support in September 2025, the results were promising - fibers significantly outperformed multiple producer instances while consuming less memory. But something kept nagging me.

Every WaterDrop producer spawns a dedicated background thread for polling librdkafka's event queue. For one or two producers, nobody cares. But Karafka runs in hundreds of thousands of production processes. Some deployments use transactional producers, where each worker thread needs its own producer instance. Ten worker threads means ten producers and ten background polling threads - each competing for Ruby's GVL, each consuming memory, each doing the same repetitive work. Things will get even more intense once Karafka consumer becomes async-friendly, as it is under development.

The Thread Problem

Every time you create a WaterDrop producer, rdkafka-ruby spins up a background thread (rdkafka.native_kafka#<n>) that calls rd_kafka_poll(timeout) in a loop. Its job is to check whether librdkafka has delivery reports ready and to invoke the appropriate callbacks.

With one producer, you get one extra thread. With 25, you get 25. Each consumes roughly 1MB of stack space. Each competes with your application threads for the GVL. And most of the time, they're doing nothing - sleeping inside poll(timeout), waiting for events that may arrive once every few milliseconds.

I wanted one thread that could monitor all producers simultaneously, reacting only when there's actual work to do.

How librdkafka Polling Works (and Why It's Wasteful)

librdkafka is inherently asynchronous. When you produce a message, it gets buffered internally and dispatched by librdkafka's own I/O threads. When the broker acknowledges delivery, librdkafka places a delivery report on an internal event queue. rd_kafka_poll() drains that queue and invokes your callbacks.

The problem is how rd_kafka_poll(timeout) waits. Calling rd_kafka_poll(250) blocks for up to 250 milliseconds. From Ruby's perspective, this is a blocking C function call. The rdkafka-ruby FFI binding releases the GVL during this call so other threads can run, but the calling thread is stuck until either an event arrives or the timeout expires.

Every rd_kafka_poll(timeout) call must release the GVL before entering C and reacquire it afterward. This cycle happens continuously, even when the queue is empty. With 25 producers, that's 25 threads constantly cycling through GVL release/reacquire. And there's no way to say "watch these 25 queues and wake me when any of them has events."

The File Descriptor Alternative

Luckily for me, librdkafka has a lesser-known API that solves both problems: rd_kafka_queue_io_event_enable().

You can create an OS pipe and hand the write end to librdkafka:

int pipefd[2];
pipe(pipefd);
rd_kafka_queue_io_event_enable(queue, pipefd[1], "1", 1);

Whenever the queue transitions from empty to non-empty, librdkafka writes a single byte to the pipe. The actual events are still on librdkafka's internal queue - the pipe is purely a wake-up signal. This is edge-triggered: it only fires on the empty-to-non-empty transition, not per-event.

The read end of the pipe is a regular file descriptor that works with Ruby's IO.select. The Poller thread spends most of its time in IO.select, which handles GVL release natively. When a pipe signals readiness, we call poll_nb(0) - a non-blocking variant that skips GVL release entirely:

100,000 iterations:
  rd_kafka_poll:    ~19ms (5.1M calls/s) - releases GVL
  rd_kafka_poll_nb: ~12ms (8.1M calls/s) - keeps GVL
  poll_nb is ~1.6x faster

Instead of 25 threads each paying the GVL tax on every iteration, one thread pays it once in IO.select and then drains events across all producers without GVL overhead.

One Thread to Poll Them All

By default, a singleton Poller manages all FD-mode producers in a single thread:

When a producer is created with config.polling.mode = :fd, it registers with the global Poller instead of spawning its own thread. The Poller creates a pipe for each producer and tells librdkafka to signal through it.

The polling loop calls IO.select on all registered pipes. When any pipe becomes readable, the Poller drains it and runs a tight loop that processes events until the queue is empty or a configurable time limit is hit:

def poll_drain_nb(max_time_ms)
  deadline = monotonic_now + max_time_ms
  loop do
    events = rd_kafka_poll_nb(0)
    return true if events.zero?       # fully drained
    return false if monotonic_now >= deadline  # hit time limit
  end
end

When IO.select times out (~1 second by default), the Poller does a periodic poll on all producers regardless of pipe activity - a safety net for edge cases like OAuth token refresh that may not trigger a queue write. Regular events, including statistics.emitted callbacks, do write to the pipe and wake the Poller immediately.

The Numbers

Benchmarked on Ruby 4.0.1 with a local Kafka broker, 1,000 messages per producer, 100-byte payloads:

Producers	Thread Mode	FD Mode	Improvement
1	27,300 msg/s	41,900 msg/s	+54%
2	29,260 msg/s	40,740 msg/s	+39%
5	27,850 msg/s	40,080 msg/s	+44%
10	26,170 msg/s	39,590 msg/s	+51%
25	24,140 msg/s	36,110 msg/s	+50%

39-54% faster across the board. The improvement comes from three things: immediate event notification via the pipe, the 1.6x faster poll_nb that skips GVL overhead, and consolidating all producers into a single polling thread that eliminates GVL contention.

The Trade-offs

Callbacks execute on the Poller thread. In thread mode, each producer's callbacks ran on its own polling thread. In FD mode with the default singleton Poller, all callbacks share the single Poller thread. Don't perform expensive or blocking operations inside message.acknowledged or statistics.emitted. This was never recommended in thread mode either, but FD mode makes it worse - if your callback takes 500ms, it delays polling for all producers on that Poller, not just one.

Don't close a producer from within its own callback when using FD mode. Callbacks execute on the Poller thread, and closing from within would cause synchronization issues. Close producers from your application threads.

How to Use It

producer = WaterDrop::Producer.new do |config|
  config.kafka = { 'bootstrap.servers': 'localhost:9092' }
  config.polling.mode = :fd
end

Pipe creation, Poller registration, lifecycle management - all handled internally.

You can differentiate priorities between producers:

high = WaterDrop::Producer.new do |config|
  config.polling.mode = :fd
  config.polling.fd.max_time = 200  # more polling time
end

low = WaterDrop::Producer.new do |config|
  config.polling.mode = :fd
  config.polling.fd.max_time = 50   # less polling time
end

max_time controls how long the Poller spends draining events for each producer per cycle. Higher values mean more events processed per wake-up but less fair scheduling across producers.

Dedicated Pollers for Callback Isolation

By default, all FD-mode producers share a single global Poller. If a slow callback in one producer risks starving others, you can assign a dedicated Poller via config.polling.poller:

dedicated_poller = WaterDrop::Polling::Poller.new

producer = WaterDrop::Producer.new do |config|
  config.kafka = { 'bootstrap.servers': 'localhost:9092' }
  config.polling.mode = :fd
  config.polling.poller = dedicated_poller
end

Each dedicated Poller runs its own thread (waterdrop.poller#0, waterdrop.poller#1, etc.). You can also share a dedicated Poller between a subset of producers to group them - for example, giving critical producers their own shared Poller while background producers use the global singleton. The dedicated Poller shuts down automatically when its last producer closes.

When config.polling.poller is nil (the default), the global singleton is used. Setting a custom Poller is only valid with config.polling.mode = :fd.

The Rollout Plan

I'm being deliberately cautious. Karafka runs in too many production environments to rush this.

Phase 1 (WaterDrop 2.8, now): FD mode is opt-in. Thread mode stays the default.

Phase 2 (WaterDrop 2.9): FD mode becomes the default. Thread mode remains available with a deprecation warning.

Phase 3 (WaterDrop 2.10): Thread mode is removed. Every producer uses FD-based polling.

A full major version cycle to test before it becomes mandatory.

What's Next: The Consumer Side

The producer was the easier target - simpler event loop, more straightforward queue management. I'm working on similar improvements for Karafka's consumer, where the gains could be even more significant. Consumer polling has additional complexity around max.poll.interval.ms and consumer group membership, but the core idea is the same: replace per-thread blocking polls with file descriptor notifications and efficient multiplexing.

Find WaterDrop on GitHub and check PR #780 for the full implementation details.