Category: Ruby

From pidfd to Shimanami Kaido: My RubyKaigi 2025 Experience

April 27, 2025 /

Introduction

I just returned from RubyKaigi 2025, which ran from April 16th to 18th at the Ehime Prefectural Convention Hall in Matsuyama. If you're unfamiliar with it, RubyKaigi is the biggest Ruby conference, with over 1,500 people showing up this year. It's always a bit crazy (in the best way possible).

The conference had an orange theme. Ehime is famous for its oranges, and the organizers love bringing local flavor to the event.

What I love most about RubyKaigi is how it bridges the gap between the Japanese and Western Ruby worlds. Despite Ruby coming from Japan, these communities often feel separate in day-to-day work. This weird divide affects not just developers but also businesses. RubyKaigi is where these worlds collide, and you get to meet the people whose code you've used for years.

There's something special about grabbing a beer with someone whose gem you depend on or chatting with Japanese Rubyists you'd never usually interact with online. These face-to-face moments make RubyKaigi different from any other Ruby conference.

Pre-Conference (Day -1 & Day 0)

My journey to RubyKaigi was smoother than usual this time. I flew from Cracow, Poland, via Istanbul, which saved me the usual hassle of going to Warsaw first (those extra hours add up!). Instead of the typical route through Tokyo, I flew directly to Osaka - another nice time-saver. On my way to Matsuyama, I stopped in Okayama to check out the castle and the historical garden.

Day 0, for me, was all about the Andpad drinkup welcome party. I got to catch up with Hasumi Hitoshi, my good friend from Japan, along with many other Japanese Rubyists. One of the highlights was meeting the "Agentic Couple" - Justin Bowen and Rhiannon Payne, the creators of Active Agents gem. Little did I know then that I'd spend much more time with them later during some post-conference sightseeing and traveling.

These pre-conference meetups are where some of the best networking happens - everyone's fresh and excited for the days ahead.

The Conference Experience

Day 1 - Talks and Official Party

As the first English speaker in my room (rubykaigi-b), I started the day by discussing bringing pidfd to Ruby. It was exciting to present on this topic, which adds better process control functionality to Ruby - something I'm passionate about, given my work with Karafka.

You can find my presentation by clicking the image below or here:

Throughout the day, I attended as many talks as possible. However, people kept grabbing me for discussions (which I wasn't complaining about at all). One standout was Tagomoris's presentation on "State of Namespace." While I'm not exactly a fan of this feature (and he knows that ;) ), I greatly respect Tagomoris. We had a great follow-up discussion where I outlined my security concerns and the changes needed in Bundler and RubyGems. Ultimately, we both agreed that we must work collectively to ensure such changes bring only good to the community.

The day wrapped up with the official party at Shiroyama Park. The organizers had reserved the biggest park in Matsuyama just for us! The beers were excellent, and the atmosphere was exactly what you'd expect from RubyKaigi - relaxed, friendly, and full of interesting discussions. This is where the real magic happens - where Japanese and Western Rubyists mix over drinks and food, breaking down those invisible barriers that usually keep our communities apart.

Day 2 - ZJIT and More Connections

Day 2 was inspiring with Maxime Chevalier-Boisvert's talk about ZJIT - the successor to YJIT. If you're not familiar with Maxime's work, she's the one who won the Ruby Prize in 2021 for her work on optimizing Ruby's performance. Her new project aims to save and reuse compiled code between executions. I strongly believe that JIT for Ruby can do much more than it does now, bringing us to another level of performance.

The social aspect continued throughout the day with various company-sponsored events. What's unique about RubyKaigi is that these events aren't just corporate marketing exercises - but genuine opportunities for people to connect. The smaller scale of the sponsor presence this year (compared to having just a few big companies) made things more interesting, with more diverse interactions possible.

Day 3 - Ractor-local GC and Hacking Day

Day 3 brought another technical highlight with Koichi Sasada's talk on Ractor-local GC. Ractors are close to my heart because I want to use them in Karafka. While they are still limited, I feel we're finally making good progress. One of the biggest limitations has been cross-ractor GC. Koichi proposed a two-stage GC where part of GC work could run independently in Ractors while some GC runs would still be locking. He sees this as a practical middle ground that's technically easier to implement than fully independent GCs - his philosophy being that we should have something rather than nothing. This approach could make Ractors much more practical for real-world applications.

After the official talks, the day continued with a hacking session. This was amazing - so many Ruby core committers were in one room. People split into groups, and everyone worked on something in their interest. I spent my time analyzing the performance of new fixes - specifically improvements to Ractors. The results looked really great, which is the best news for me.

I need to investigate one interesting thing further: when parsing JSON in separate threads, it's about 10% faster than with the baseline, despite Ruby having GVL. That's an unexpected finding that may impact my future Karafka feature development.

The combination of talks and hacking sessions on Day 3 perfectly captured what makes RubyKaigi special - deep technical discussions followed by hands-on collaboration with some of the smartest people in the Ruby community.

Post-Conference Adventures

Days 4-5 - The Unofficial Adventures Begin

The conference officially ended on Day 3, but the real adventure had just begun. Various companies organized smaller events, and I showed up at one of them. On this "unofficial" day, I attended a drink-up sponsored by codeTakt that was super fun - it's always great to talk more Ruby in casual settings.

The next morning, I started Day 5 with a relaxing session at Dogo Onsen, one of Japan's oldest hot springs. Later, I did some sightseeing around Matsuyama and found a house that looked surprisingly similar to mine - just the Japanese version! I met up with Peter Zhu, and we went to visit some shrines. He collected goshuin (temple stamps) along the way. Later that day, I connected with other RubyKaigi attendees, including Marty Haught from RubyCentral, and we explored Matsuyama Castle together.

Day 6 - The Shimanami Kaido Adventure

One of the most memorable parts of my extended trip was the Shimanami Kaido bicycle tour with Marty and Justin, whom I'd met at the Day 0 Andpad event. The Shimanami Kaido is a famous cycling route that connects several islands via bridges and is located about an hour from Matsuyama.

We covered 60km in one day, which was a lot but totally worth it. Things got interesting when we left the main track to see some temples and head to a port. That's when we discovered there were no immediate direct ferries back to our starting point from where we ended up.

Google Maps saved the day by suggesting we hop to a small island called Oge (大下島). This tiny island has maybe 500 residents, mostly elderly people. We were the only visitors and spent about 45 minutes experiencing life on such a remote Japanese island. The whole detour was one of the craziest things we did. Still, it perfectly showed the spirit of unexpected adventure that makes these post-conference trips so memorable.

The entire cycling route was amazing. The bridges, the sea views, the small island communities - everything was incredible. I highly recommend it to anyone visiting the area after RubyKaigi.

Reflections and Why RubyKaigi Matters

Reflecting on my time in Matsuyama, what I notice most about RubyKaigi isn't just the great talks - those you can watch later on YouTube. The unique atmosphere and connections make this conference stand out from any other tech event I've attended.

RubyKaigi is great at bridging what I see as an unnecessarily isolated divide between the European-American Ruby scene and the Japanese one. This isolation creates real challenges for collaboration and, to some extent, leads to Japanese businesses operating separately from the global Ruby ecosystem. Many Japanese developers use RubyKaigi as a rare opportunity to practice their English and connect with the broader community despite their excellent technical writing skills.

I particularly appreciate how the conference keeps a real, technical-friendly vibe rather than feeling commercial. Unlike some conferences dominated by a few large corporate sponsors, RubyKaigi had many smaller sponsors, creating a more diverse and balanced environment. While I noticed fewer Western companies represented at the sponsor booths (Sentry was there, and maybe two others), this actually added to the conference's unique feel.

The fact that many attendees arrive days early and leave days later makes the event more than just a conference - it becomes something more meaningful. People treat their trip to Japan as part of their vacation and part of their professional development. This extended timeframe allows for deeper connections and more relaxed sightseeing. Matsuyama's calmer atmosphere compared to Tokyo, Osaka, or Sendai adds to this appeal - despite the tourist presence, the scale feels more manageable and peaceful.

From an organizational standpoint, RubyKaigi is in a class of its own. I've never attended another conference so well-organized and thoughtfully executed. It's an amazing event that I highly recommend to anyone wanting technical knowledge and meaningful connections with the global Ruby community. This conference never fails to remind me why I fell in love with Ruby and its community in the first place.

Summary and Final Thoughts

Looking back at my RubyKaigi 2025 experience, I realize how Japan continues to be remarkably generous with opportunities for unexpected connections. Each time I visit, I meet people I would never encounter otherwise - and often, they're not even from the IT world.

In Osaka, at a sake place recommended by fellow conference attendees, I had a memorable two-hour conversation with a retired man in his 70s. Despite his age, he was incredibly sharp and actively attended English school specifically to meet more people from around the world. These encounters show what makes Japan - particularly RubyKaigi - so special.

The conference itself remains the best Ruby event worldwide, not just for its technical content but for its unique ability to bridge communities. Excellent organization, meaningful international connections, and Japan's unique hospitality create an experience far beyond a typical tech conference. Whether cycling the Shimanami Kaido, exploring tiny islands, or simply sharing a beer with developers whose code you use daily, RubyKaigi offers something truly special.

I'm already looking forward to RubyKaigi 2026. If you've never been, start planning now - this conference is worth every mile traveled.

Breaking the Rules: RPC Pattern with Apache Kafka and Karafka

January 26, 2025 /

Introduction

Using Kafka for Remote Procedure Calls (RPC) might raise eyebrows among seasoned developers. At its core, RPC is a programming technique that creates the illusion of running a function on a local machine when it executes on a remote server. When you make an RPC call, your application sends a request to a remote service, waits for it to execute some code, and then receives the results - all while making it feel like a regular function call in your code.

Apache Kafka, however, was designed as an event log, optimizing for throughput over latency. Yet, sometimes unconventional approaches yield surprising results. This article explores implementing RPC patterns with Kafka using the Karafka framework. While this approach might seem controversial - and rightfully so - understanding its implementation, performance characteristics, and limitations may provide valuable insights into Kafka's capabilities and distributed system design.

The idea emerged from discussing synchronous communication in event-driven architectures. What started as a theoretical question - "Could we implement RPC with Kafka?" - evolved into a working proof of concept that achieved millisecond response times in local testing.

In modern distributed systems, the default response to new requirements often involves adding another specialized tool to the technology stack. However, this approach comes with its own costs:

  • Increased operational complexity,
  • Additional maintenance overhead,
  • And more potential points of failure.

Sometimes, stretching the capabilities of existing infrastructure - even in unconventional ways - can provide a pragmatic solution that avoids these downsides.

Disclaimer: This implementation serves as a proof-of-concept and learning resource. While functional, it lacks production-ready features like proper timeout handling, resource cleanup after timeouts, error propagation, retries, message validation, security measures, and proper metrics/monitoring. The implementation also doesn't handle edge cases like Kafka cluster failures. Use this as a starting point to build a more robust solution.

Architecture Overview

Building an RPC pattern on top of Kafka requires careful consideration of both synchronous and asynchronous aspects of communication. At its core, we're creating a synchronous-feeling operation by orchestrating asynchronous message flows underneath. From the client's perspective, making an RPC call should feel synchronous - send a request and wait for a response. However, once a command enters Kafka, all the underlying operations are asynchronous.

Core Components

Such an architecture has to rely on several key components working together:

  • Two Kafka topics form the backbone - a command topic for requests and a result topic for responses.
  • A client-side consumer, running without a consumer group, that actively matches correlation IDs and starts from the latest offset to ensure we only process relevant messages.
  • The commands consumer in our RPC server that processes requests and publishes results
  • A synchronization mechanism using mutexes and condition variables that maintain thread safety and handles concurrent requests.

Implementation Flow

A unique correlation ID is always generated when a client initiates an RPC call. The command is then published to Kafka, where it's processed asynchronously. The client blocks execution using a mutex and condition variable while waiting for the response. Meanwhile, the message flows through several stages:

  • command topic persistence,
  • consumer polling and processing,
  • result publishing,
  • result topic persistence,
  • and finally, the client-side consumer matching of the correlation ID with the response and completion signaling,

Below, you can find a visual representation of the RPC flow over Kafka. The diagram shows the journey of a single request-response cycle:

Design Considerations

This architecture makes several conscious trade-offs. We use single-partition topics to ensure strict ordering, which limits throughput but simplifies correlation and provides exactly-once processing semantics - though the partition count and other things could be adjusted if higher scale becomes necessary. The custom consumer approach avoids consumer group rebalancing delays, while the synchronization mechanism bridges the gap between Kafka's asynchronous nature and our desired synchronous behavior. While this design prioritizes correctness over maximum throughput, it aligns well with typical RPC use cases where reliability and simplicity are key requirements.

Implementation Components

Getting from concept to working code requires several key components to work together. Let's examine the implementation of our RPC pattern with Kafka.

Topic Configuration

First, we need to define our topics. We use a single-partition configuration to maintain message ordering:

topic :commands do
  config(partitions: 1)
  consumer CommandsConsumer
end

topic :commands_results do
  config(partitions: 1)
  active false
end

This configuration defines two essential topics:

  • Command topic that receives and processes RPC requests
  • Results topic marked as inactive since we'll use a custom iterator instead of a standard consumer group consumer

Command Consumer

The consumer handles incoming commands and publishes results back to the results topic:

class CommandsConsumer < ApplicationConsumer
  def consume
    messages.each do |message|
      Karafka.producer.produce_async(
        topic: 'commands_results',
        # We evaluate whatever Ruby code comes in the payload
        # We return stringified result of evaluation
        payload: eval(message.raw_payload).to_s,
        key: message.key
      )

      mark_as_consumed(message)
    end
  end
end

We're using a simple eval to process commands for demonstration purposes. You'd want to implement proper command validation, deserialization, and secure processing logic in production.

Synchronization Mechanism

To bridge Kafka's asynchronous nature with synchronous RPC behavior, we implement a synchronization mechanism using Ruby's mutex and condition variables:

class Accu
  include Singleton

  def initialize
    @running = {}
    @results = {}
  end

  def register(id)
    @running[id] = [Mutex.new, ConditionVariable.new]
  end

  def unlock(id, result)
    return false unless @running.key?(id)

    @results[id] = result
    mutex, cond = @running.delete(id)
    mutex.synchronize { cond.signal }
  end

  def result(id)
    @results.delete(id)
  end
end

This mechanism maintains a registry of pending requests and coordinates the blocking and unblocking of client threads based on correlation IDs. When a response arrives, it signals the corresponding condition variable to unblock the waiting thread.

The Client

Our client implementation brings everything together with two main components:

  1. A response listener that continuously checks for matching results
  2. A blocking command dispatcher that waits for responses
class Client
  class << self
    def run
      iterator = Karafka::Pro::Iterator.new(
        { 'commands_results' => true },
        settings: {
          'bootstrap.servers': '127.0.0.1:9092',
          'enable.partition.eof': false,
          'auto.offset.reset': 'latest'
        },
        yield_nil: true,
        max_wait_time: 100
      )

      iterator.each do |message|
        next unless message

        Accu.instance.unlock(message.key, message.raw_payload)
      rescue StandardError => e
        puts e
        sleep(rand)
        next
      end
   end

   def perform(ruby_remote_code)
      cmd_id = SecureRandom.uuid

      Karafka.producer.produce_sync(
        topic: 'commands',
        payload: ruby_remote_code,
        key: cmd_id
      )

      mutex, cond = Accu.instance.register(cmd_id)
      mutex.synchronize { cond.wait(mutex) }

      Accu.instance.result(cmd_id)
    end
  end
end

The client uses Karafka's Iterator to consume responses without joining a consumer group, which avoids rebalancing delays and ensures we only process new messages. The perform method handles the synchronous aspects:

  • Generates a unique correlation ID
  • Registers the request with our synchronization mechanism
  • Sends the command
  • Blocks until the response arrives

Using the Implementation

To use this RPC implementation, first start the response listener in a background thread:

# Do this only once per process
Thread.new { Client.run }

Then, you can make synchronous RPC calls from your application:

Client.perform('1 + 1')
#=> Remote result: 2

Each call blocks until the response arrives, making it feel like a regular synchronous method call despite the underlying asynchronous message flow.

Despite its simplicity, this implementation achieves impressive performance in local testing - roundtrip times as low as 3ms. However, remember this assumes ideal conditions and minimal command processing time. Real-world usage would need additional error handling, timeouts, and more robust command processing logic.

Performance Considerations

The performance characteristics of this RPC implementation are surprisingly good, but they come with important caveats and considerations that need to be understood for proper usage.

Local Testing Results

In our local testing environment, the implementation showed impressive numbers.

A single roundtrip can be completed in as little as 3ms. Even when executing 100 sequential commands:

require 'benchmark'

Benchmark.measure do
  100.times { Client.perform('1 + 1') }
end
#=> 0.035734   0.011570   0.047304 (  0.316631)

However, it's crucial to understand that these numbers represent ideal conditions:

  • Local Kafka cluster
  • Minimal command processing time
  • No network latency
  • No concurrent load

Summary

While Kafka wasn't designed for RPC patterns, this implementation demonstrates that with careful consideration and proper use of Karafka's features, we can build reliable request-response patterns on top of it. The approach shines particularly in environments where Kafka is already a central infrastructure, allowing messaging architecture to be extended without introducing additional technologies.

However, this isn't a silver bullet solution. Success with this pattern requires careful attention to timeouts, error handling, and monitoring. It works best when Kafka is already part of your stack, and your use case can tolerate slightly higher latencies than traditional RPC solutions.

This fascinating pattern challenges our preconceptions about messaging systems and RPC. It demonstrates that understanding your tools deeply often reveals capabilities beyond their primary use cases. While unsuitable for every situation, it provides a pragmatic alternative when adding new infrastructure components isn't desirable.

Copyright © 2025 Closer to Code

Theme by Anders Noren — Up ↑