This release mostly solves problems related to message deserialization and normalizes some of the naming conventions to ease during the upgrade to the upcoming
Note: This release is the last release with
ruby-kafka under the hood. We’ve already started the process of moving to
Note: If you are using
Sidekiq-Backend plugin, please make sure that you’ve processed all the jobs from your Sidekiq queue before upgrading Karafka gems.
Table of Contents
- 1 Changes (features, incompatibilities, etc)
- 2 Documentation
- 3 Getting started with Karafka
Changes (features, incompatibilities, etc)
consumer#metadata is now
This change is trivial: if you use batch consuming mode and you use the
Consumer#metadata method, replace it with
# Karafka 1.3 class UsersConsumer < ApplicationConsumer def consume puts metadata end end # Karafka 1.4 class UsersConsumer < ApplicationConsumer def consume puts batch_metadata end end
Message metadata available under
Up to version
1.3, all the message metadata would be directly available under the root scope of the
params object using both direct method reference as well as with
While it felt like “The Rails way”, it had several side-effects, amongst which the biggest were the need of having a hash like API, issues with accessing metadata without payload deserialization, and a lack of clear separation between payload and the metadata.
From now on, you can use the
params.metadata object to fetch all the metadata.
Note: we’ve preserved the direct metadata values fetching from the
params object to preserve backwards compatibility.
# 1.3 params['partition'] #=> 0 params.partition #=> 0 # 1.4 params['partition'] #=> NoMethodError (undefined method '') # This will work due to backward compatibility params.partition #=> 0 # This is the recommended way of accessing metadata params.metadata.partition #=> 0 # This will also work as metadata is a struct now params.metadata[:partition] #=> 0 params.metadata['partition'] #=> 0
Message metadata access allowed without message deserialization
When accessing metadata, the payload is not being deserialized until
#payload method is being used.
null message support in the default JSON deserializer
When the Kafka message payload is
nil, deserialization won’t fail. Support for it was added as some of the Karafka users use log compaction with a
nil payload. In case like that,
#payload will return
Karafka::Params::Params no longer inherits from a
Karafka::Params::Params is now just a struct. This change is introduced to normalize the setup, limit the corner cases and simplify the interface only to methods that are really needed.
Our Wiki has been updated accordingly to the
1.4 status. Please notify us if you find any incompatibilities.
Getting started with Karafka
If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to just clone our example repository:
git clone https://github.com/karafka/example-app ./example_app
then, just bundle install all the dependencies:
cd ./example_app bundle install
and follow the instructions from the example app Wiki.