Kafka test

Results

Important: These numbers are to be considered 'ballpark' or 'order of magnitude' — and probably specific to this particular combination of hardware. You will almost for certain get different results on other setups.

Each test was adjusted to create 'almost very high CPU load on the broker'. Then they were restarted and ran for 12 hours.
And then the numbers were adjusted to somewhat normalize throughput to the CPU load, so that 'msg/sec' is ~ the max possible with that particular configuration. Lost messages is total over 12 hours.

It's pretty clear that waiting for the MetaData from the broker for each message costs a lot. It should be done only if the producer really, really needs the information (partition, offset, timestamp et al) for auditing or logging purposes. And asking for the MetaData is not in itself protection from message loss.

There also seems to be a minor overhead on acks=1, but it's too small to really make a difference out in the wild. And it doesn't really compare to the value of not having lost messages.

Note - because someone mentioned it: Yes. It is a Future that is returned - get() can be called in another thread. But that doesn't magic up extra CPU-clicks on the broker, and that is where the limitation is.

Image: Tolga Aslantürk @ Pexels