A Short Guide to Redis Streams

Redis Streams are one of those features that quietly changed what you can do with Redis. If you've used Redis for caching or pub/sub before, streams unlock a whole new category of workloads: event sourcing, message queues, activity feeds, and time-series style data — all backed by an in-memory data structure that's surprisingly cheap to run.

What is a Redis Stream?

A Redis stream is a data structure that acts like an append-only log but also implements several operations to overcome some of the limits of a typical append-only log. These include random access in O(1) time and complex consumption strategies, such as consumer groups.

The stream type was introduced with Redis 5.0, which models a log data structure in a more abstract way, however the essence of the log is still intact: like a log file, often implemented as a file open in append only mode, Redis streams are primarily an append only data structure.

Unlike a plain string entry, a stream entry is not just a string, but is instead composed of one or multiple field-value pairs. This way, each entry of a stream is already structured, like an append only file written in CSV format where multiple separated fields are present in each line.

Streams vs. Pub/Sub

People often reach for Pub/Sub first, but streams solve a different problem. The main difference lies in the persistence of messages and how they are consumed. While Pub/Sub relies on the broadcasting of transient messages (i.e. if you don't listen, you miss a message), Redis Stream use a persistent, append-only data type that retains messages until the stream is trimmed. Another difference in consumption is that Pub/Sub registers a server-side subscription. Redis pushes arriving messages to the client while Redis Streams require active polling.

In short: if you can't afford to drop messages, you want streams.

The Core Commands

You really only need a handful of commands to be productive:

  • XADD — Appends a new message to a stream. Creates the key if it doesn't exist. O(1) when adding a new entry, O(N) when trimming where N being the number of entries evicted.
  • XRANGE / XREVRANGE — read entries by ID range, forward or reverse.
  • XREAD — read new entries, optionally blocking until something arrives.
  • XREADGROUP — Returns new or historical messages from a stream for a consumer in a group. Blocks until a message is available otherwise.
  • XACK — Returns the number of messages that were successfully acknowledged by the consumer group member of a stream.
  • XAUTOCLAIM — Changes, or acquires, ownership of messages in a consumer group, as if the messages were delivered to as consumer group member.

Consumer Groups

Consumer groups are what make Redis Streams viable as a real message queue. Multiple workers join a group, Redis hands each one a different slice of the stream, and tracks which messages each consumer has acknowledged.

Even better, a Stream, like any other Redis data structure, is asynchronously replicated to slaves and persisted into AOF and RDB files. However what may not be so obvious is that also consumer groups full state is propagated to AOF, RDB and slaves, so if a message is pending in the master, also the slave will have the same information. Similarly, after a restart, the AOF will restore the consumer groups state.

One thing worth knowing for production: AOF must be used with a strong fsync policy if persistence of messages is important in your application. By default the asynchronous replication will not guarantee that XADD commands or consumer groups state changes are replicated: after a failover something can be missing depending on the ability of slaves to receive the data from the master.

Why Streams Are So Cheap

The internal representation is the real magic. Antirez (Redis's creator) ran the numbers comparing streams to the old Sorted Set + Hash pattern: Sorted Set + Hash memory usage = 220 MB (242 RSS), Stream memory usage = 16.8 MB (18.11 RSS). This is more than an order of magnitude difference (13 times difference exactly), and it means that use cases that yesterday were too costly for in-memory now are perfectly viable. The magic is all in the representation of Redis Streams: the macro nodes can contain several elements that are encoded in a data structure called listpack in a very compact way.

When to Reach for Streams

A few patterns where streams really shine:

  • Event sourcing — append every state change, replay to rebuild state.
  • Work queues — XADD producers, XREADGROUP workers, XACK on success, XAUTOCLAIM to recover from dead consumers.
  • Activity feeds and time series — even the most basic use case of time series is, obviously, a huge one here, because before Streams Redis was a bit hopeless in regard to such use case. The memory characteristics and flexibility of streams, plus the ability to have capped streams (see the XADD options), is a very important tool in the hands of the developer.

Wrapping Up

Redis Streams are deceptively simple: an append-only log with IDs, field-value entries, and consumer groups bolted on. But that combination covers a huge range of use cases that previously required either a heavier broker like Kafka or awkward Redis patterns built on sorted sets. If you're already running Redis, streams are almost certainly worth a look before you add another piece of infrastructure to your stack.