Amazon SQS vs Kinesis: Which One Should You Use?

Introduction

Amazon Simple Queue Service (SQS) and Amazon Kinesis are both highly scalable, fully managed messaging and streaming data services provided by Amazon Web Services (AWS). Both services can be used to integrate and decouple distributed applications, enable communication between microservices, and process and analyze large amounts of data in real time.

However, there are some key differences between SQS and Kinesis that make them suitable for different use cases. In this article, we will compare the two services and provide guidelines on when to use each one. The short version: SQS hands each message to one worker and forgets it, Kinesis keeps an ordered log that many readers can replay. Almost every other difference follows from that.

Amazon SQS

Amazon SQS is a managed message queue service that enables applications to send and receive messages between different components without losing messages or requiring each component to be always available. SQS offers two types of message queues: standard and FIFO (first-in-first-out).

Standard queues provide best-effort ordering, meaning that messages are generally delivered in the order they are sent, but occasionally out-of-order delivery may occur. They also deliver at least once, so the same message can arrive twice. In exchange, throughput is nearly unlimited: you do not provision capacity and there is no per-second quota to plan around.

FIFO queues are designed to guarantee that messages are delivered in the exact order they are sent and without duplicates. Ordering is per message group, set with the MessageGroupId attribute. A FIFO queue handles 300 API calls per second without batching and 3,000 messages per second with batching; high throughput mode raises that into the tens of thousands per second by spreading message groups across partitions.

Both types accept messages up to 256 KB (larger payloads go through the extended client library, which stores the body in S3), keep a message for up to 14 days, and delete it once a consumer acknowledges it.

SQS is a good choice for applications that need a simple and reliable messaging solution with low latency, such as sending notifications, processing logs, or triggering background jobs. SQS is also widely used for building event-driven architectures and distributed systems, where multiple components need to communicate and coordinate their actions.

Amazon Kinesis

Amazon Kinesis is a fully managed, scalable, and real-time data streaming service that allows applications to process and analyze large amounts of data from various sources, such as website clickstreams, application logs, IoT devices, and social media feeds. Kinesis offers three main types of data streams: Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics.

Kinesis Data Streams is a stream of data records, where each record can be up to 1 MB in size. Data streams can be sharded, meaning that they can be divided into multiple shards, each of which can support up to 1000 records per second or up to 1 MB per second data input and up to 2 MB per second data output. Records stay in the stream for 24 hours by default and up to 365 days if you pay for extended retention, and reading a record does not remove it. On-demand mode scales shards for you; provisioned mode makes you pick the count.

Kinesis Data Firehose is a fully managed service that can automatically load streaming data into other AWS services, such as S3, Redshift, and OpenSearch, for storage and analysis. Firehose can also transform and compress data on the fly, before delivering it to the destination.

Kinesis Data Analytics (now Amazon Managed Service for Apache Flink) is a service that allows you to run SQL or Flink jobs on streaming data and get real-time insights and analytics.

Kinesis is a good choice for applications that need to process and analyze large amounts of streaming data in real time, such as clickstream analysis, fraud detection, IoT analytics, and social media analysis. Kinesis is also widely used for building real-time data pipelines and streaming applications, where data needs to be ingested, transformed, and delivered to multiple destinations.

Comparison

Now that we have reviewed the main features and capabilities of SQS and Kinesis, let’s compare the two services in more detail.

• Delivery model. SQS delivers each message to one consumer, and that consumer deletes it. Kinesis writes records to a log that any number of consumers read independently.
• Ordering. SQS standard: best effort. SQS FIFO: strict, per message group. Kinesis: strict, per shard, with a sequence number on every record.
• Duplicates. SQS standard: at least once. SQS FIFO: exactly once within a five-minute deduplication window. Kinesis: at least once; a retried PutRecord can write the same data twice.
• Throughput. SQS standard: effectively unlimited. SQS FIFO: 3,000 messages per second with batching, more in high throughput mode. Kinesis: 1 MB or 1,000 records per second per shard on writes, scaled by adding shards.
• Retention and replay. SQS: up to 14 days, and a consumed message is gone. Kinesis: 24 hours to 365 days, and any consumer can rewind to a sequence number or a timestamp.
• Fan-out. SQS: none by itself; put SNS in front of several queues. Kinesis: built in, with enhanced fan-out giving each consumer its own 2 MB per second per shard.
• Consumer scaling. SQS: add more workers, they all poll the same queue. Kinesis: one active reader per shard per consumer application, so parallelism equals shard count.
• Message size. SQS: 256 KB. Kinesis: 1 MB.
• Pricing. SQS: per million requests, roughly $0.40 for standard and $0.50 for FIFO, first million free each month. Kinesis: per shard-hour (about $0.015) plus per million 25 KB payload units in provisioned mode, or per GB in on-demand mode. An idle SQS queue costs nothing; an idle provisioned Kinesis shard still costs about $11 a month.

SQS and Kinesis are similar in some aspects, such as being fully managed, scalable, and fault-tolerant services. However, they also have some significant differences that make them suitable for different use cases.

SQS is a good choice for applications that need a simple and reliable messaging service with low latency and at-least-once delivery. SQS is also widely used for building event-driven architectures and distributed systems, where multiple components need to communicate and coordinate their actions.

On the other hand, Kinesis is a good choice for applications that need to process and analyze large amounts of streaming data in real time. Kinesis is also widely used for building real-time data pipelines and streaming applications, where data needs to be ingested, transformed, and delivered to multiple destinations.

Ordering guarantees: SQS FIFO vs Kinesis

Ordering is the question I get asked most, because both services promise it and they mean different things.

In a FIFO queue, order is kept inside a message group. You choose the group with MessageGroupId: an order ID, a user ID, a device ID. Messages in the same group are delivered one at a time, in order, and the next one is not released until the previous one is deleted or its visibility timeout expires. Messages in different groups are delivered in parallel. So the number of distinct group IDs is your parallelism, and one slow message blocks only its own group.

In Kinesis, order is kept inside a shard. You choose the shard indirectly with a partition key, which is hashed to pick one. Records with the same key land on the same shard and get increasing sequence numbers. A consumer reads a shard sequentially and keeps a checkpoint of the last sequence number it processed. If the consumer crashes, it resumes from the checkpoint and reprocesses anything after it, which is why Kinesis consumers must be idempotent.

The practical difference is what happens with a bad message. In SQS FIFO, a message that keeps failing stays at the head of its group until it goes to a dead-letter queue, and only that group waits. In Kinesis, a record that keeps failing stalls the whole shard for every consumer, because the checkpoint cannot move past it. With a Lambda consumer, set BisectBatchOnFunctionError, a MaximumRetryAttempts limit and an on-failure destination, otherwise one poison record blocks the shard until it ages out of retention.

The second difference is duplicates. FIFO deduplicates for you: send the same MessageDeduplicationId twice within five minutes and the second one is dropped. Kinesis does not; if your producer retries a PutRecord after a timeout, the record may be written twice, and your consumer has to detect that.

Which one for a chat application?

A chat application is a good test case because it needs both ordering and fan-out, and neither service gives you both for free.

Messages in one conversation must arrive in order, so the conversation ID is the natural key: MessageGroupId in FIFO, partition key in Kinesis. Every participant’s connection needs a copy of the message, which is fan-out. And nobody replays a chat log from the stream; the messages go into a database and clients read history from there.

For delivering messages to connected clients, SQS FIFO with one queue per delivery service, fed by SNS for fan-out, is the simpler fit. It has no shard planning, it costs nothing while quiet, and ordering per conversation is exactly the FIFO guarantee. Kinesis makes sense once you also want to run analytics on the message stream (moderation models, trend detection, search indexing) from the same feed, because each of those is a separate consumer replaying the same log. Plenty of teams use both: SQS FIFO on the delivery path and a Kinesis stream fed by the same producer for everything that reads the history.

Can you put SQS in front of Kinesis?

Yes, and it is a common pattern in two directions.

SQS in front of Kinesis is a buffer for bursty producers. Kinesis rejects writes that exceed a shard’s 1 MB per second with ProvisionedThroughputExceededException, and a producer that cannot slow down loses data. Writing to SQS first, then having a worker drain the queue into the stream at a steady rate, absorbs the spike. On-demand streams reduce the need for this but do not remove it, because on-demand scaling reacts to load rather than anticipating it.

Kinesis in front of SQS is the fan-out and replay layer. The stream keeps the ordered history; a consumer reads it and pushes individual work items into one or more SQS queues, where a pool of workers processes them with retries and a dead-letter queue. This is how you get Kinesis retention with SQS-style worker scaling.

Choosing

• Background jobs, task queues, decoupling two services: SQS standard.
• The same, but order matters per entity or duplicates are unacceptable: SQS FIFO.
• Several independent systems need the same events, or you need to replay history: Kinesis Data Streams.
• Events just need to land in S3 or a warehouse with no custom consumer: Kinesis Data Firehose.
• Analytics over a window of recent events: Managed Service for Apache Flink on a stream.

In summary, SQS is a message queue service that is used for communication and coordination between components, while Kinesis is a data stream service that is used for real-time processing and analysis of streaming data. Depending on your specific requirements and use case, you can choose the appropriate service to meet your needs.

If SQS turns out to be the right fit, the next decision is queue type: Amazon SQS: Standard vs FIFO Queues Compared goes through the ordering and duplication trade-off. To wire it into a service, see Spring Boot and Amazon SQS.