SQS vs Kinesis: cost at scale
Where is the profitability boundary? Which one is best fit for you system? Let’s find out together if/how much you can save!
AWS SQS is a queue, AWS Kinesis is streaming solution. They have way different features, functionalities, characteristics and guarantees. Those services are not interchangeable in all use cases. In some cases it could work though.
I am going to compare both services from profitability point of view only.
TL;DR; SQS in high volume systems might be way more expensive than Kinesis.
Check out how it could work for your system: https://www.aws-geek.com/events
Introduction
There might be a point in time at which your system will start to become popular (hopefully!). In case your system is built with Event Driven Architecture then obviously amount of events sent will raise. Potentially way above expectations. At that point it is worth to review architecture and do a small redesign.
In our system we were consuming events using SQS which was optimal for a long time. When amount of events increased, then price of our system started to raise proportionally to its popularity.
It was built using Serverless architecture, so with pay-per-use model it seems reasonable, right? Yeah, but let’s take a look what we can do to save some money.
Kinesis vs SQS pricing
When you look into details of SQS and Kinesis we can see major differences in pricing and their model.
Without going into details we can see that price is roughly based on:
- SQS: requests and data transfer
- Kinesis OnDemand: data transfer in/from Kinesis
- Kinesis Provisioned: requests to Kinesis and hourly fee for shards.
Thing that is easy to miss here is that for SQS:
- we need to invoke at least 3 operations for each event: send, receive, delete
- each new consumer means duplication of all events. Which can be achieved e.g. with fan-out pattern using SNS.
SQS pricing has tiering for amount of requests and requests themselves are billed differently based on their payloads. Apart from that you may have multiple consumers. In other words calculation might be tricky.
Please take a look at the calculator I have prepared for back of the envelope estimations for this kind of processing that takes that into account. Calculator assumes that every message is processed only once.
Link to calculator: https://www.aws-geek.com/events
What is an effect?
I have heard many times that Kinesis is an expensive service. This has been proven wrong with Kinesis Firehose for audit storage. Again, IT DEPENDS on scale. In our cases we save a lot of money because of the scale of our system. This is why it is so important to monitor the cost of your solutions and adapt to changing reality.
How to check requests used by your SQS?
In case you would like to check/monitor usage of requests (so cost) of your SQS you can use CloudWatch to check how many requests your service is using. You can start with this CloudWatch Metrics using snippet:
{
"metrics": [
[ { "expression": "m1+m2+m3+m4", "label": "Expression1", "id": "e1", "period": 86400 } ],
[ "AWS/SQS", "NumberOfEmptyReceives", "QueueName", "<YOUR_QUEUE_NAME>", { "region": "eu-west-1", "id": "m1", "visible": false } ],
[ ".", "NumberOfMessagesDeleted", ".", ".", { "id": "m2", "visible": false } ],
[ ".", "NumberOfMessagesReceived", ".", ".", { "id": "m3", "visible": false } ],
[ ".", "NumberOfMessagesSent", ".", ".", { "id": "m4", "visible": false } ]
],
"view": "timeSeries",
"stacked": false,
"region": "eu-west-1",
"title": "Total requests",
"period": 86400,
"stat": "Sum"
}Then you can use those values and check what will be the cost of Kinesis.
Summary
Designing a solution that is fast, scalable, secure and reliable is true craftsmanship. Building solutions with an eye to the economic impact they will have on your organisation is yet another level.
It does not take much effort to overspend in the cloud. This is why it is so important to look at the cost of your solutions to choose the right service/technology for the problem you are solving.
This article was meant to just give you another idea where you can save some money. Let me know if it was helpful to you and I hope it was a good read :)
Pricing deep dive
First let’s understand HOW pricing of both services works. Then we can compare pricing.
SQS
Pricing on SQS is based on REQUESTS.
What is a request? API action with payload. Each 64KB of payload chunk is calculated as another action. What are examples of actions?
- sending,
- receiving,
- deleting,
- changing visibility of messages.
To lower down cost of your service you can batch operations. For example:
SendMessagewith 1KB = 1 requestSendMessagewith 100KB = 2 requests (ceil(100 / 64))SendMessagewith 256KB = 4 requestsSendMessageBatchwith 10 messages each with 1 KB payload = 1 requestSendMessageBatchwith 2 messages each with 128 KB payload = 4 requests
Always handle failures when using Batch operations with AWS SDKs. They are not throwing/returning errors. SQS response failures response are described here.
Amount of requests is applied based on tiers:
Apart from that there is data transfer fee:
Kinesis
There are 2 modes available: Provisioned and On-Demand.
On-Demand mode
In this mode AWS scales our shards automatically for us. It’s the best option if you don’t want to manage Kinesis scaling or you can’t predict events in your system. Pricing is based on:
- Stream/hour fee, so base price is:
$0.045*24h*30days=$32.4/month - Data ingestion is rounded to 1KB chunks
- Data retrieved without any rounding
- Extended time of data stored — in case you want to have your events stored in Kinesis for longer than default 24 hours
- Enhanced fan-out data retrieval
Provisioned mode
It requires our ownership and maintenance, so it’s obviously cheaper. However, we have to scale the Kinesis shards ourselves. Could be a good option for predictable throughput, or if we are happy to trade cost savings for operational complexity.
Base unit of Kinesis pricing is shard. Shard provides:
- up to 1MB/s of write or 1000 records/s
- up to 2MB/s of read — regardless of amount of consumers or for each consumer in case of using enhanced fan-out
Another key factor is the data ingestion, which is calculated in payload units. Each unit is rounded to a 25KB chunk. We can ingest up to 25GB per million units.
Kinesis vs SQS price comparison details
Sending event pricing
Kinesis On-Demand pricing is constant value per GB so, it’s the easiest to compare to.
On-Demand vs Provisioned mode
To have comparable value we must have same measure. We can calculate cost per GB by solving equation: 1GB/<documentSize>*<PUT units> ≥ <On-Demand Ingest/GB>10^9[B]/x [B]*$0,0165/1000000reqs≥$0.091we can calculate that on-demand ingestion rate is cheaper only until document size is lower than 181 B
On-Demand vs SQS
To have comparable value we must have same measure. We can calculate cost per GB by solving equation: 1GB/<documentSize>*<Requests per million fee> ≥ <On-Demand Ingest/GB>10^9[B]/x [B]*$0,4/1000000reqs≥$0.091we can calculate that on-demand ingestion rate is cheaper only until document size is lower than 4395 B
Reading event pricing
Kinesis On-Demand: 0.045$ / GB
Kinesis in provisioned mode is free. (AKA hidden in shard/hour fee)
SQS: 1GB/<documentSize>*0.4$/1000000 + long-polling fees requests
Enhanced fanout pricing
Kinesis On-Demand: $0.057 / GB
Kinesis Provisioned: 0.0147$ / GB + (0.017$/shard-hour * 720 * consumers)
SQS: Not supported, but in SQS we always have to duplicate all events for each consumer which could be implemented in combination with SNS. It will increase our cost a lot as it requires to send duplicate of every message for every new consumer.
Confirmation pricing
Kinesis On-Demand: free
Kinesis in provisioned mode: free.
SQS: 1 request
Price for processing of GB of data
Scenario of processing GB of events assuming message with size of 5000[B]:
