(p)re:invent season updates
List of updates that potentially will change or affect the way I build on AWS.
Time
Last year, AWS announced microsecond precision clocks (here) on their machines, and this year, AWS announced that they have tamed time. This is the biggest announcement this season, in my opinion. As long as it sounds fantastic, to be frank, it is still hard for me to believe. Yet, the biggest cloud provider could achieve that, as they control the whole infrastructure on all layers up to the cards and drivers and invest heavily in the hardware to make it happen.
With great clocks comes great reponsibility.
To prove it, they put many services with active-active multi-region support into preview NoSQL and SQL databases that work exactly based on those clocks. Yes, clocks are essential technology behind DSQL, DynamoDB Multi-Region Strong Consistency tables, and an update that didn’t get so much attention: Amazon MemoryDB also got global active-active support.
What is more, it pairs perfectly with the technology used to build journals (do you remember “the log is the database”?), which are an atomic, distributed, scalable replication system that works as an internal log service that was built internally at AWS for a last 10 years. It is the thing that powers multiple services at AWS like: S3, Lamda, Kinesis, Dynamo and many others.
I can’t wait to see what they will come up with next.
Part of the keynote related to that announcement:
Still, some part of my mind is rebelling against that idea, thinking that in case of a slight failure or higher desynchronization, it might end up as a total disaster and noone will be able to sort out those affected transactions. In the end, “everything fails all the time”, so it should also apply to those clocks, right? Now let’s get back on earth (I mean up until the DSQL announcement)
Amazon DynamoDB reduces prices for on-demand throughput and global tables
What is great is that it was introduced at 14.11, and price reductions were applied starting at the beginning of the month.
0% effort, 100% effect. That’s great! This is exactly what we want to see in the cloud — effortless benefits from the platform.
Regular table:
✅ On-demand -50%
Global table:
✅ On-demand -67%
✅ Provisioned -33%
Announcement: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-dynamo-db-reduces-prices-on-demand-throughput-global-tables/
Amazon DynamoDB global tables previews multi-Region strong consistency [Preview]
“DynamoDB global tables is a fully managed, serverless, multi-Region, and multi-active database used by tens of thousands of customers. With this new capability, you can now build highly available multi-Region applications with a Recovery Point Objective (RPO) of zero, achieving the highest level of resilience“
- 🚨 Must have at least 2 replicas
- 🚨 Limited to 10 000 RCU
- 🚨 Limited to 10 000 WCU
- 🚨 No support for TTL
- 🚨 No support for Local Secondary Indexes
- 🚨 No support for transactions
Announcement: https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-dynamodb-global-tables-previews-multi-region-strong-consistency/
Session about update
Introducing Amazon Aurora DSQL [Preview]
I can’t wait to see DSQL out of the preview. It looks like an SQL DB I would love to use. AWS took a monolithic database architecture and disaggregated it into scalable parts, which has implications, and it will be fascinating to see improvements in the product.
You have to be aware of its limitations and what it was designed to do. DSQL is optimized for transactional workloads in all sorts of scales (yes, zero included), doing small reads & writes with “5–10 statements per transaction”. It is NOT meant to be great for your analytics workloads, though.
✅ Scales to zero
✅ Uses OCC (Optimistic Concurrency Control). Earliest transaction start wins.
✅ Multi-region with Active-Active
✅ Connection pool: no RDS proxy, no data API. You can just connect from your Serverless components
✅ Blocks standard user password-based authentication. Supports token-based authentication using IAM. You can connect using a temporary token as a password with a DB connection. Example here: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/SECTION-tutorials-lambda.html.
🚨 Foreign keys are not supported — this one brought a lot of controversies. Yet, if we take a look at the screen from Verner Worgel’s Keynote, we can realize that this might be a design decision. That was called Database Dark Arts by Canva CTO Brendan Humphreys.
🚨 Views are not supported
🚨 Sequences are not supported
🚨 Can’t create an index if data already exists
🚨 Transaction can modify up to 10,000 rows (where 1 row is for the primary key + 1 extra for each index; if you have extra secondary indices, each one will be counted as an extra row)
🚨 Max transaction time is 5 minutes
🚨 Full list of unsupported features https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html
🚨 Table governance:
- You can’t delete all rows at once
DELETE FROM <table> - You can’t remove all records between the dates when there were more than 10,000 records, for example. This smells like the root cause of many failures.
🚨 Schema evolution: You can’t add a column with a default value. A default value can be added only when the table is created. I believe that is because of the 10,000 per transaction limit.
Announcement: https://aws.amazon.com/blogs/database/introducing-amazon-aurora-dsql/
I recommend you watch these materials. They are great sessions with deep dives, covering the “how” and the “why”. I have already watched them twice, and I think I will go back to them.
In case you prefer to read material covered also on Marc Brooker’s blog:
Announcing Amazon ElastiCache for Valkey
- On-Demand version:
✅No-brainer 20% price drop in comparison to the Redis version - Serverless version:
✅ Fixed minimum $90/month issue — it’s $6/month on Valkey - ✅ Price based on a more granular storage unit than 1GB — [now 100MB]
- ✅ 33% price drop in comparison to Redis
- 🚨 More expensive in all configurations with storage larger than 500MB, although it is perfect for small-volume, high-traffic situations.
I wrote about it more when it was announced here:
Announcement: https://aws.amazon.com/about-aws/whats-new/2024/10/amazon-elasticache-valkey/
AWS Lambda supports Amazon S3 as a failed-event destination for asynchronous and stream event sources
There is a new failure destination for stream-based ESMs. Previously, we could choose only SNS or SQS. Now, we can move failed events to S3.
Why is that great? Previously, you had to write custom logic to persist that event, regardless of the storage, as reported failures contained only record metadata, not the event itself. With S3 as a failure destination, the whole event and metadata are persisted without any code. S3 is a great destination as we can react to new items with events and analyze them there. It is durable storage with lifecycle control, security, and multiple cost tiers.
Announcement: https://aws.amazon.com/about-aws/whats-new/2024/11/aws-lambda-s3-failed-event-destination-stream-event-sources/
AWS Step Functions simplifies developer experience with Variables and JSONata transformations
Simplify Step Functions with JSONata transformations (full list here) and share state variables across the steps. This reduces the amount of custom code needed to build Step Functions.
If you would like to do a deep dive on the topic, I recommend Eric Johnson’s talk from re:invent this year and much shorter videos specifically on JSONata.
Amazon CloudFront announces VPC origins
It makes exposing VPC origins much better as you can expose the internal load balancers, improving security posture. Previously, you had to expose, e.g., a public load balancer that had to be protected with WAF, and you couldn’t limit it to your CloudFront. You could limit access to ANY CloudFront, which was a vector of attack. To solve it, there were tokens passed between distribution and origin that typically were rotated and so on. You get the drill. Now, it is much easier to reference CloudFront security groups and focus on business value.
- ✅ Uses ENI and security group for access
- 🚨 Only 1 security group is created for all distributions so that it might trigger some vigilant SecOps in case of multi-distribution accounts
- 🚨 No CloudFormation support
Announcement: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-cloudfront-vpc-origins/
S3 Tables & S3 Metadata [Preview]
Managed Apache Iceberg tables based on S3. S3 metadata is a product based on AWS-managed S3 tables that automatically stores S3 metadata as an iceberg table.
Iceberg is a game changer — at least for me. It makes auditing and GDPR so much easier than a parquet-based glue table.
- ✅ Streaming already supported by Firehose
- ✅ Way better approach than current Iceberg tables on S3. 3x faster query throughput, 10x higher transactions/second.
- ✅ Automated compaction, vacuum & snapshot management
Session about S3 Tables metadata:
Announcements:
https://aws.amazon.com/blogs/aws/introducing-queryable-object-metadata-for-amazon-s3-buckets-preview/
https://aws.amazon.com/blogs/aws/new-amazon-s3-tables-storage-optimized-for-analytics-workloads/
Amazon DynamoDB introduces warm throughput for tables and indexes
Now, you can get ready for the expected spike of requests.
💵 — 1-time charge billed on a difference between the current warm setup and the requested one for each RCU & WCU. The table in my account that has not been used for a long time starts with 12000 RCU and 4000 WCU.
Announcement: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-dynamodb-warm-throughput-ondemand-provisioned-tables/
AWS Lambda now supports AWS Fault Injection Service (FIS) actions
It's great to see AWS-curated FIS actions to simulate AWS Lambda's failures. Previously, we could simulate such failures with a community-based layer with bugs that were not fixed for a while.
We can simulate probabilistically:
- function slow executions with invocation delays
- function errors
- HTTP integration failures with ALB, API-GW, and VPC Lattice
Based on the screen provided in the blog post by AWS experiments were marked with annotations — it would be great addition to the CloudWatch to have such annotations and experiments could be one of them.
Announcement: https://aws.amazon.com/blogs/mt/introducing-aws-fault-injection-service-actions-to-inject-chaos-in-lambda-functions/
Available actions: https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#aws-lambda-actions-reference
Introducing new Event Source Mapping (ESM) metrics for AWS Lambda
- ✅ FilteredOutEventCount — now you can know why your system stopped getting events
- ✅ FailedInvokeEventCount — finally metric tracking partial failures
- ✅ DroppedEventCount — events dropped due to max age or max retry attempts reached
Announcement: https://aws.amazon.com/blogs/compute/introducing-new-event-source-mapping-esm-metrics-for-aws-lambda/
Introducing resource control policies (RCPs) to centrally restrict access to AWS resources
New organizational policy to centrally control resource permissions. For example, centrally disable putting objects without server-side encryption or enforce https communication
✅ Permission guardrails on resource actions
✅ Applicable on account, OU and root
🚨 Affects only member accounts
🚨 Does not grant any access, defines maximum available permissions
🚨 Do not affect service-linked roles
🚨 Do not affect AWS-managed KMS keys
Supported services: S3, STS, KMS, SQS, Secrets Manager
Announcement: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_rcps.html
Amazon Aurora Serverless v2 supports scaling to zero capacity
Everyone was waiting for it. This always makes me wonder how many people are not aware of the dual write problem and do not use the Outbox pattern.
- ✅ Scales to zero
- ✅ Resumes in 15 seconds if paused for less than 24 hours
- 🚨 Resumes in “longer than 15 seconds” if paused for more than 24 hours. What longer means? Who knows, but I haven’t tested that.
- 🚨 Won’t scale to zero if you are using RDS Proxy
- 🚨Won’t fully scale to zero if replication is turned on, but you can scale down readers.
- 🚨 Won’t scale to zero on zero-ETL integration with Redshift
Noteworthy
Amazon CloudFront announces origin modifications using CloudFront Functions
✅ Cheaper, easier and faster origin redirection
Check out all possible params: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/helper-functions-origin-modification.html
Securely share AWS resources across VPC and account boundaries with PrivateLink, VPC Lattice, EventBridge, and Step Functions
AWS Lambda supports Customer Managed Key (CMK) encryption for Zip function code artifacts
The title is self-explanatory, I guess.
Announcement: https://aws.amazon.com/about-aws/whats-new/2024/11/aws-lambda-cmk-encryption-zip-function-code-artifacts/
Storage Browser for Amazon S3 is now generally available
It is a useful component that can grant access to S3, which might be very handy for moving some end users away from network drives before redesigning legacy systems.
Optimize compute resources on Amazon ECS with Predictive Scaling
Now you can scale not only reactively, but also proactively to always be ready to handle your loads.
Announcement: https://aws.amazon.com/blogs/containers/optimize-compute-resources-on-amazon-ecs-with-predictive-scaling/
