Are you using SAM deployments? There is a chance that you have an issue that you are not aware of.

TL;DR

Changing only env variables in your SAM template might not update your lambda config. So you should consider adding in SAM template:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

Intro

I have been using SAM for serverless application deployments for a couple of years now. I use the AutoPublishAlias feature to publish the latest lambda version and point it to the specified alias. If you are not using aliases, I think you should start doing so, but this article is not about that.

Today I found a serious issue related to environment variable updates. Updating only the value of an environment variable (without any other changes in the template/code) can result in a successful deployment that isn’t picked up by your lambda function. So you might think your production has been updated with a new config when it hasn’t. It could get even worse if you update not all, but only some of your application lambdas.

Test Scenario

If you would like to check it on your own I have prepared simple test case for that scenario. It is simple stack with 2 same lambda functions that returns PUT_PINEAPPLE_ON_PIZZA env variable.

Steps to reproduce:

1. Checkout https://github.com/SodaDev/sam-variables-update on tag before-fix

2. Run make deploy to deploy base version

3. Run test ./test.sh

4. Update variable mapping to true by updating root template or checkout tag fixed

Mappings:
  Properties:
    PizzeriaProps:
      ShouldPutPizzaOnPineapple: true

5. Redeploy with make deploy

6. Rerun test with ./test.sh and observe that response is the same despite changing env variable

7. Add extra AWS::LanguageExtensions Transform operation to see that now your changes are reflected in update-vars template. Now very important thing: ORDER OF TRANSFORM OPERATION MATTERS!

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

8. Rerun test with ./test.sh

Now, as you can see, variable update is reflected and you can also see that ExecutedVersion is different in the lambda executions. What does this mean? That in the NotUpdatingVars lambda function the new version was not created and therefore the alias was not updated either. What is interesting is that the configuration itself is updated. Only the new version is not recognised and the alias is not updated.

Summary

I am not sure if there is a better way to solve this problem. I have tested using AutoPublishAliasAllProperties and a few other ways. The only thing that works is to add additional AWS::LanguageExtensions before AWS::Serverless-2016–10–31. If you know of a better way, please drop me a line, but for now I will keep adding AWS::LanguageExtensions to my SAM templates.