Pushing the CloudFormation Bleeding Edge: Native Modular Templates

When the YAML format for CloudFormation was launched in September 2016, many of the users knew it was only a matter of time until the commonly used pattern of including multiple YAML files into a single file made its way into CloudFormation. On March 28, 2017, AWS did exactly that by launching the AWS::Include Transform, albeit with surprising lack of fanfare.

While YAML was not a prerequisite to having this feature, it made it infinitely easier leverage as an end-user. There are very important things which I have discovered as I integrated AWS::Include into my daily work; some of these are documented fully, others are documented partially, and others not at all.

Terms of Use:

Partials - The snippets of CloudFormation stored in S3

Master - The template executed by the end-user

Includes - The AWS::Includes Transform

Key Points

Your Partials may be in either JSON or YAML

Partials must use only the long form of a Function Call (e.g Fn::Sub not !Sub)

Change sets are required for use of Includes

Partials must be accessible by the end-user’s STS assumption for CloudFormation Public ACL Read is not required if you have a good bucket policy

Partials are included into the Master before evaluation of functions Prevents using Fn::Sub in a Location directive for dev/prod s3 path of the Partials

evaluation of functions Errors which occur in Partials create unusual errors on evaluation

Understanding scope is very important

Nested Includes calls is not supported (i.e. your partials template cannot have another partial in it)

Now I will dive into some of these key points in detail, use cases for where to AWS::Include, and lessons learned from living on the bleeding edge.

In a future post, I will detail use cases where Includes is ideal for your business such as creating predictable IAM Roles or a multi-engine RDS template

Key Points in Details

Execution Model

The execution model when using Includes is through CloudFormation Change Sets, which is a great way to enforce a known checkpoint but brings in difficulties for people who don’t use CloudFormation daily. When you use a Includes and you want to make a new stack, you are left with two options:

Create the stack within the AWS Console - the console automatically creates a blank stack, change set, and prompts for CAPABILITY_NAMED_IAM Create a “blank” stack (e.g. just a wait handle) and then create a change set against that stack with CAPABILITY_NAMED_IAM in the create-change-set call

Mixing JSON and YAML

This adventure to using Includes was my first significant effort into using YAML for all my CloudFormation, up until this point the need for YAML (specifically inline comments) was not worth the time it would take to rewrite what we had been using to date.

One of the use-cases I leveraged Includes for is deploying IAM Policies and Roles for Federated Login as it requires predictable Role naming. I have found in practice it is easier to write the policies in JSON but as I was doing YAML now, I decided to keep it in pure YAML for readability.

AWS has, thankfully, provided the ability to continue keeping your Partials in either language regardless of the Master’s language. You may choose to keep your JSON templates for things like IAM Policies and use YAML for the Master.

What I did leverage from time to time was cfn-flip to ensure my YAML syntax was inline with the JSON evaluation. As the templates are included they are converted to JSON, so doing this is a reasonable checkpoint for yourself.

Evaluation Logic and Order

As I said above you cannot have any other Function evaluated before the includes happens. This means that you cannot do something like this:

Parameters: PartialsEnv: Type: String Default: prod AllowedValues: - prod - dev ... Resources: MyTestResource: Fn::Transform: Name: AWS::Include Parameters: Location: Fn::Sub: s3://my-partials-bucket/${PartialsEnv}/resources/test_resource.yaml

This will throw an error that you must provide a valid S3 URI/Object. I have raised this with support and an RFE has been created to allow this or something like it be accepted.

While the order of operations is not specific to Includes, I was bitten by the unwritten order more than once. For your reference, I have compiled an incomplete list of the evaluation precedence steps here:

Mappings References Lookups Conditional Statements Substitutions

I will try to obtain the information on a complete list of these in the future and make a separate post on that.

The reason I included these is that you are more likely to run into trying to do Mappings inside Conditionals using Reference Lookups and that will fail with unpredictable results.

Errors & Debugging

I have compiled a list of the most common issues I have run into within Includes and YAML in general

Common Errors Causes Circular Dependancies Incorrect/Invalid Reference in the Partial Invalid Policy Syntax for IAM on execution A * IAM Policy Resource was not "*" Invalid MajorEngineVersion ##.# for SqlServer Option Group SqlServer Option Groups are ##.## and during the YAML to JSON conversion, it drops the superfluous 4th digit, quoting such as "13.00" fixes this Must Provide valid S3 URI or S3 Object 1. You are referencing a private S3 Object with no Bucket Policy 2. You are trying to do a Function within the S3 Location call 3. You are trying to reference a S3 Object which has no valid CloudFormation items

Scope and Use In Practice

When using Includes, its very important to pay attention to how Scope is in use. You cannot have two Includes calls in a single Scope level. I have detailed interesting use cases around scope below and its important all of these items are mutually exclusive within a single template (#1) or within sections (#2/#3).

1 - Use the transform section of the template

In this example I show how you can replace an entire template with Includes except Parameters, which can not be part of Includes.

Master Template

--- AWSTemplateFormatVersion: 2010-09-09 Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/my_stack.yaml # Parameters cannot be in an Includes Parameters: MyParam: Type: String

Partials Template

Metadata: ... # Your Metadata Conditionals: ... # Your Conditionals Mappings: ... # Your Mappings Resources: ... # Your Resources Outputs: ... # Your Outputs

2 - A section level

In this example I show how you can take all of the Resources, Outputs, etc of a template and put them into a Partial

Master Template

Mappings: Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/mappings/my_mappings.yaml # Parameters cannot be in an Includes Parameters: MyParam: Type: String ... Resources: Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/resources/my_resources.yaml Outputs: Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/outputs/my_outputs.yaml

Partials Templates

# my_mappings.yaml AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Global Account Information Parameters: - MyParam ParameterLabels: # These values must be quoted to add white space MyParam: default: 'My Parameter: '

# my_resources.yaml MyLogicalKMSResourceName: Type: AWS::KMS::Key Properties: Description: | My KMS Example Resource Enabled: true ... MyLogicalWaitResourceName: Type: AWS::CloudFormation::WaitConditionHandle

# my_outputs.yaml MyLogicalKMSResourceOutput: Description: | KMS ARN Example Value: Ref: MyLogicalKMSResourceName Export: Name: Fn::Sub: ${AWS::StackName}-MyLogicalKMSResourceOutput

3 - Multiple Resources

In this example I show you can use excludes to abstract the body of each resource and output into their own Partial

... Parameters: MyParam: Type: String ... Resources: MyLogicalKMSResourceName: Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/resources/kms_key.yaml MyLogicalWaitResourceName: Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/resources/wait_handle.yaml ... Outputs: MyLogicalKMSResourceOutput: Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/outputs/kms_key.yaml ...

Partials Templates

#resources/kms_key.yaml Type: AWS::KMS::Key Properties: Description: | My KMS Example Resource Enabled: true ...

#resources/kms_key.yaml Type: AWS::CloudFormation::WaitConditionHandle

#outputs/kms_key.yaml Description: | KMS ARN Example Value: Ref: MyLogicalKMSResourceName Export: Name: Fn::Sub: ${AWS::StackName}-MyLogicalKMSResourceOutput

4 - Within a resource

This example shows how to use Includes to provide some modularity to the end-user while maintaining some attributes which are common in the Partial. When you get to this method, as the warning below notates scope is very tricky and you should take care. In the example I create an RDSDBParameterGroup which allows the Master Template to specify what the Parameters are in use for this RDS. Additionally, I show a second method of advanced scoping which allows you to have an Includes at the end of a section which provides any “generic” items, as a result of scoping, any use of this must be at the end of the Section or it will be overridden. I also demonstrate that regardless of how a resource is declared (e.g. RDSDBParameterGroup is “within a resource”) the Logical ID persists in the template after compilation (e.g the Output for RDSDBParameterGroup is in the Generic Outputs Includes)

Note: This method is considered advanced and requires significant testing

Master Template

Mappings: ... # Parameters cannot be in an Includes Parameters: MyParam: Type: String ... Resources: RDSDBParameterGroup: Type: AWS::RDS::DBParameterGroup Properties: # Common items for all Parameter Groups Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/resources/rds_parameter_group.yaml # Custom Parameters per RDS which are set by the "stack owner" Parameters: sql_mode: IGNORE_SPACE timezone: UTC # Must be the last item in the section # Includes a series of generic resources Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/resources/general_resouces.yaml Outputs: MyLogicalKMSResourceOutput: Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/outputs/kms_key.yaml # Must be the last item in the section # Includes a series of generic outputs Fn::Transform: Name: AWS::Include Parameters: Location: s3://my-partials-bucket/outputs/general_outputs.yaml

Partials Templates

# resources/rds_parameter_group.yaml Description: RDS DB parameter group Family: rdsengine-12.3 Tags: - Key: Name Value: rds-engine-12.3-parameter-group

# resources/general_resouces.yaml MyLogicalKMSResourceName: Type: AWS::KMS::Key Properties: Description: | My KMS Example Resource Enabled: true ... MyLogicalWaitResourceName: Type: AWS::CloudFormation::WaitConditionHandle

# outputs/kms_key.yaml Description: | KMS ARN Example Value: Ref: MyLogicalKMSResourceName Export: Name: Fn::Sub: ${AWS::StackName}-MyLogicalKMSResourceOutput

# outputs/general_outputs.yaml RDSDBParameterGroupOutput: Description: | A RDS Parameter Group Example Value: Ref: RDSDBParameterGroup Export: Name: Fn::Sub: ${AWS::StackName}-RDSDBParameterGroup

Lessons Learned

Numbers on convert are not maintained with precision (13.00 becomes 13.0)

When debugging, make a “fat” template if you run into issues. Many “good errors” are hidden by Change Sets and Includes and therefore if you run into weird issues, make a template with everything in it first and then break it out once its working.

Make a test blank stack (see above) when trying to develop as a rollback rolls back normally preventing continual stack create/delete cycles

Validate all your YAML first and not through a CloudFormation YAML linter, the stricter the YAML linter the better

When in doubt, use cfn-flip and see if its still works

Develop IAM policies in IAM first, then use a JSON to YAML converter to embed into your templates

New features have new bugs, you may call support and want to hit your head on your desk when its something “simple”, but those occurrences are outweighed by the times its a true bug

References

Announcement: https://aws.amazon.com/about-aws/whats-new/2017/03/aws-cloudformation-supports-authoring-templates-with-code-references-and-amazon-vpc-peering/

AWS Documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/create-reusable-transform-function-snippets-and-add-to-your-template-with-aws-include-transform.html