Fn::Sub for multi-line userdata on CloudFormation (JSON)

Introduction

CloudFormation supports both YAML and JSON formats for writing template documents. While YAML is the more popular choice, JSON is ubiquitous in web development and offers less ambiguity due to its treatment of whitespaces. But a huge downside is JSON’s lack of support for multi-line strings.

EC2 userdata scripts is one such string and writing it in JSON is a challenge. The difficulty is more pronounced when the userdata script contains template variables that has to be substituted with actual values.

In this post, we explore a workaround to achieve this use-case.

Our end goal, in YAML

Let’s start with what we’re aiming for, but written in YAML:

Parameters:
    Message:
        Type: "String"

Resources:
    Instance:
        Type: "EC2::Instance"
        Properties:
            # Other properties are omitted for brevity.
            UserData:
                Fn::Base64: !Sub |
                    #!/bin/bash -xe
                    # System Updates
                    yum -y update
                    yum -y upgrade
                    echo ${Message}

What this template does is:

1. Define a Message as a template parameter.
2. Create an EC2 instance logical resource.
3. Set a userdata for the instance that updates the system and prints the given Message, transformed to base64 as required.

Writing userdata scripts in YAML is easy with the | directive. The ${Message} part of the script is then replaced by its actual value in runtime because of the Sub function. The output is finally passed to the Fn::Base64 function for conversion.

Writing the equivalent in JSON

Because multi-line strings are not supported in JSON, we can utilize the Fn::Join:

{
    "Parameters": {
        "Message": {
            "Type": "String"
        }
    },
    "Resources": {
        "Instance": {
            "Type": "EC2::Instance",
            "Properties": {
                "UserData": {
                    "Fn::Base64": {
                        "Fn::Join": [
                            "\n",
                            [
                                "#!/bin/bash -xe",
                                "yum -y update",
                                "yum -y upgrade",
                                "echo ${Message}"
                            ]
                        ]
                    }
                }
            }
        }
    }
}

But this wouldn’t work exactly the same way as there’s no variable substitution happening. Unfortunately, chaining the function calls in the exact same way as the YAML version results in an error:

"Properties": {
    "UserData": {
        "Fn::Base64": {
            "Fn::Sub": {
                "Fn::Join": [
                    "\n",
                    [
                        "#!/bin/bash -xe",
                        "yum -y update",
                        "yum -y upgrade",
                        "echo ${Message}"
                    ]
                ]
            }
        }
    }
}

Template error: One or more Fn::Sub intrinsic functions don’t specify expected arguments. Specify a string as first argument, and an optional second argument to specify a mapping of values to replace in the string

This is an unexpected behavior as Fn::Join returns a string. To circumvent this, we can explicitly call the Fn::Sub function on the lines that need it:

"Properties": {
    "UserData": {
        "Fn::Base64": {
            "Fn::Join": [
                "\n",
                [
                    "#!/bin/bash -xe",
                    "yum -y update",
                    "yum -y upgrade",
                    {
                        "Fn::Sub": "echo ${Message}"
                    }
                ]
            ]
        }
    }
}

The JSON template now functionally behaves the same way as its YAML counterpart.

Conclusion

To write multi-line strings that require variable substitution in CloudFormation JSON templates, use the combination of Fn::Join and per-line Fn::Sub function calls. Or alternatively, decide if this is too much of a hassle and use YAML instead.