Preprocessor for CloudFormation templates with support for loops and flexible include statements
npm install @znemz/cfn-includecfn-include is a preprocessor for CloudFormation templates which extends CloudFormation's intrinsic functions.
For example, Fn::Include provides a convenient way to include files, which can be local, a URL or on an S3 bucket (with proper IAM authentication if necessary). It supports both JSON and YAML as input and output format. CloudFormation's tag syntax for YAML (e.g. !GetAtt) is supported as well.
cfn-include tries to be minimally invasive, meaning that the template will still look and feel like an ordinary CloudFormation template. This is what sets cfn-include apart from other CloudFormation preprocessors such as CFNDSL, StackFormation and AWSBoxen. There is no need to use a scripting language or adjust to new syntax. Check them out though, they might be a better fit for you.
Functions
- cfn-include
- Installation
- Synopsis
- CLI
- Example
- Fn::Include
- Examples
- Include a file from a URL
- Include a file in the same folder
- Include an AWS API response
- Include Globs
- Include Inject State
- Fn::Map
- Fn::Flatten
- Fn::FlattenDeep
- Fn::GetEnv
- Fn::Length
- Fn::Merge
- Fn::DeepMerge
- Fn::Sequence
- Fn::Stringify
- Fn::Uniq
- Fn::Compact
- Fn::Concat
- Fn::Sort
- Fn::SortedUniq
- Fn::SortBy
- Fn::SortObject
- Fn::ObjectKeys
- Fn::ObjectValues
- Fn::Filenames
- Fn::StringSplit
- Fn::Without
- Fn::Omit
- Fn::OmitEmpty
- Fn::Eval
- Fn::IfEval
- Fn::JoinNow
- Fn::SubNow
- Fn::RefNow
- Fn::ApplyTags
- Fn::Outputs
- More Examples
- Proxy Support
- Compatibility
- Web Service
Tag-based syntax is available in YAML templates. For example,Fn::Include becomes !Include.
You can either install cfn-include or use a web service to compile templates.
```
npm install --global @znemz/cfn-include
The web service can be called with your favorite CLI tool such as curl.
``
curl https://api.netcubed.de/latest/template -XPOST -d @template.json
cfn-include
- path
location of template. Either path to a local file, URL or file on an S3 bucket (e.g. s3://bucket-name/example.template)
Options:
* -m, --minimize minimize JSON output [false]--metadata
* add build metadata to output [false]-t, --validate
* validate compiled template [false]-y, --yaml
* output yaml instead of json [false]--bucket
* bucket name required for templates larger than 50k--prefix
* prefix for templates uploaded to the bucket ['cfn-include']--version
* print version and exit--context
* template full path. only utilized for stdin when the template is piped to this scriptcat examples/base.template | ./bin/cli.js --context examples/base.template
example: --enable
* different options / toggles: ['env','eval'] [string] [choices: 'env','eval','env.eval' etc...]env
* pre-process env vars and inject into templates as they are processed looks for $KEY or ${KEY} matches-i, --inject
* JSON string payload to use for template injection. (Takes precedence over process.env (if enabled) injection and will be merged on top of process.env)--doLog
* console log out include options in recurse step. Shows caller parameter to aid debugging nested function calls.--ref-now-ignore-missing
* do not fail if Fn::RefNow reference cannot be resolved; instead return in standard CloudFormation Ref syntax--ref-now-ignores
* comma-separated list of reference names to ignore if not found (e.g., OptionalRef1,OptionalRef2)cfn-include also accepts a template passed from stdin
``
cat mytemplate.yml | cfn-include
`yaml`
Mappings:
Region2AMI: !Include https://api.netcubed.de/latest/ami/lookup?platform=amzn2
Resources:
Instance:
Type: AWS::EC2::Instance
Properties:
ImageId: !FindInMap [Region2AMI, !Ref AWS::Region, AMI]
UserData:
Fn::Base64:
Fn::Sub: !Include { type: string, location: userdata.sh }
This is what the userdata.sh looks like:
`bash`
#!/bin/bash
/opt/aws/bin/cfn-init -s ${AWS::StackId} -r MyInstance --region ${AWS::Region}
`bash`
cfn-include synopsis.json > output.templateyou can also compile remote files
cfn-include https://raw.githubusercontent.com/nmccready/cfn-include/master/examples/synopsis.json > output.template
The output will be something like this:
`javascript`
{
"AWSTemplateFormatVersion": "2010-09-09",
"Mappings": {
"Region2AMI": {
"Metadata": {
"Name": "amzn-ami-hvm-2016.09.0.20161028-x86_64-gp2",
"Owner": "amazon",
"CreationDate": "2016-10-29T00:49:47.000Z"
},
"us-east-2": {
"AMI": "ami-58277d3d"
},
// ...
} },
"Resources": {
"Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": {
"FindInMap": [ "Region2AMI", { "Ref": "AWS::Region" }, "AMI" ]
},
"UserData": {
"Fn::Base64": {
"Fn::Sub": {
"Fn::Join": ["", [
"#!/bin/bash\n",
"\"/opt/aws/bin/cfn-init -s ${AWS::StackId} -r MyInstance --region ${AWS::Region}\n",
""
] ] } } } } } } }
Place Fn::Include anywhere in the template and it will be replaced by the contents it is referring to. The function accepts an object. Parameters are:
- location: The location to the file can be relative or absolute. A relative location is interpreted relative to the template. Included files can in turn include more files, i.e. recursion is supported.
- ignoreMissingVar: If set to true the function will not throw an error if a variable is not found (unset). Instead, the variable will be replaced by an empty string. Defaults to false.true
- ignoreMissingFile: If set to the function will not throw an error if the file is not found. Instead, the function will be replaced by an empty string. Defaults to false.json
- type (optional): either , string or api. Defaults to json. string will include the file literally which is useful in combination with Fn::Sub. api will call any AWS API and return the response which can be included in the template. Choose json for both JSON and YAML files. The literal type is deprecated and uses the infamous Fn::Join syntax.type
- context (optional, deprecated): If is literal a context object with variables can be provided. The object can contain plain values or references to parameters or resources in the CloudFormation template (e.g. { "Ref": "StackId" }). Use Mustache like syntax in the file. This option is deprecated in favor of the Fn::Sub syntax (see examples below)."jmespath"
- parser (optional):
- string: default is {location, query, parser}
- object : default is "jmespath"[location, query, parser]
- array: : default is "lodash"location|query|parser
- string (split |) : default is "lodash"type
- query (optional): If is json, array, or string split |
- JMESPath query can be provided. The file to include is then queried using the value as a JMESPath expression.
- Lodash _.get query
Only applicable if type is api:
- service: Service to call (see AWSJavaScriptSDK, case sensitive, e.g. EC2, CloudFormation)updateStack
- action: Action to call (case sensitive, e.g. , describeRegions){ StackName: "MyStack" }
- parameters (optional): Parameters passed to action (e.g. )AWS_DEFAULT_REGION
- region (optional): Either or this parameter have to be set which specifies the region where the API call is made.
You can also use a plain string if you want the default behavior, which is simply including a JSON file.
- isGlob (optional): Forces the usage of glob to spit out an array of includes
- inject (optional): Pass in localized env / options to be injected into a template
#### Include via Query
`yamlall equivalent
Fn::Include:
location: ./t/includes/complex.json
query: bulb[1].c
Fn::Include:
location: ./t/includes/complex.json
query: bulb.1.c
parser: lodash
#### Include a file from a URL
`yaml
!Include https://example.com/include.json// equivalent to
Fn::Include:
type: json
location: https://example.com/include.json
`Include a file from an S3 bucket. Authentication is handled by
aws-sdk. See Setting AWS Credentials for details.`yaml
!Include s3://bucket-name/include1.json
`#### Include a file in the same folder
`yaml
!Include include.json
`Include a file literally and make use of
Fn::Sub:`yaml
Fn::Sub:
Fn::Include:
type: string
location: https://example.com/userdata.txt
`#### Include an AWS API response
IE: loop through all regions and return the image id of a specific AMI:
`yaml
Fn::Merge:
Fn::Map:
- Fn::Include:
action: describeRegions
query: "Regions[*].RegionName[]"
service: EC2
type: api
- _:
AMI:
Fn::Include:
action: describeImages
parameters:
Filters:
- Name: manifest-location
Values:
- amazon/amzn-ami-hvm-2016.03.3.x86_64-gp2
query: "Images[*].ImageId | [0]"
region: _
service: EC2
type: api
`Output as JSON:
`json
{ "ap-south-1": { "AMI": "ami-ffbdd790" },
"eu-west-1": {"AMI": "ami-f9dd458a" },
"ap-southeast-1": { "AMI": "ami-a59b49c6" },
...
}
`#### Include Globs
Essentially imagine if you had several yaml or json files you wanted to include.
`
./src/
files/
- one.yml
- two.yml
- three.yml
- four.yml
main.yml
`Before Glob you would have to do:
main.yml
`yml
Fn::Map:
- [one, two, three]
- [FILE]
- Fn::Include: ./files/${FILE}.yml
`With Glob
main.yml
`yml
Fn::Include: ./files/*.yml
`or (say you need to ignore something)
`yml
Fn::Include:
location: ./files/!(four).yml
isGlob: true
`#### Include Inject State
This feature uses the exact same logic as doEnv in that all env variables are traversed and replaced however this is
with localized state for the included file.
File to inject to:
toInject.yml - your include file`yml
SomeResource:
Name: ${LOCALIZED_NAME}
`Consume it and add some custom state
`yml
Fn::Include:
location: ./toInject.yml
inject:
LOCALIZED_NAME: CustomName
`yields
`yml
SomeResource:
Name: CustomName
`Fn::Map
Fn::Map is the equivalent of the lodash map() function allowing for the transformation of an input array or object to an output array.
By default the string _ is used as the variable in the map function. A custom variable can be provided as a second parameter, see Fn::Flatten for an example. If a custom variable is used, the variable will also be replaced if found in the object key, see Fn::Merge for an example.`yaml
Fn::Map:
- [80, 443]
- CidrIp: 0.0.0.0/0
FromPort: _
ToPort: _
IpProtocol: tcp
``json
[
{
"CidrIp": "0.0.0.0/0",
"FromPort": "80",
"ToPort": "80"
},
{
"CidrIp": "0.0.0.0/0",
"FromPort": "443",
"ToPort": "443"
}
]
`Custom variables can be specified as a single value, of as a list of up to three values. If a list is specified, the second variable is used as index and the third (if present) as size.
`yaml
Fn::Map:
- !Sequence [A, C]
- [NET, INDEX, N]
- Subnet${NET}:
Type: 'AWS::EC2::Subnet'
Properties:
CidrBlock: !Select [INDEX, !Cidr [MyCIDR, N, 8]]
``json
[{
"SubnetA": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": { "Fn::Select": [ 0, { "Fn::Cidr": [ "MyCIDR", 3, 8 ] } ] }
}
}
}, {
"SubnetB": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": { "Fn::Select": [ 1, { "Fn::Cidr": [ "MyCIDR", 3, 8 ] } ] }
}
}
}, {
"SubnetC": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": { "Fn::Select": [ 2, { "Fn::Cidr": [ "MyCIDR", 3, 8 ] } ] }
}
}
}]
`
Fn::Flatten
This function flattens an array a single level. This is useful for flattening out nested
Fn::Map calls.`yaml
SecurityGroupIngress:
Fn::Flatten:
Fn::Map:
- [80, 443]
- $
- Fn::Map:
- [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16]
- CidrIp: _
FromPort: $
ToPort: $
IpProtocol: tcp
`Results in:
`json
{
"SecurityGroupIngress": [
{
"CidrIp": "10.0.0.0/8",
"FromPort": "80",
"ToPort": "80",
"IpProtocol": "tcp"
},
{
"CidrIp": "172.16.0.0/12",
"FromPort": "80",
"ToPort": "80",
"IpProtocol": "tcp"
},
{
"CidrIp": "192.168.0.0/16",
"FromPort": "80",
"ToPort": "80",
"IpProtocol": "tcp"
},
{
"CidrIp": "10.0.0.0/8",
"FromPort": "443",
"ToPort": "443",
"IpProtocol": "tcp"
},
{
"CidrIp": "172.16.0.0/12",
"FromPort": "443",
"ToPort": "443",
"IpProtocol": "tcp"
},
{
"CidrIp": "192.168.0.0/16",
"FromPort": "443",
"ToPort": "443",
"IpProtocol": "tcp"
}
]
}
`Fn::FlattenDeep
This function flattens an array as many levels as possible. This is useful for flattening out nested
Fn::Map calls.`yaml
SecurityGroupIngress:
Fn::FlattenDeep:
Fn::Map:
- [80, 443]
- $
- Fn::Map:
- [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16]
- Fn::Map:
- [tcp, udp]
- PROTOCOL
- CidrIp: _
FromPort: $
ToPort: $
IpProtocol: PROTOCOL
`Results in:
`json
{
"SecurityGroupIngress": [
{
"CidrIp": "10.0.0.0/8",
"FromPort": "80",
"ToPort": "80",
"IpProtocol": "tcp"
},
{
"CidrIp": "10.0.0.0/8",
"FromPort": "80",
"ToPort": "80",
"IpProtocol": "udp"
},
{
"CidrIp": "172.16.0.0/12",
"FromPort": "80",
"ToPort": "80",
"IpProtocol": "tcp"
},
{
"CidrIp": "172.16.0.0/12",
"FromPort": "80",
"ToPort": "80",
"IpProtocol": "udp"
},
{
"CidrIp": "192.168.0.0/16",
"FromPort": "80",
"ToPort": "80",
"IpProtocol": "tcp"
},
{
"CidrIp": "192.168.0.0/16",
"FromPort": "80",
"ToPort": "80",
"IpProtocol": "udp"
},
{
"CidrIp": "10.0.0.0/8",
"FromPort": "443",
"ToPort": "443",
"IpProtocol": "tcp"
},
{
"CidrIp": "10.0.0.0/8",
"FromPort": "443",
"ToPort": "443",
"IpProtocol": "udp"
},
{
"CidrIp": "172.16.0.0/12",
"FromPort": "443",
"ToPort": "443",
"IpProtocol": "tcp"
},
{
"CidrIp": "172.16.0.0/12",
"FromPort": "443",
"ToPort": "443",
"IpProtocol": "udp"
},
{
"CidrIp": "192.168.0.0/16",
"FromPort": "443",
"ToPort": "443",
"IpProtocol": "tcp"
},
{
"CidrIp": "192.168.0.0/16",
"FromPort": "443",
"ToPort": "443",
"IpProtocol": "udp"
}
]
}
`Fn::GetEnv
`yaml
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !GetEnv [BUCKET_NAME, !Ref AWS::NoValue]
`The second argument is optional and provides the default value and will be used of the environmental variable is not defined. If the second argument is omitted
!GetEnv BUCKET_NAME and the environmental variable is not defined then the compilation will fail.Fn::Length
Fn::Length returns the length of a list or expanded section.
Fn::Merge
Fn::Merge will merge an array of objects into a single object. See lodash / merge for details on its behavior. This function is useful if you want to add functionality to an existing template if you want to merge objects of your template that have been created with Fn::Map.Fn::Merge accepts a list of objects that will be merged together. You can use other cfn-include functions such as Fn::Include to pull in template from remote locations such as S3 buckets.`yaml
Fn::Merge:
- !Include s3://my-templates/my-template.json - !Include s3://my-templates/my-other-template.json
- Parameters:
MyCustomParameter:
Type: String
Resources:
MyBucket:
Type: AWS::S3::Bucket
`Fn::DeepMerge
Fn::DeepMerge will deeply merge an array of objects and arrays into a single object. See deepmerge for details on its behavior. This function is useful if you want to add functionality to an existing template if you want to merge objects of your template that have been created with Fn::Map.Fn::DeepMerge accepts a list of objects that will be merged together. You can use other cfn-include functions such as Fn::Include to pull in template from remote locations such as S3 buckets.To understand it better besides the below example refer to this test. Note that all arrays are concatenated.
Why does
Fn::Merge still exist? Answer: Backwards compatibility for expected behavior.`yaml
Fn::DeepMerge:
- !Include s3://my-templates/my-template.json - !Include s3://my-templates/my-other-template.json
- Parameters:
MyCustomParameter:
Type: String
Resources:
MyBucket:
Type: AWS::S3::Bucket
`This snippet shows how multiple subnets can be created for each AZ and then merged with the rest of the template.
`yaml
Resources:
IAMUser:
Type: AWS::IAM::User
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Fn::Merge:
Fn::Map:
- [A, B]
- AZ
- Subnet${AZ}:
Type: AWS::EC2::Subnet
``json
{
"Resources": {
"SubnetA": {
"Type": "AWS::EC2::Subnet"
},
"SubnetB": {
"Type": "AWS::EC2::Subnet"
},
"SG": {
"Type": "AWS::EC2::SecurityGroup"
}
}
}
`Fn::Sequence
Fn::Sequence generates a sequence of numbers of characters. You can specify the start, end and step.`yaml
!Sequence [1, 4]generates
[1, 2, 3 4]!Sequence [1, 10, 2]
generates
[1, 3, 5, 7, 9]!Sequence [a, d]
generates
[a, b, c, d]
`Fn::Sequence can be used in combination with Fn::Map to generate complex objects:`
Fn::Map:
- !Sequence [a, c]
- AZ
- Subnet${AZ}:
Type: AWS::EC2::Subnet
`Fn::Stringify
Fn::Stringify will take the passed value and transform it to a JSON string. This is useful for parameters that require a JSON document as a string. Using this function, you can keep writing your configuration in YAML and let the function transform it into a JSON string.Another useful application is the use of this function in a config file passed as
--cli-input-json parameter.`yaml
stack.config.yml
StackName: MyStack
TemplateBody:
Fn::Stringify: !Include mytemplate.yml
Parameters:
- ParameterKey: Foo
ParameterValue: Bar
`You can then simply run the following command to deploy a stack:
`
cfn-include stack.config.yml > stack.config.json
aws cloudformation create-stack --cli-input-json file://stack.config.json
`Fn::Uniq
This function filters only the unique elements of an array
`yaml
SecurityGroupIngress:
Fn::Uniq:
Fn::Flatten:
- [1, 2]
- [3, 4]
- [1, 4, 6]
`Results in:
`json
{
"SecurityGroupIngress": [
1,
2,
3,
4,
6
]
}
`Fn::Compact
This function removes falsy elements same as lodash
`yaml
SecurityGroupIngress:
Fn::Compact:
- 1
- a
- ""
- false
- true
`Results in:
`json
{
"SecurityGroupIngress": [
1,
"a",
true
]
}
`Fn::Concat
_.concat
`yaml
Fn::Concat:
- [a, b, c]
- d
`Results in:
`json
[
"a",
"b",
"c",
"d"
]
`Fn::Sort
$ ./bin/cli.js examples/sort.yaml`json
[
1,
20,
22,
30,
30,
33.3,
40,
5.5,
50,
50
]
`Fn::SortedUniq
$ ./bin/cli.js examples/sortedUniq.yaml`json
[
1,
20,
22,
30,
33.3,
40,
5.5,
50
]
`Fn::SortBy
$ ./bin/cli.js examples/sortBy.yaml`json
[
{
"name": "Ana",
"age": 12
},
{
"name": "Ana",
"age": 31
},
{
"name": "Bob",
"age": 17
},
{
"name": "Colby",
"age": 35
},
{
"name": "Fred",
"age": 50
},
{
"name": "Jack",
"age": 40
},
{
"name": "Ted",
"age": 20
},
{
"name": "Zed",
"age": 90
}
]
`Fn::SortObject
$ ./bin/cli.js examples/sortObject.yaml`json
{
"a": "hi",
"c": "z",
"h": 20,
"i": true,
"z": 1
}
`Fn::ObjectKeys
This function uses Object.keys
`yaml
FamilyNames:
Fn::ObjectKeys:
Ted: 18
Lucy: 5
Tom: 10
`Results in:
`yaml
FamilyNames:
- Ted
- Lucy
- Tom
`Fn::ObjectValues
This function uses Object.values
`yaml
FamilyAges:
Fn::ObjectValues:
Ted: 18
Lucy: 5
Tom: 10
`Results in:
`yaml
FamilyAges:
- 18
- 5
- 10
`Fn::Filenames
`yaml
Fn::Filenames:
location: "../t/fixtures"
omitExtension: true
``json
[
"deep",
"foobar",
"subfolder",
"synopsis",
"verydeep",
"include1",
"include2"
]
`Fn::StringSplit
Useful for injected Includes which you need to run
Fn::Map upon.`yaml
Fn::StringSplit:
string: "A,B,C"
separator: "," # defaults to this so it can be omitted
``yaml
- A
- B
- C
`Fn::Without
`yaml
Fn::Without:
- ["a", "b", "c", "d"]
- ["b", "c"]
``yaml
- "a",
- "d"
`Fn::Omit
`yaml
b: b
`Fn::OmitEmpty
See omitEmpty test file, file and expectations (output).
In summary falsy values are omitted from an object except
false and 0.Fn::Eval
Opt in to use
eval in your templates. This is disabled by default.--enable eval is required to turn on options.doEval in the include function.`yaml
Fn::Eval:
state: [1, 2, 3]
script: >
state.map((v) => 2 * v);
``yaml
- 2
- 4
- 6
`Fn::IfEval
Opt in to use
eval in your templates. This is disabled by default.--enable eval is required to turn on options.doEval in the include function.`yaml
Fn::IfEval:
inject:
lastName: bear
# doLog: true
evalCond: ('$lastName' === 'bear')
truthy:
Name: Yogi
LastName: Bear
falsy:
Name: Fred
LastName: Flint
``yaml
Name: Yogi
LastName: Bear
`Fn::JoinNow
`yaml
Fn::JoinNow:
- ""
- - "arn:aws:s3:::c1-acme-iam-cache-engine-"
- ${AWS::AccountId}
- "-us-east-1$CFT_STACK_SUFFIX"
``yaml
arn:aws:s3:::c1-acme-iam-cache-engine-${AWS::AccountId}-us-east-1$CFT_STACK_SUFFIX
`Fn::SubNow
Fn::SubNow performs immediate string substitution similar to AWS CloudFormation's Fn::Sub, but evaluates during template preprocessing rather than at stack creation time. It supports variable substitution using ${VariableName} syntax and AWS pseudo-parameters.The function supports two input formats:
String format:
`yaml
Fn::SubNow: "arn:aws:s3:::bucket-${BucketSuffix}"
`Array format with variables:
`yaml
Fn::SubNow:
- "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${LogGroupName}"
- LogGroupName: /aws/lambda/my-function
`Supported AWS pseudo-parameters:
-
${AWS::AccountId} - AWS Account ID
- ${AWS::Region} - AWS Region
- ${AWS::StackName} - Stack name
- ${AWS::StackId} - Stack ID
- ${AWS::Partition} - AWS Partition (e.g., 'aws')
- ${AWS::URLSuffix} - URL suffix (e.g., 'amazonaws.com')Variables can be provided via:
1. The
inject option passed to cfn-include
2. Explicit variables in the array format (takes precedence over inject)
3. Environment variables when using the doEnv optionExample with environment variables:
`yaml
BucketName: !SubNow "my-bucket-${Environment}-${AWS::Region}"
`With
CFN_INCLUDE_DO_ENV=true and environment variable ENVIRONMENT=prod, this resolves to:
`yaml
BucketName: "my-bucket-prod-us-east-1"
`Fn::RefNow
Fn::RefNow resolves a reference immediately during template preprocessing, similar to AWS CloudFormation's Fn::Ref but evaluated at processing time rather than stack creation time. It resolves references to parameters, variables, and AWS pseudo-parameters.Basic syntax:
`yaml
BucketRef:
Fn::RefNow: BucketName
`or using YAML tag syntax:
`yaml
BucketRef: !RefNow BucketName
`Supported AWS pseudo-parameters:
-
AWS::AccountId - AWS Account ID (environment: AWS_ACCOUNT_ID or AWS_ACCOUNT_NUM, fallback: ${AWS::AccountId})
- AWS::Region - AWS Region (environment: AWS_REGION, fallback: ${AWS::Region})
- AWS::StackName - Stack name (environment: AWS_STACK_NAME, fallback: ${AWS::StackName})
- AWS::StackId - Stack ID (environment: AWS_STACK_ID, fallback: ${AWS::StackId})
- AWS::Partition - AWS Partition (environment: AWS_PARTITION, default: 'aws')
- AWS::URLSuffix - URL suffix (environment: AWS_URL_SUFFIX, default: 'amazonaws.com')
- AWS::NotificationARNs - SNS topic ARNs for notifications (environment: AWS_NOTIFICATION_ARNS, fallback: ${AWS::NotificationARNs})Reference resolution priority:
1. AWS pseudo-parameters (with environment variable fallbacks)
2. Variables from the
inject option
3. Variables from the current scope (useful with Fn::Map)Reference indirection:
If a resolved reference is a string, it will be treated as a reference name and resolved again. This enables reference chaining, useful when using
Fn::RefNow with Fn::Map:`yaml
Fn::Map:
- [BucketVar1, BucketVar2]
- BucketName:
Fn::RefNow: _
`With
inject: { BucketVar1: "my-bucket-1", BucketVar2: "my-bucket-2" }, this via $ Bucket1=my-bucket-1 BucketVar2=my-bucket-2 cnf-include examples/refNow.yml --enable env,eval via exports / envor
$ cnf-include examples/refNow.yml --enable env,eval --inject '{"BucketVar1":"my-bucket-1","BucketVar2":"my-bucket-2"}' via injectresolves to:
`yaml
BucketVar1: my-bucket-1
BucketVar2: my-bucket-2
`The
_ placeholder resolves to "BucketVar1" or "BucketVar2", which are then resolved again to their actual values.Example with injected variables:
`yaml
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
Fn::RefNow: BucketName
`With
inject: { BucketName: "my-app-bucket" }, this resolves to:
`yaml
BucketName: "my-app-bucket"
`Example with AWS pseudo-parameters:
`yaml
LogGroup:
Fn::RefNow: AWS::StackName
`With
AWS_STACK_NAME=my-stack, this resolves to:
`yaml
LogGroup: "my-stack"
`Unresolved AWS pseudo-parameters:
If an AWS pseudo-parameter is not set via environment variables, it falls back to a placeholder string (e.g.,
${AWS::AccountId}). Only AWS::Partition and AWS::URLSuffix have hardcoded defaults since they are rarely environment-specific.Error handling:
If a reference cannot be resolved,
Fn::RefNow will throw an error. Ensure all referenced parameters and variables are available via inject, scope, or environment variables.Resolving LogicalResourceIds:
Fn::RefNow can also resolve references to CloudFormation Resource LogicalResourceIds, enabling you to construct ARNs or other resource-specific values during template preprocessing. When a reference matches a LogicalResourceId in the Resources section, Fn::RefNow will automatically generate the appropriate ARN based on the resource type and properties.Supported Resource Types for ARN/Name Resolution:
-
AWS::IAM::ManagedPolicy - Returns policy ARN (supports Path)
- AWS::IAM::Role - Returns role ARN (supports Path)
- AWS::IAM::InstanceProfile - Returns instance profile ARN (supports Path)
- AWS::S3::Bucket - Returns bucket ARN
- AWS::Lambda::Function - Returns function ARN
- AWS::SQS::Queue - Returns queue ARN
- AWS::SNS::Topic - Returns topic ARN
- AWS::DynamoDB::Table - Returns table ARN
- AWS::RDS::DBInstance - Returns DB instance ARN
- AWS::SecretsManager::Secret - Returns secret ARN
- AWS::KMS::Key - Returns key ARNExample with AWS::IAM::ManagedPolicy:
`yaml
Resources:
ObjPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
ManagedPolicyName: teststack-CreateTestDBPolicy-16M23YE3CS700
Path: /CRAP/ IAMRole:
Type: AWS::IAM::Role
Properties:
ManagedPolicyArns:
- Fn::RefNow: ObjPolicy # Resolves to: arn:aws:iam::${AWS_ACCOUNT_ID}:policy/CRAP/teststack-CreateTestDBPolicy-16M23YE3CS700
`Returning Resource Names Instead of ARNs:
By default,
Fn::RefNow returns the ARN for supported resource types. However, if the key name ends with Name (e.g., RoleName, BucketName, FunctionName), it automatically returns the resource name/identifier instead:`yaml
Resources:
MyRole:
Type: AWS::IAM::Role
Properties:
RoleName: MyExecutionRole RoleArn:
Fn::RefNow: MyRole # Returns: arn:aws:iam::${AWS_ACCOUNT_ID}:role/MyExecutionRole
RoleName:
Fn::RefNow: MyRole # Returns: MyExecutionRole (because key ends with "Name")
`This intuitive approach makes templates more readable and follows natural CloudFormation naming conventions.
CLI Options for Fn::RefNow:
The
cfn-include command provides CLI options to control how unresolved references are handled:-
--ref-now-ignore-missing: Do not fail if a Fn::RefNow reference cannot be resolved. Instead, the reference will be returned in AWS CloudFormation's standard Ref syntax (e.g., { Ref: 'UnresolvedRef' }), allowing CloudFormation to resolve it at stack creation time.
- --ref-now-ignores : Comma-separated list of reference names to ignore if not found. These references will be returned in Ref syntax instead of throwing an error.Example usage:
`bash
Ignore all unresolved references
cfn-include template.yaml --ref-now-ignore-missingIgnore specific reference names
cfn-include template.yaml --ref-now-ignores "OptionalRef1,OptionalRef2"Combine both options
cfn-include template.yaml --ref-now-ignore-missing --ref-now-ignores "SpecificRef"
`This is useful for templates that reference CloudFormation parameters or other resources that may not be available at template processing time but will be available at stack creation time.
Fn::ApplyTags
See ApplyTags test file.
Fields:
(T|t)ags: sequence of {Key, Value} objects to me merged in as Tags properties of a taggable resource.
resources: Object mapping of resources, this is usually your root CFT.Resources block.Fn::UpperCamelCase
`yaml
Name: !UpperCamelCase foo-bar # yields FooBar
`Fn::Outputs
This helper transformation simplifies the definition of output variables and exports.
`yaml
Outputs:
Fn::Outputs:
Version: !GetEnv [VERSION, "1.0.0"]
BucketArn: ${Bucket.Arn}
BucketPolicy:
Condition: HasBucketPolicy
Value: ${BucketPolicy}
Subnets:
- ${SubnetA},${SubnetB},${Provided}
- Provided: ${SubnetC}
`This will translate into:
`yaml
Outputs:
Version:
Value: !Sub "1.0.0"
Export:
Name: !Sub ${AWS::StackName}:Version BucketArn:
Value: !Sub ${Bucket.Arn}
Export:
Name: !Sub ${AWS::StackName}:BucketArn
BucketPolicy:
Value: !Sub ${BucketPolicy}
Condition: HasBucketPolicy
Export:
Name: !Sub ${AWS::StackName}:BucketPolicy
Subnets:
Value:
Fn::Sub:
- ${SubnetA},${SubnetB},${Provided}
- Provided: ${SubnetC}
Export:
Name: !Sub ${AWS::StackName}:Subnets
`More Examples
See /examples for templates that call an API Gateway endpoint to collect AMI IDs for all regions. There is also a good amount of tests that might be helpful.
A common pattern is to process a template, validate it against the AWS validate-template API, minimize it and upload the result to S3. You can do this with a single line of code:
`bash
cfn-include example.template -t -m | aws s3 cp - s3://bucket-name/output.template
`Proxy Support
cfn-include honors proxy settings defined in the https_proxy environmental variable. The module will attempt to load proxy-agent. Make sure proxy-agent is installed since it is not a dependency for this module.Compatibility
Node.js versions 8 and up are supported both on Windows and Linux.
Web Service
curl https://api.netcubed.de/latest/template?[options] -XPOST -d @
-
path the contents of
path will be POSTed to the web service. See man curl for details.Options:
Options are query parameters.
-
validate=false do not validate template [true]Developer Documentation
$3
The
cfn-include preprocessor supports comprehensive debug logging for tracing template processing. When enabled, the doLog option logs all arguments at each recursion level in the template processing pipeline.#### Enabling doLog
CLI:
`bash
cfn-include template.yaml --doLog
`Programmatic:
`javascript
include({
template: myTemplate,
url: 'file:///path/to/template.yaml',
doLog: true
})
`#### Understanding Caller Parameter
The
recurse function now includes a caller parameter to help identify which function triggered each recursion step. This is invaluable for debugging complex templates with nested function calls. The caller parameter provides a trace path like:-
recurse:isArray - Processing an array
- Fn::Map - Inside an Fn::Map function
- Fn::Include - Inside an Fn::Include function
- recurse:isPlainObject:end - Final plain object processing
- handleIncludeBody:json - JSON body being processedWhen
doLog is enabled, the console output will show the caller for each recursion:
`javascript
{
base: {...},
scope: {...},
cft: {...},
rootTemplate: {...},
caller: "Fn::Map",
doEnv: false,
doEval: false,
...
}
`This makes it easy to trace execution flow through nested
Fn::Include, Fn::Map, Fn::RefNow, and other functions.#### Example Debug Output
`bash
$ cfn-include examples/base.template --doLog | head -50
{
base: {
protocol: 'file',
host: '/Users/SOME_USER/code',
path: '/examples/base.template'
},
scope: {},
cft: { AWSTemplateFormatVersion: '2010-09-09', ... },
rootTemplate: { AWSTemplateFormatVersion: '2010-09-09', ... },
caller: 'recurse:isPlainObject:end',
doEnv: false,
doEval: false,
inject: undefined,
doLog: true
}
`$3
#### CLI Options for Reference Resolution
Two new CLI options control how unresolved
Fn::RefNow references are handled:-
--ref-now-ignore-missing: Do not fail if a reference cannot be resolved. Instead, return the reference in CloudFormation's standard Ref syntax, allowing CloudFormation to resolve it at stack creation time.-
--ref-now-ignores : Comma-separated list of specific reference names to ignore if not found. Useful for optional references.Example usage:
`bash
Ignore all unresolved references
cfn-include template.yaml --ref-now-ignore-missingIgnore specific references
cfn-include template.yaml --ref-now-ignores "OptionalParam,CustomRef"Combine both
cfn-include template.yaml --ref-now-ignore-missing --ref-now-ignores "SpecificRef"
`#### rootTemplate Parameter
The
recurse function now receives the complete rootTemplate for all recursion calls. This enables Fn::RefNow to resolve references to CloudFormation resources defined anywhere in the template, even when processing deeply nested includes or function results.$3
New intrinsic functions for immediate string substitution and joining:
-
Fn::SubNow - Performs immediate string substitution similar to Fn::Sub, but evaluates at template processing time
- Fn::JoinNow - Joins array elements into a string at template processing timeSee the main documentation sections above for detailed usage.
$3
The template processing follows this call chain for better debugging:
1. Entry point calls
recurse() with caller: undefined
2. Array elements call recurse() with caller: 'recurse:isArray'
3. Each Fn::* function calls recurse() with caller: 'Fn::FunctionName'
4. Final plain object recursion uses caller: 'recurse:isPlainObject:end'
5. Include body processing uses caller: 'handleIncludeBody:json'When combined with
--doLog, this provides complete visibility into how cfn-include processes your templates.To compile the synopsis run the following command.
`
curl -Ssf -XPOST https://api.netcubed.de/latest/template -d '{"Fn::Include":"https://raw.githubusercontent.com/nmccready/cfn-include/master/examples/synopsis.json"}' > output.template
``