Serverless Framework Plugin for interacting with AWS Batch
npm install @rlyle1179/serverless-aws-batch
npm install serverless-aws-batch --save-dev
`
Modify the serverless.yml file to include the plugin:
`yaml
plugins:
- serverless-aws-batch
`
Next we need to define our AWS Batch Compute Resource
`yaml
provider:
name: aws
region: us-east-2
runtime: python3.7
batch:
Type: [EC2 | SPOT] # Required
BidPercentage: # Optional. Defaults to 100 if Type = SPOT (you always pay lowest market price)
SecurityGroupIds: # Required
-
Subnets: # Required
-
InstanceTypes: # Optional
- # Default c5.large (cheapest)
MinvCpus: # Optional. Default 0
MaxvCpus: # Optional. Default 2
Tags: # Optional
: # Default "Name": "AWS Batch Instance - "
`
And then define our AWS Batch Job Definition
on the function definition
`yaml
functions:
hello:
handler:
batch:
ContainerProperties:
Memory: # Optional. Default 2048
Vcpus: # Optional. Default 1
Command: # Optional. Defaults to " Ref::event"
JobRoleArn: # Optional. Defaults to package.iamRoleStatements
Environment:
TEST_ENV_1: Test Value 1
RetryStrategy:
Attempts: # Optional. Defaults to 1
Timeout:
AttemptDurationSeconds: # Optional. Defaults to 300
`
And now you should be able to write your batch function like you would any other serverless lambda function:
`python3
import logging
Setup our logger to work both locally and with both AWS CloudWatch
if len(logging.getLogger().handlers) > 0:
logging.getLogger().setLevel(logging.INFO)
else:
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
def hello(event, context):
logger.info(f"Hello World: {event}")
`
`yaml
functions:
hello:
handler: handler.hello
events:
- http:
path: hello
method: get
batch:
ContainerProperties:
Memory: 2048
Vcpus: 1
RetryStrategy:
Attempts: 1
Timeout:
AttemptDurationSeconds: 3600
`
Implementation
What happens behind the scenes is that the Serverless Framework's ".zip" artifact gets installed into a
lambci/lambda:\
* _Note: Currently using copy of the docker images (https://cloud.docker.com/u/justinram11/repository/list) that unsets the
ACCESS_KEY_ID and SECRET_ACCESS_KEY environmental variables so that we can use the role attached to the EC2 instance._
A "schedule" lambda function is then created with the same name as the regular Serverless Framework's lambda function
that can be called in to pass the input event to the AWS Batch Job. The "schedule" lambda function can also be subscribed
to events the same way a regular Serverless Lambda Function is.
!serverless-batch
Example
After deploying with sls deploy, navigate to the AWS Lambda Console
and create a new Test Event with any event value and click the "Test" button.
The lambda function will automatically create a new AWS Batch Job which should be visible on the AWS Batch Console
Logs are visible in CloudWatch under the /aws/batch/job Log Group
serverless.yml
`yaml
service: serverless-demo
provider:
name: aws
region: us-east-2
runtime: python3.7 | nodejs12.x # TODO Select one
# Creates a SPOT Compute Environment using the VPC defined in our Resources
batch:
Type: SPOT
BidPercentage: 100
SecurityGroupIds:
- Ref: AllowAllSecurityGroup
Subnets:
- Ref: PublicSubnet
MinvCpus: 0
MaxvCpus: 2
# Allows the Batch Job (code written in handler.py) to list all of our S3 Buckets.
iamRoleStatements:
- Effect: "Allow"
Action:
- "s3:ListAllMyBuckets"
Resource: "*"
plugins:
# TODO Required if a python project
# - serverless-python-requirements
- serverless-aws-batch
package:
include:
- handler.py
exclude:
- .serverless/**
- node_modules/**
functions:
hello:
handler: handler.hello
events:
- http:
path: hello
method: get
# Creates a Batch Job with 2GB of memory and 1 vCPU.
batch:
ContainerProperties:
Memory: 2048
Vcpus: 1
RetryStrategy:
Attempts: 1
Timeout:
AttemptDurationSeconds: 3600
WARNING: Should not be used in production (allows all traffic)
VPC: "Batch VPC"
- CIDR: 10.224.0.0/16
- Subnet: "Batch a Public Subnet"
- CIDR: 10.224.0.0/20
- Allows all incoming and outgoing traffic
- Security Group: "Batch Allow All"
- Allows all incoming and outgoing traffic
resources:
Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.224.0.0/16
EnableDnsSupport: true
EnableDnsHostnames: true
Tags:
- Key: Name
Value: Batch VPC
InternetGateway:
Type: AWS::EC2::InternetGateway
Properties:
Tags:
- Key: Name
Value: Batch Internet Gateway
InternetGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
InternetGatewayId: !Ref InternetGateway
VpcId: !Ref VPC
PublicSubnet:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: ${self:provider.region}a
CidrBlock: 10.224.0.0/20
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: Batch ${self:provider.region}a Public Subnet
PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: Batch Public Routes
DefaultPublicRoute:
Type: AWS::EC2::Route
DependsOn: InternetGatewayAttachment
Properties:
RouteTableId: !Ref PublicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
PublicSubnetRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PublicRouteTable
SubnetId: !Ref PublicSubnet
PublicNetworkAcl:
Type: AWS::EC2::NetworkAcl
DependsOn: VPC
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: AWS Batch Public ACL
InboundPublicNetworkAclAllowAll:
Type: AWS::EC2::NetworkAclEntry
Properties:
NetworkAclId: !Ref PublicNetworkAcl
RuleNumber: 100
Protocol: -1
RuleAction: allow
Egress: false
CidrBlock: 0.0.0.0/0
PortRange:
From: 0
To: 65535
OutboundPublicNetworkAclAllowAll:
Type: AWS::EC2::NetworkAclEntry
Properties:
NetworkAclId: !Ref PublicNetworkAcl
RuleNumber: 100
Protocol: -1
RuleAction: allow
Egress: true
CidrBlock: 0.0.0.0/0
PortRange:
From: 0
To: 65535
PublicSubnetNetworkAclAssociation1:
Type: AWS::EC2::SubnetNetworkAclAssociation
Properties:
SubnetId: !Ref PublicSubnet
NetworkAclId: !Ref PublicNetworkAcl
AllowAllSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: Batch Allow All
GroupDescription: "Security group for batch instances that allows all traffic"
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: "-1"
CidrIp: 0.0.0.0/0
SecurityGroupEgress:
- IpProtocol: "-1"
CidrIp: 0.0.0.0/0
`
$3
handler.py
`python3
import json
import time
import logging
import boto3
Setup our logger to work with both AWS CloudWatch and locally
if len(logging.getLogger().handlers) > 0:
logging.getLogger().setLevel(logging.INFO)
else:
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
Setup our boto3 clients
s3 = boto3.client('s3')
def hello(event, context):
logger.info(f"Hello world: {event}")
response = s3.list_buckets()
logger.info(f"S3 Buckets: {response}")
`
$3
handler.js
`nodejs
'use strict';
const process = require("process");
const AWS = require("aws-sdk");
const s3 = new AWS.S3();
module.exports.hello = (event, context, callback) => {
console.log(Received event: ${JSON.stringify(event)});
console.log(process.env.AWS_ACCESS_KEY_ID);
console.log(process.env.AWS_SECRET_ACCESS_KEY);
s3.listBuckets({}, function(err, data) {
console.log(List buckets data: ${data} err: ${err});
});
};
``