AWS SSM Parameter Store secrets management for Docker containers
Secure way to provide environment secrets to docker containers from AWS Parameter Store.
Storing and using secret information securely with AWS SSM
The SSM Parameters store is a great way to securely store hierarchical configuration data. It is widely used by AWS to provide parameters it own services. The names of the parameters are guaranteed to be unique and of arbitrary structure, allowing users to store nested structures of configuration data. In this post I will examine how to use Parameters Store efficiently to provide settings and secrets to Docker containers executed on ECS.
The code, examples and additional information for this post is available on the GitHub, and images instrumented with SSM bootstrap are available from the Docker hub.
AWS SSM Parameters Store Values
The SSM allows to use both single and list values for parameters. Optionally the value
can be encrypted with the KMS key. The encryption allows user to keep environmental secrets in SSM, such as service to service authentication, docker registry auth strings, database credentials, X.509 keys and everything else that fits into the 4096 characters. Values that do not fit into 4096 characters, can be stored into the S3 bucket and pre-signed access URL can be written into the SSM parameter instead.
Using the Parameter Store
The main approach to usage of the SSM parameters store is give your ECS task a role to access SSM API and call it on service start. The service itself is responsible for communication with SSM, in a safe and scalable way. This presents a natural problem in case of multiple services, probably written in different languages, each having a unique implementation of SSM nested structures reading code. It is better to decouple configuration and secrets retrieval logic from the application code with a formal contract. Environmental variables are the most common way to pass configuration values to the application code in the Docker containers and it is convenient to use them for SSM stored configuration as well.
Docker images and entrypoints
Application service packaged as Docker image can benefit from the use of intermediate base images which contain bootstrapping code and decouple secrets management from the application itself. The intermediate image can define an entrypoint, used to perform bootstrapping steps, communicate with AWS services are prepare container environment before executing the service code. The image itself can be based on a number of possible runtime environments: NodeJS, Ruby or Python. The entrypoint in docker image receives the `CMD` value of the image using it as a base, or an executable argument of `docker run`, and can invoke it directly after the setup. Below is an example of the intermediate Docker image adding SSM bootstrapping capability to an alpine based image, such as NodeJS's node:alpine.
ARG BASE FROM $BASE # Install python runtime for the bootstrap script RUN apk update && \ apk add python py-pip py-yaml && \ pip install awscli && \ pip install boto3 # Copy bootstrap scripts to image COPY src/ssm-bootstrap.py /usr/bin/ssm-bootstrap COPY src/kickstart.sh /usr/bin/kickstart RUN chmod +x /usr/bin/ssm-bootstrap /usr/bin/kickstart ENTRYPOINT /usr/bin/kickstart
Here we have an entrypoint defined as /usr/bin/kickstart script. This script will execute /usr/bin/ssm-bootstrap to communicate with SSM and save environment file and other files. Lets examine the contents of the /usr/bin/kickstart:
#!/bin/sh # query SSM parameters store for secrets and save # files and environment variables ssm-bootstrap --environ /tmp/app_environ --root /app/ [ -f /tmp/app_environ ] && . /tmp/app_environ exec "$@"
The kickstart script uses /usr/bin/ssm-bootstrap to create /tmp/app_environ file, loads it as environment and passes execution to it's arguments. So that executed process would inherit environment populated with data from the /tmp/app_environ file. Additionally /usr/bin/ssm-bootstrap would write a number of files at root path specified as --root argument. Such files can be various encryption keys, salts and certificates shared across your environments. Using SSM bootstrapping to provide files to Docker containers allows to avoid volume mounts to containers and specialization of your ECS host. Otherwise you have to bake such files into your an AMI or somehow persist on host's filesystem from external storage, so that files could be volume mounted into running containers.
SSM Parameters Namespaces
The actual communications with SSM to retrieve environmental variables is done by the /usr/bin/ssm-bootstrap tool. It is the place where logic to build parameter paths is implemented and shared across all services based on the intermediate Docker image. The interface between infrastructure code and application code, defined as environment variables and (smallish) files, is implemented once and reused everywhere else. Potential structure of the parameter names can be arbitrary, but in general it is useful to have at least 2 nested levels of parameters:
- ECS Cluster - Each service on a specific ECS cluster. The cluster scope corresponds to an instance of portable environment, usually CloudFormation stack(s).
- ECS Service/Container name - Specific service on a specific ECS cluster.
Integration with the ECS
SSM bootstrap is using ECS metadata file determine the cluster and container name it is executed for. It must be enabled in your `ecs.config` on cluster instances. When ECS runs Docker container, it makes metadata file available inside the container. This metadata contains cluster name and container name used by ECS.
Below is an example of the ECS Task Definition in CloudFormation, lets examine what configuration path are available to such service:
ExampleTask:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Name: example-service
Image: ...
Essential: true
PortMappings:
- ContainerPort: 8080
Environment:
AWS_DEFAULT_REGION: !Ref AWS::Region
...
ExampleCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: example-cluster
ExampleService:
Type: AWS::ECS::Service
Properties:
TaskDefinition: !Ref ExampleTask
Cluster: !Ref ExampleCluster
...Notice that parameter names are built using Cluster and Container Name. This information about running container is read from ECS container metadata file, so this needs to be enabled in your ECS agent configuration. The source code for the /usr/bin/ssm-bootstrap utility can be found on the GitHub.
Name structure for nested configurations
With the configuration above the service would have the following paths in SSM injected into it's environment:
Also the following paths would be saved as files:
Task Role and Policy for SSM access
In order to access SSM API the task still needs to have a role allowing that. Below is an example of such role in CloudFormation syntax.
ExampleTaskRole:
Type: AWS::IAM::Role
Properties:
RoleName: ExampleTaskRole
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- sts:AssumeRole
ExamplePolicy:
Type: AWS::IAM::Policy
Properties:
Roles:
- !Ref ExampleTaskRole
PolicyName: ExamplePolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Action:
- kms:Decrypt
Resource: !Sub "arn:aws:kms:${AWS::Region}:alias/my-key"
Effect: Allow
- Action:
- ssm:GetParameter
- ssm:GetParameters
- ssm:DescribeParameters
Resource: "arn:aws:ssm:*"
Effect: AllowUsing published images
The images with SSM bootstrap middleware layer are published to the Docker Hub and available for general use. Simply specify the base image for your runtime in the FROM statement and your service will automatically receive configuration from the SSM Parameters Store. Below is an example of a generic Dockerfile for NodeJS based service:
FROM stan1y/ssm-bootstrap:node-alpine-latest WORKDIR /app COPY src/ /app/src/ COPY .npmrc package.json /app/ RUN npm install && rm -f .npmrc CMD npm start