Музыку играет самый лучший mpv. И надо установить отдельно yay -S mpv.

Получилось, как мне думается, нормально. И я уже больше не буду использовать tera. Устанавливать пока что можно только из репозитория, но потом будет и AUR.

The topic of ECS autoscaling is a vast area of heated discussions and broken dreams. It is quite hard to come up with efficient scaling policies for your ECS services. And the more distributed your architecture, the more issues with cascading load and increasing latency you are going to face. But fear not, the promised salvation in form of autoscaling for your services is here to save the day and distribute your computing load evenly across your micro services. So lets examine what we have to work with to achieve that.

Scaling services

Autoscaling of ECS services is implemented as an automated action executed upon an event: scale in or scale out. The source of such event can be either an alarm with StepScaling type of policy or TargetTrackingScaling type. Usage of the target tracking is very similar to the implementation for DynamoDB, with selection of ECSServiceAverageCPUUtilization and ECSServiceAverageMemoryUtilization metrics available for tracking. Notice that ECS can track only average metrics of the service, so it means you need to make sure that tasks have load distributed evenly on load balancer. Significant gaps between maximum and average consumption can lead to a termination of a task with out of memory or CPU and lead to 502 errors.

New and Old

ECS scaling policies can be combined to produce even greater efficiency in load distribution. Usage of the `StepScaling` policies to handle scale out events on ALB or SQS metrics by estimating the load in the input source. ALB's Target Group metrics such as `AWS/ApplicationELB/RequestCountPerTarget` are good baseline to start policies. Size of the SQS queue is another example of deterministic metric to estimate incoming load for service.

Combination of StepScaling and TargetTrackingScaling looking at ECSServiceAverageCPUUtilization or ECSServiceAverageMemoryUtilization can allow greater flexibility in the how your service can react on load. If it is possible to determine of the service in question is mostly CPU or memory bound, then selection of a threshold for one of these average metrics should be pretty easy by observing the service under generated test load.

CloudFormation support for ECS scaling

To define an ECS service with scaling policies in CloudFormation you need to have a cluster, instance role for EC2 hosts and other essentials omitted from this example.

First we need a service role to perform scaling actions on our behalf.

ScalingRole:
  Type: AWS::IAM::Role
  Properties:
    RoleName: ScalingRole
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - application-autoscaling.amazonaws.com
          Action:
            - sts:AssumeRole
            
ScalingRolePolicy:
  Type: AWS::IAM::Policy
  Properties:
    Roles:
      - !Ref ScalingRole
    PolicyName: ScalingRolePolicyPolicy
    PolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Effect: Allow
          Resource: '*'
          Action:
            - application-autoscaling:*
            - ecs:RunTask
            - ecs:UpdateSerice
            - ecs:DescribeServices
            - cloudwatch:PutMetricAlarm
            - cloudwatch:DescribeAlarms
            - cloudwatch:GetMetricStatistics
            - cloudwatch:SetAlarmState
            - cloudwatch:DeleteAlarms

Now we're going to have a look at a service definition, its target group for ALB, scaling targets and policies and a CloudWatch alarm. For this example we are going to define ExampleCPUAutoScalingPolicy for a new capacity to grow to a value so that current usage ECSServiceAverageCPUUtilization accounts for 50% and ExampleRequestsAutoScalingPolicy when we have more than 1000 requests per target in a minute.

ExampleTargetGroup:
  Type: AWS::ElasticLoadBalancingV2::TargetGroup
  Properties:
    Port: 80
    Protocol: HTTP
    VpcId: !Ref VpcId
    HealthCheckIntervalSeconds: 30
    HealthCheckPath: /status
    HealthCheckTimeoutSeconds: 15
    HealthyThresholdCount: 2
    UnhealthyThresholdCount: 6
    Matcher:
      HttpCode: 200
    TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: 30
        
ExampleService:
  Type: AWS::ECS::Service
  Properties:
    TaskDefinition: !Ref ExampleTask # omitted
    PlacementStrategies:
      - Field: attribute:ecs.availability-zone
        Type: spread
    DesiredCount: 1
    Cluster: example-cluster # omitted
    LoadBalancers:
      - TargetGroupArn: !Ref ExampleTargetGroup
    ContainerPort: 8080
    ContainerName: example-service

ExampleAutoScalingTarget:
  Type: AWS::ApplicationAutoScaling::ScalableTarget
  Properties:
    MaxCapacity: !Ref MaxServicesCount # parameters
    MinCapacity: !Ref MinServicesCount
    ResourceId:
      Fn::Sub:
        - service/ExampleCluster/${ServiceName}
        - ServiceName: !GetAtt ExampleService.Name
    RoleARN: !GetAtt ScalingRole.Arn
    ScalableDimension: ecs:service:DesiredCount
    ServiceNamespace: ecs
    
ExampleCPUAutoScalingPolicy:
  Type: AWS::ApplicationAutoScaling::ScalingPolicy
  Properties:
    PolicyName: ExampleCPUAutoScalingPolicy
    PolicyType: TargetTrackingScaling
    ScalingTargetId: !Ref ExampleAutoScalingTarget
    TargetTrackingScalingPolicyConfiguration:
      DisableScaleIn: True
      TargetValue: 50
      ScaleInCooldown: 60
      ScaleOutCooldown: 60
      PredefinedMetricSpecification:
        PredefinedMetricType: ECSServiceAverageCPUUtilization
        
ExampleRequestsAutoScalingPolicy:
  Type: AWS::ApplicationAutoScaling::ScalingPolicy
  Properties:
    PolicyName: ExampleRequestsAutoScalingPolicy
    PolicyType: StepScaling
    ScalingTargetId: !Ref ExampleAutoScalingTarget
    ScalableDimension: ecs:service:DesiredCount
    ServiceNamespace: ecs
    StepScalingPolicyConfiguration:
      AdjustmentType: ChangeInCapacity
      Cooldown: 60
      MetricAggregationType: Average
      StepAdjustments:
        - MetricIntervalLowerBound: 0
          ScalingAdjustment: 1
        - MetricIntervalUpperBound: 0
          ScalingAdjustment: -1

ExampleRequestsAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    MetricName: RequestCountPerTarget
    Namespace: AWS/ApplicationELB
    Statistic: Sum
    Period: 60
    EvaluationPeriods: 1
    Threshold: 1000
    ComparisonOperator: GreaterThanOrEqualToThreshold
    AlarmActions:
      - !Ref ExampleRequestsAutoScalingPolicy
    OKActions:
      - !Ref ExampleRequestsAutoScalingPolicy
    Dimensions:
      - Name: TargetGroup
        Value: !GetAtt ExampleTargetGroup.TargetGroupFullName

Notice that parameters section of the ExampleCPUAutoScalingPolicy resource contains DisableScaleIn: true for a specific reason. In order to guarantee that requests per target scaling events have priority over target tracking, the scale in logic of tracking can be disabled completely.

Stability is the key

Ok, so now we have the service scaling up and down based on the number of requests per target in ELB. However, you would notice that the threshold in StepAdjustments for scale up starts right after scale down. It means that your service's desired count would oscillate around some value, going up and down with new tasks spun up. To allow for a window of stability, you need to have a range with ScalingAdjustment: 0, where you would have a boundary to increase & decrease desired count. That way it is possible to make alarm to alert on the scale in boundary, and StepAdjustments to interpret the range. Lets see the example, where we want to scale out on more than RequestsScaleOutThreshold requests per target, and scale in on less than RequestsScaleInThreshold:

ExampleRequestsAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    MetricName: RequestCountPerTarget
    Namespace: AWS/ApplicationELB
    Statistic: Sum
    Period: 60
    EvaluationPeriods: 1
    Threshold: 500 # scale in boundary to trigger the alarm
    ComparisonOperator: GreaterThanOrEqualToThreshold
    AlarmActions:
      - !Ref ExampleRequestsAutoScalingPolicy
    Dimensions:
      - Name: TargetGroup
        Value: !GetAtt ExampleTargetGroup.TargetGroupFullName

ExampleRequestsAutoScalingPolicy:
  Type: AWS::ApplicationAutoScaling::ScalingPolicy
  Properties:
    PolicyName: ExampleRequestsAutoScalingPolicy
    PolicyType: StepScaling
    ScalingTargetId: !Ref ExampleAutoScalingTarget
    ScalableDimension: ecs:service:DesiredCount
    ServiceNamespace: ecs
    StepScalingPolicyConfiguration:
      AdjustmentType: ChangeInCapacity
      Cooldown: 60
      MetricAggregationType: Average
      StepAdjustments:
        - MetricIntervalLowerBound: !Ref RequestsScaleOutThreshold
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: !Ref RequestsScaleInThreshold
          MetricIntervalUpperBound: !Ref RequestsScaleOutThreshold
          ScalingAdjustment: 0
        - MetricIntervalUpperBound: !Ref RequestsScaleInThreshold
          ScalingAdjustment: -1

Here we have a range between MetricIntervalLowerBound=RequestsScaleInThreshold and MetricIntervalUpperBound=RequestsScaleOutThreshold where ScalingAdjustment=0 and no changes are done to desired count. This will ensure that oscillation of desired count does not happen for you.

Further reading

Another approach would be to define to alarms, one to scale out and one to scale in. Each would have a specific range and specific policy associated. Such approach in fact is used quite a lot, but the problem is that CloudWatch alarms are not free, in fact they are pretty expensive.

Additional details are found over the AWS::ApplicationAutoScaling::ScalableTarget and AWS::CloudWatch::Alarm documentation.

Storing and using secret information securely with AWS SSM

The SSM Parameters store is a great way to securely store hierarchical configuration data. It is widely used by AWS to provide parameters it own services. The names of the parameters are guaranteed to be unique and of arbitrary structure, allowing users to store nested structures of configuration data. In this post I will examine how to use Parameters Store efficiently to provide settings and secrets to Docker containers executed on ECS.

The code, examples and additional information for this post is available on the GitHub, and images instrumented with SSM bootstrap are available from the Docker hub.

AWS SSM Parameters Store Values

The SSM allows to use both single and list values for parameters. Optionally the value

can be encrypted with the KMS key. The encryption allows user to keep environmental secrets in SSM, such as service to service authentication, docker registry auth strings, database credentials, X.509 keys and everything else that fits into the 4096 characters. Values that do not fit into 4096 characters, can be stored into the S3 bucket and pre-signed access URL can be written into the SSM parameter instead.

Using the Parameter Store

The main approach to usage of the SSM parameters store is give your ECS task a role to access SSM API and call it on service start. The service itself is responsible for communication with SSM, in a safe and scalable way. This presents a natural problem in case of multiple services, probably written in different languages, each having a unique implementation of SSM nested structures reading code. It is better to decouple configuration and secrets retrieval logic from the application code with a formal contract. Environmental variables are the most common way to pass configuration values to the application code in the Docker containers and it is convenient to use them for SSM stored configuration as well.

Docker images and entrypoints

Application service packaged as Docker image can benefit from the use of intermediate base images which contain bootstrapping code and decouple secrets management from the application itself. The intermediate image can define an entrypoint, used to perform bootstrapping steps, communicate with AWS services are prepare container environment before executing the service code. The image itself can be based on a number of possible runtime environments: NodeJS, Ruby or Python. The entrypoint in docker image receives the `CMD` value of the image using it as a base, or an executable argument of `docker run`, and can invoke it directly after the setup. Below is an example of the intermediate Docker image adding SSM bootstrapping capability to an alpine based image, such as NodeJS's node:alpine.

ARG BASE
FROM $BASE

# Install python runtime for the bootstrap script
RUN apk update && \
  apk add python py-pip py-yaml && \
  pip install awscli && \
  pip install boto3

# Copy bootstrap scripts to image
COPY src/ssm-bootstrap.py /usr/bin/ssm-bootstrap
COPY src/kickstart.sh /usr/bin/kickstart

RUN chmod +x /usr/bin/ssm-bootstrap /usr/bin/kickstart
ENTRYPOINT /usr/bin/kickstart

Here we have an entrypoint defined as /usr/bin/kickstart script. This script will execute /usr/bin/ssm-bootstrap to communicate with SSM and save environment file and other files. Lets examine the contents of the /usr/bin/kickstart:

#!/bin/sh
# query SSM parameters store for secrets and save 
# files and environment variables

ssm-bootstrap --environ /tmp/app_environ --root /app/
[ -f /tmp/app_environ ] && . /tmp/app_environ
exec "$@"

The kickstart script uses /usr/bin/ssm-bootstrap to create /tmp/app_environ file, loads it as environment and passes execution to it's arguments. So that executed process would inherit environment populated with data from the /tmp/app_environ file. Additionally /usr/bin/ssm-bootstrap would write a number of files at root path specified as --root argument. Such files can be various encryption keys, salts and certificates shared across your environments. Using SSM bootstrapping to provide files to Docker containers allows to avoid volume mounts to containers and specialization of your ECS host. Otherwise you have to bake such files into your an AMI or somehow persist on host's filesystem from external storage, so that files could be volume mounted into running containers.

SSM Parameters Namespaces

The actual communications with SSM to retrieve environmental variables is done by the /usr/bin/ssm-bootstrap tool. It is the place where logic to build parameter paths is implemented and shared across all services based on the intermediate Docker image. The interface between infrastructure code and application code, defined as environment variables and (smallish) files, is implemented once and reused everywhere else. Potential structure of the parameter names can be arbitrary, but in general it is useful to have at least 2 nested levels of parameters:

• ECS Cluster - Each service on a specific ECS cluster. The cluster scope corresponds to an instance of portable environment, usually CloudFormation stack(s).
• ECS Service/Container name - Specific service on a specific ECS cluster.

Integration with the ECS

SSM bootstrap is using ECS metadata file determine the cluster and container name it is executed for. It must be enabled in your `ecs.config` on cluster instances. When ECS runs Docker container, it makes metadata file available inside the container. This metadata contains cluster name and container name used by ECS.

Below is an example of the ECS Task Definition in CloudFormation, lets examine what configuration path are available to such service:

ExampleTask:
  Type: AWS::ECS::TaskDefinition
  Properties:
    ContainerDefinitions:
      - Name: example-service
        Image: ...
        Essential: true
        PortMappings:
          - ContainerPort: 8080
        Environment:
          AWS_DEFAULT_REGION: !Ref AWS::Region
    ...

ExampleCluster:
  Type: AWS::ECS::Cluster
  Properties:
    ClusterName: example-cluster

ExampleService:
  Type: AWS::ECS::Service
  Properties:
    TaskDefinition: !Ref ExampleTask
    Cluster: !Ref ExampleCluster
    ...

Notice that parameter names are built using Cluster and Container Name. This information about running container is read from ECS container metadata file, so this needs to be enabled in your ECS agent configuration. The source code for the /usr/bin/ssm-bootstrap utility can be found on the GitHub.

Name structure for nested configurations

With the configuration above the service would have the following paths in SSM injected into it's environment:

• /example-cluster/environment/somevar
• /example-cluster/example-service/environment/anothervar

Also the following paths would be saved as files:

• /example-cluster/files/somefile
• /example-cluster/example-service/files/anotherfile

Task Role and Policy for SSM access

In order to access SSM API the task still needs to have a role allowing that. Below is an example of such role in CloudFormation syntax.

ExampleTaskRole:
  Type: AWS::IAM::Role
  Properties:
    RoleName: ExampleTaskRole
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - ecs-tasks.amazonaws.com
          Action:
            - sts:AssumeRole
      
      
ExamplePolicy:
  Type: AWS::IAM::Policy
  Properties:
    Roles:
      - !Ref ExampleTaskRole
    PolicyName: ExamplePolicy
    PolicyDocument:
      Version: 2012-10-17
      Statement:
        - Action:
            - kms:Decrypt
          Resource: !Sub "arn:aws:kms:${AWS::Region}:alias/my-key"
          Effect: Allow
        - Action:
            - ssm:GetParameter
            - ssm:GetParameters
            - ssm:DescribeParameters
          Resource: "arn:aws:ssm:*"
          Effect: Allow

Using published images

The images with SSM bootstrap middleware layer are published to the Docker Hub and available for general use. Simply specify the base image for your runtime in the FROM statement and your service will automatically receive configuration from the SSM Parameters Store. Below is an example of a generic Dockerfile for NodeJS based service:

FROM stan1y/ssm-bootstrap:node-alpine-latest

WORKDIR /app
COPY src/ /app/src/
COPY .npmrc package.json /app/

RUN npm install && rm -f .npmrc
CMD npm start