I tried AWS Copilot CLI which can build ECS at high speed

Hi, this is [DMM WEBCAMP AdventCalender] will be responsible for day 20 (https://qiita.com/advent-calendar/2020/infratop), likes AWS and Docker @tkt_horikoshi.

This time, I tried AWS Copilot, which can build ECS on a command line basis, so I would like to share the usability.

Introduction

It's only a short time since I started as an engineer mentor at DMM WEB CAMP, but it seems that some of the students are preparing the development environment with Docker.

If you are developing with Docker, you want to be able to execute applications on a container basis without putting files directly on EC2. But I don't know what to start with.

I hope it will be an opportunity for those who have such thoughts to take a step forward.

What is ECS

ECS is an abbreviation for Amazon Elastic Container Service, which is a service that deploys Dockernized applications and provides an execution environment. You can choose between EC2 and Fargate as your computing resources, but in particular Fargate (https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/AWS_Fargate.html) is a container computing engine that eliminates provisioning and maintenance costs.

It's a very attractive service, and I'm indebted to it in my main business, but it's also true that building from scratch costs a fair amount.

Touch AWS Copilot

AWS Copilot allows you to quickly deploy container applications to ECS on a command line basis. It's hard to believe that you can make ECS from the command line, but I'll do it.

Basic terminology

We have summarized the concept that you should understand when using Copilot. You can use Copilot to create each of the following from the command line:

In this article, I will mainly touch on Application, Service, and Environment.

Today's goal

I would like to use Copilot to deploy my application to ECS and display the Rails welcome screen.

Application preparation

Rails preparation

Dockerfile

First, set up your application to work with Docker. Place the Dockerfile in the root directory.

FROM ruby:2.7

RUN mkdir /app
WORKDIR /app

RUN curl https://deb.nodesource.com/setup_12.x | bash
RUN curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list

RUN apt-get update -qq && \
    apt-get install -y nodejs yarn

COPY Gemfile* ./
RUN bundle install
COPY . .

COPY entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/entrypoint.sh
ENTRYPOINT ["entrypoint.sh"]

EXPOSE 3000
CMD ["rails", "server", "-b", "0.0.0.0"]

Specifying config.hosts

Add config.hosts to config/environments/development.rb, config/environments/production.rb to specify the domain for domain permission.

This time, we want to allow subdomains for both the test environment and the production environment, so specify as follows.

Rails.application.configure do
...

  config.hosts << ".takat0-h0rikosh1.com"

...
end

Nginx preparation

Set Nginx to work with Docker as well. Place various configuration files in the root directory by cutting the directory called nginx.

For the convenience of being able to switch $ {SERVER_HOST} with an environment variable, we have prepared a template for the configuration file as shown below.

nginx:nginx/nginx.conf.template

upstream app {
  server ${SERVER_HOST}:3000;
}

server {
  listen 80;
  server_name localhost;

  root /app/public;

  client_max_body_size 100m;
  error_page 404             /404.html;
  error_page 505 502 503 504 /500.html;
  try_files  $uri/index.html $uri @app;
  keepalive_timeout 5;

  location / {
    proxy_pass http://app;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_redirect off;
    proxy_connect_timeout 30;
  }
}

Next is the Dockerfile. Use envsubst in CMD to replace variables and output to /etc/nginx/conf.d/nginx.conf. The default for SERVER_HOST is localhost.

nginx/Dockerfile

FROM nginx:alpine

RUN rm -f /etc/nginx/conf.d/*

ADD nginx.conf.template /etc/nginx/conf.d/nginx.conf.template

ENV SERVER_HOST localhost

CMD envsubst \
    '$$SERVER_HOST' \
    < /etc/nginx/conf.d/nginx.conf.template > \
    /etc/nginx/conf.d/nginx.conf \
    && nginx -g 'daemon off;'

Operation check

Create docker-compose.yml and check the operation.

docker-compose.yml

version: "3.8"

services:
  app:
    build: .
    volumes:
      - .:/app
    ports:
      - "3000:3000"
    environment:
      RAILS_ENV: development

  proxy:
    build:
      context: nginx
    ports:
      - 80:80
    depends_on:
      - app
    environment:
      SERVER_HOST: host.docker.internal

Let's start the container

$ docker-compose up
Recreating copilot-playground_app_1 ... done
Recreating copilot-playground_proxy_1 ... done
Attaching to copilot-playground_app_1, copilot-playground_proxy_1
app_1    | => Booting Puma
app_1    | => Rails 6.0.3.4 application starting in development 
app_1    | => Run `rails server --help` for more startup options
app_1    | Puma starting in single mode...
app_1    | * Version 4.3.7 (ruby 2.7.2-p137), codename: Mysterious Traveller
app_1    | * Min threads: 5, max threads: 5
app_1    | * Environment: development
app_1    | * Listening on tcp://0.0.0.0:3000
app_1    | Use Ctrl-C to stop

For the time being, I was able to confirm the operation at hand.

Try deploying with Copilot

Install

$ brew install aws/tap/copilot-cli

Reference: https://github.com/aws/copilot-cli

Create an Application

Create a ** Application ** with your own domain. Name ** Application ** welcome. Run copilot app init in the root directory.

$ copilot app init welcome --domain takat0-h0rikosh1.com


✔ Created the infrastructure to manage services and jobs under application welcome.

✔ The directory copilot will hold service manifests for application welcome.

Recommended follow-up actions:
- Run `copilot init` to add a new service or job to your application.

Somehow I can proceed to run copilot init.

Service creation and provision deployment of test environment

If you execute copilot init as prompted by the execution log, you can proceed with the ** Service ** settings step by step.

$ copilot init

Welcome to the Copilot CLI! We're going to walk you through some questions
to help you get set up with an application on ECS. An application is a collection of
containerized services that operate together.

First, you will be asked about the ** Service ** workload type, so select Load Balanced Web Service here. I was asked what to do with the name, so I chose ** app **.

  Which workload type best represents your architecture?  [Use arrows to move, type to filter, ? for more help]
  > Load Balanced Web Service
    Backend Service
    Scheduled Job

What do you want to name this Load Balanced Web Service? [? for help]
app

Then you will be asked to select a Dockerfile. Here, select Rails Dockerfile.

for welcome?  [Use arrows to move, type to filter, ? for more help]
  > ./Dockerfile
    nginx/Dockerfile
    Enter custom path for your Dockerfile
    Use an existing image instead

Finally, type y when asked if you want to deploy to the test environment.

Would you like to deploy a test environment? [? for help](y/N)
y

When the interaction with the CLI is completed so far, provisioning starts with great momentum, and the deployment is completed in a few minutes. Since the URL is displayed at the end of the execution log, try displaying it with a browser.

Oh, it came out.

Just in case, let's go into the AWS Management Console and take a quick look at the contents.

--ECS cluster - --ECS service

• ALB -

How about the network configuration?

• VPC - --Subnet - --Security group -

It seems that they have prepared the entire VPC with the particle size of ** Environment **.

It's amazing, I've reached the point where I can run an application on ECS, including DNS settings, just on the command line.

Move Nginx with Sidecar pattern

As some of you may have already noticed, at this point the browser access goes to the Rails server through ALB.

So, let's run the Nginx container to the ECS task where the Rails server container is running so that we can proxy ALB traffic.

The approach of moving a container with an auxiliary role to a container within an ECS task is called the Sidecar pattern (https://aws.amazon.com/jp/blogs/compute/nginx-reverse-proxy-sidecar-container-on-amazon-ecs/).

Push to ECR

Push the Nginx Docker image to ECR. https://docs.aws.amazon.com/ja_jp/AmazonECR/latest/userguide/getting-started-cli.html

#Repository creation
$ aws ecr create-repository --repository-name welcome/proxy

#Image push
$ aws ecr get-login-password --region ap-northeast-1 | docker login --username AWS --password-stdin 593653836732.dkr.ecr.ap-northeast-1.amazonaws.com && \
  docker build -t welcome/proxy nginx && \
  docker tag welcome/proxy:latest 593653836732.dkr.ecr.ap-northeast-1.amazonaws.com/welcome/proxy:latest && \
  docker push 593653836732.dkr.ecr.ap-northeast-1.amazonaws.com/welcome/proxy:latest

Manifest file fix

When copilot init is executed, a manifest file containing the setting information is created, so modify it as follows.

copilot/app/manifest.yml

name: app
type: Load Balanced Web Service

image:
  build: ./Dockerfile
  port: 3000

http:
  path: 'app'
  #add to
  healthcheck: '/'
  targetContainer: 'proxy'

cpu: 256
memory: 512
count: 1

#add to
sidecars:
  proxy:
    port: 80
    image: 593653836732.dkr.ecr.ap-northeast-1.amazonaws.com/welcom/proxy:latest

Deploy

Run the copilot svc deploy command to deploy ** Service **.

$ copilot svc deploy

As shown below, we have confirmed that the app and proxy containers are running for the ECS task.

It was confirmed that load balancing was performed to the added proxy container.

At this point, you can now proxy with Nginx.

Deploy Production environment

Now, I would like to try deploying a production environment.

Setting environment variables

This time I just wanted to display the welcome screen, so I actually omitted it, but I think that it is necessary to set production in RAILS_ENV when deploying in the Production environment, but in that case, put the following in the manifest file Append.

copilot/app/manifest.yml

#abridgement

environments:
  production:
    variables:
      RAILS_ENV: production

Reference: https://aws.github.io/copilot-cli/docs/developing/environment-variables/

Environment creation

Run the copilot env init command to create the ** Environment ** with --prod. I named it production.

$ copilot env init \
  --name production \
  --default-config \
  --prod

At this point, you can see that the VPC and ECS Cluster have been created.

• VPC -
• ECS Cluster

As you can see, the service has not been executed yet.

Service deployment

Let's try running copilot svc deploy for production.

$ copilot svc deploy --env production

#abridgement

✔ Deployed app, you can access it at https://app.production.welcome.takat0-h0rikosh1.com.

Oh, it came out.

Deployment of the Productioin environment is completed with two commands.

Clean up

All the resources included in ** Application ** created by Copilot will be deleted cleanly with the following command. If you don't need it, erase it before you are charged for it.

$ copilot app delete

Finally

I tried to build ECS from the command line using AWS Copilot. It's a great time to be able to be crisper than I expected. In particular, it is amazing that you can take it to the point of hosting applications without being aware of the network configuration. If you can master it, your productivity will increase considerably.

You can try it in no time, and you can create and delete it freely from the command line, so it is highly recommended for those who want to Dockernize the application but feel heavy.

If you are interested, please try it.

I will put the source code of this time here. https://github.com/takat0-h0rikosh1/dmm-webcamp-advent-calender-2020

reference

• https://aws.github.io/copilot-cli/
• https://github.com/aws/copilot-cli
• https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/Welcome.html
• https://docs.aws.amazon.com/ja_jp/AmazonECR/latest/userguide/what-is-ecr.html
• https://qiita.com/minamijoyo/items/63ae57b99d4a4c5d7987
• https://qiita.com/kodai_0122/items/67c6d390f18698950440
• https://qiita.com/eighty8/items/0288ab9c127ddb683315