Preface

AWS Lambda is event-driven and can execute as much code as needed when needed without an EC2 instance, and it costs money just to launch billing. Unlike EC2, it is a very powerful and attractive service that charges only for what you use completely.

However, I feel that this service is often troublesome and annoying, probably because there are many unique concepts that did not exist in the past, contrary to its lightweight and simple concept.

That's why I'm making a tool called Lamvery because I want to use it more easily and practically. In the part of deployment that I felt was an issue, I was able to create a flow that can be said to be my best practice at the moment, so I would like to introduce it.

Three challenges in deploying Lambda function

① Create a deployment package

Node.js It's relatively easy because you can zip each node_modules with npm install https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/nodejs-create-deployment-pkg.html
Java It feels basic like Maven or Gradle. https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/lambda-java-how-to-create-deployment-package.html
Python A rather disappointing procedure (Chapter Creating a Deployment Package Using a Python Environment Created with Virtualenv) https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

② Version control

--Lambda's special versioning specifications http://dev.classmethod.jp/cloud/aws/lambda-versioning/ --Association of branch operation such as Git with Lambda environment (staging and production, etc.)

③ Development of deployment flow

I haven't seen any best practices yet, or I've rarely seen anything around here. [^ 1]

What can be solved by this method

① Create Python deployment package (Node.js should also work)

First, Lamvery itself has the ability to easily create a deploy package within virtauluenv and deploy it. So all you have to do is create a clean virutalenv environment in CircleCI's Python environment, install and deploy the required libraries in it. You can simply deploy the current directory and below in a zip, so Node.js should be able to do the same if you include package.json in the repository and hit npm install. [^ 2]

(2) Linking with Git branch utilizing Lambda's special versioning specifications

Utilizing the alias attached to the version of Lambda, and taking advantage of the characteristic that the function can be executed by specifying the alias individually, it provides an individual execution environment associated with the branch.

③ Pull Request-based deployment flow

So-called such a guy → http://d.hatena.ne.jp/naoya/20140502/1399027655 It is a story that if you can do ②, you can do it.

Methods and procedures

By the way, here is the actual verification of this content. https://github.com/marcy-terui/lamvery-circleci-deploy

The place to enable the corresponding repository in CircleCI is the same as usual, so I will omit it.

1. IAM settings

Not only the IAM Role assigned to the Lambda function, but also the IAM User set in CircleCI is required.

IAM Role policy assigned to Lambda function

This depends on what you want the function to do, but if you want to transfer confidential information using KMS, which is a unique function of Lamvery, you will have the following privileges.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:kms:<region>:<account-number>:key/<key-id>"
            ]
        }
    ]
}

Replace <region>, <account-number>, <key-id> as appropriate. Click here for how to make a KMS key (issue a key-id) ↓ https://docs.aws.amazon.com/ja_jp/kms/latest/developerguide/create-keys.html

** Make a note of ARN [^ 3] as this Role will be used in subsequent chapters. ** **

IAM User policy to set on CircleCI

I don't think there is much change here. If it is troublesome to set one by one, lambda: * allows all resources (" Resource ":" * "), but in some cases it may be ant, but ʻiam: PassRole` allows all resources It's very dangerous, so you should stop it. [^ 4]

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:*",
                "iam:PassRole"
            ],
            "Resource": [
                "arn:aws:lambda:<region>:<account-number>:function:<function-name>",
                "arn:aws:lambda:<region>:<account-number>:function:<function-name>:*",
                "<function-role-arn>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "lambda:CreateFunction",
                "lambda:ListFunctions",
                "lambda:ListVersionsByFunction"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Replace <region>, <account-number>, <function-name> as appropriate. The <function-role-arn> contains the Role ARN [^ 3] created earlier.

** Make a note of this User Credential as we will use it in the next chapter. ** **

Get IAM User Credential and set it to CircleCI

Open Project Settings
Select AWS Permission
Enter the Credential

2. Lamvery installation and configuration

Install Lamvery and generate a config file

$ pip install lamvery
$ lamvery init
lamvery: Output initial file: .lamvery.yml
lamvery: Output initial file: .lamvery.exclude.yml
lamvery: Output initial file: .lamvery.event.yml
lamvery: Output initial file: .lamvery.secret.yml

Edit the configuration file

Replace each setting below region and configuration as appropriate. If the part of {{env ['AWS_LAMBDA_ROLE']}} is a Private repository, I think it's okay if the ARN of Role is in the repository. If you want to pass from environment variables in the same way, you can set it on CircleCI by selecting "Project Settings" → "Environment Variables".

`yaml:.lamvery.yml`


profile: null
region: us-east-1
versioning: false
default_alias: master
configuration:
  name: lamvery-deploy-sample
  runtime: python2.7
  role: {{ env['AWS_LAMBDA_ROLE'] }}
  handler: lambda_function.lambda_handler
  description: This is a sample lambda function.
  timeout: 10
  memory_size: 128

Setting points

versioning: false
By default, Lambda versioning is disabled and enabled with command options as needed.
default_alias: master
An alias to be added at deployment time if not specified in the command option. It doesn't have to be master, but avoid aliases for production (described later) to prevent operation mistakes.

Implement Function and list required libraries

The implementation of function is just Python coding, so I will omit it. In the above example, handler is lambda_function.lambda_handler, so the file will be as follows.

`lambda_function.py`


import lamvery


def lambda_handler(event, context):
    print(lamvery.secret.get('foo'))

In Python projects, I often write it in requirements.txt, so this time I do it. It is recommended to install the necessary libraries in virtualenv and write them to a file as shown below.

pip install flake8
pip freeze > requirements.txt

This will allow you to install the same library by running the following on CircleCI:

pip install -r requirements.txt

Describe CircleCI settings

Describe as follows.

`circle.yml`


---
machine:
  python:
    version: 2.7

dependencies:
  pre:
    - |
      virtualenv .venv
      source .venv/bin/activate
      pip install -r requirements.txt

test:
  override:
    - |
      source .venv/bin/activate
      flake8 lambda_function.py

deployment:
  master-head:
    branch: master
    commands:
      - |
        source .venv/bin/activate
        lamvery deploy
  staging:
    branch: staging
    commands:
      - |
        source .venv/bin/activate
        lamvery deploy -a staging -p
  production:
    branch: production
    commands:
      - |
        source .venv/bin/activate
        lamvery deploy -a production -p

Setting points

--Create a clean virtauluenv environment with dependencies.pre and install the required libraries in it Lamvery is also included in this (If you do not pass confidential information, you do not need Lamvery in the Function itself, so you can install it separately) --No command options when deploying in the master branch With no options, it is versioning: false, default_alias: master, so the special version [^ 5] that is always used when versioning $ LATEST is disabled has the alias master. It will be attached. As a result, ** Lambda has a capacity limit, so updating the master branch will always update the new version. I don't want it to be published, but HEAD can always be runnable **. --staging and production explicitly add an option to enable alias names and versioning You can specify the alias name with -a and enable versioning with -p.

Let's deploy! Make a pull request and merge it.

Let's see the deployment result.

If you look at the colored logs, you can see that a new version of 4 has been published and has been renamed to production.

By the way, I think that the alias production-pre is set at the same time, but this is for rollback, and you can rollback in an emergency as follows. You can hit it at hand, or you can hit it with a ChatBot.

$ lamvery rollback -a production
lamvery: [Function] Previous version: 2
lamvery: [Alias] production: 4 -> 2

About execution of deployed function

When executing from various events, please note that each alias has an individual ARN [^ 3]. It will be specified as follows. In the case of API, the alias may be specified by the parameter Qualifier. arn:aws:lambda:<region>:<account-number>:function:<function-name>:<alias>

Summary

One of the purposes of creating a tool called Lamvery was to create a Lambda function deployment flow that I thought was "this!" So I introduced it. There is no big difference from the method used in EC2 etc. so far, and I am conscious of the flow that can be used without discomfort, but if you have any opinions such as it is better to do this, I would be grateful if you could give us feedback from the following.

https://github.com/marcy-terui/lamvery

TODO: Creating a CloudFormation template that automates the front of CircleCI

[^ 1]: The only thing I found was this. It's done very well, but this time it's more focused on deployment and in a different direction. [^ 2]: Node.js has not been verified, so I'd be happy if you could try it and report the results (Java ...?) [^ 3]: Abbreviation for Amazon Resource Name [^ 4]: You can assign any Role → You can specify it if you have a Role with Admin or Power User authority → You can do whatever you want via Lambda [^ 5]: It's a difficult expression, but I can't think of anything else. .. ..

[PYTHON] Lambda function deploy best practices with CircleCI + Lamvery

Preface

Three challenges in deploying Lambda function

① Create a deployment package

② Version control

③ Development of deployment flow

What can be solved by this method

① Create Python deployment package (Node.js should also work)

(2) Linking with Git branch utilizing Lambda's special versioning specifications

③ Pull Request-based deployment flow

Methods and procedures

1. IAM settings

IAM Role policy assigned to Lambda function

IAM User policy to set on CircleCI

Get IAM User Credential and set it to CircleCI

2. Lamvery installation and configuration

Install Lamvery and generate a config file

Edit the configuration file

yaml:.lamvery.yml

Implement Function and list required libraries

lambda_function.py

Describe CircleCI settings

circle.yml

Setting points

About execution of deployed function

Summary

`yaml:.lamvery.yml`

`lambda_function.py`

`circle.yml`