This is I'll do it this year too! AWS Lambda tied up Advent Calendar 2015 16th article, but updated from time to time (last updated: 2016-07-15)

Introduction

All information provided is as of 2015-07-15. Development is ongoing and is subject to change.

What is Lamvery?

Sauce because it's okay

It will be here https://github.com/marcy-terui/lamvery

Overview

It is a tool that supports the deployment of Lambda function and the setting and management of peripheral functions including the Function itself. It is made of Python and I mainly use it in Python, but Node.js is also supported for the time being, and I have confirmed the minimum operation.

Based on the concept of ʻUser-friendly deploy and management tool for AWS Lambda function`, we are developing with a focus on how to handle one Function easily and conveniently and the probability of a practical deployment flow. It is positioned as a tool.

Also, I think that it is valuable to add additional functions that can not be realized without hooking to the deploy of Function, and it is a feature that such tools are actively added.

How to read

The author reads "Ranberi".

Etymology

Originally [How to deploy Python code written in AWS documentation](https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/lambda-python-how-to-create-deployment-package .html) is too troublesome and I thought I could do something about it, so I made "** Lam ** bda + ** v ** irtual ** e ** nv + p ** y ** thon" It feels like I picked it up and connected it to [^ 1], but now I can't help but think that the short and cool name was good.

Why compare with other similar tools

lambda-uploader, Kappa, etc.

--The configuration file is JSON (YAML is better) --Unclear separation between configuration file and command line options --The path of the virtualenv library is solid, the storage efficiency is poor, and the deployment package (zip) becomes bloated --There is no distinction between .py and .pyc, and one of them is included in the archive [^ 3]

Apex Please check here. In particular, you should be careful about the rollback specifications because you can see at a glance that "Oh, this is a bad guy". http://qiita.com/marcy-terui/items/db8dae512af3c553fe72

Why did you make Lamvery without using the above?

--I wanted an easy-to-use configuration file --The configuration file is interpreted as YAML and jinja2 template, so you can embed code and variables. --Many tools do not have practical parameters and option designs --Although it is a private standard, the configuration file and command line options are separated by the following policy. --Configuration file = what should be version controlled --Command option = Temporary use --Probability of practical deployment flow - http://qiita.com/marcy-terui/items/900b72efb38f9b26e8f0 --Create a deployment package that meets the Python and virtualenv specifications --. Py is not stored in the archive, only .pyc is stored, etc. [^ 4]

I wanted a tool that fits in my hand and can be used practically, and as I used it, I seemed to want various things, so I thought I should make it myself.

Rough list of features and things you can do

--Setting description by YAML + Jinja2 template --Creating a package with excellent storage efficiency (Python only) --Secure deployment and rollback --Deployment hook --Passing environment variables --Transfer of confidential information using KMS --Delivery of confidential files using KMS --API deployment and management on API Gateway --Setting up and managing scheduled execution with CloudWatch Events --CloudWatch Logs Log Viewing

How to use

Installation

PyPI
Since archiving becomes inefficient and the directory becomes dirty, it is recommended to expand the virtualenv virtual environment in a subdirectory instead of the directory itself where the Function itself is placed.

virtualenv -p <path-to-python2.7> .venv
. .venv/bin/activate
pip install lamvery

echo "deb https://dl.bintray.com/willyworks/deb trusty main" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install lamvery
export PATH=/opt/lamvery/bin:$PATH

echo "
[bintraybintray-willyworks-rpm]
name=bintray-willyworks-rpm
baseurl=https://dl.bintray.com/willyworks/rpm/centos/\$releaserver/\$basearch/
gpgcheck=0
enabled=1
" | sudo tee -a /etc/yum.repos.d/bintray-willyworks-rpm.repo
sudo yum install lamvery
export PATH=/opt/lamvery/bin:$PATH

Setting method

First, create a template of the configuration file with the following command.

lamvery init

Now that you have a template for some configuration files, edit it. As mentioned above, it is interpreted as jinja2 template, so if you enclose it in {{}}, it will be variable expansion, and if you enclose it in {%%}, it will be expanded. You can embed code. Also, I put the environment variables that I think are often used in a variable called ʻenv` so that I can write them concisely.

The following is a setting example.

basic configuration

`yaml:.lamvery.yml`


profile: private
region: us-east-1
versioning: true
default_alias: test
clean_build: false
configuration:
  name: lamvery-test
  runtime: python2.7
  role: {{ env['AWS_LAMBDA_ROLE'] }}
  handler: lambda_function.lambda_handler
  description: This is sample lambda function.
  timeout: 10
  memory_size: 128
  vpc_config:
    subnets:
    - subnet-cadf2993
    security_groups:
    - sg-4d095028

Event (CloudWatch Events) settings

`yaml:.lamvery.event.yml`


rules:
- name: foo
  description: bar
  schedule: 'rate(5 minutes)'
  targets:
  - id: test-target-id
    input:
      this:
      - is: a
      - sample: input

Confidential Information (KMS) Settings

`yaml:.lamvery.secret.yml`


key_id: xxxx-yyyy-zzzz
cipher_texts: {}
secret_files: {}

Exclusion list

`yaml:.lamvery.exclude.yml`


- ^\.lamvery\.yml$
- ^\.lamvery\.event\.yml$
- ^\.lamvery\.secret\.yml$
- ^\.lamvery\.exclude\.yml$

Action hook

`yaml:.lamvery.hook.yml`


build:
  pre:
  - pip install -r requirements.txt -t ./
  post: []

API Gateway

`yaml:.lamvery.api.yml`


api_id: myipugal74
stage: dev
cors:
  origin: '*'
  methods:
  - GET
  - OPTION
  headers:
  - Content-Type
  - X-Amz-Date
  - Authorization
  - X-Api-Key
configuration:
  swagger: '2.0'
  info:
    title: Sample API
  schemes:
  - https
  paths:
    /:
      get:
        produces:
        - application/json
        parameters:
        - name: sample
          in: query
          required: false
          type: string
        responses:
          '200':
            description: 200 response
            schema:
              $ref: '#/definitions/Sample'
  definitions:
    Sample:
      type: object

Setting items

basic configuration

profile
ʻAws configure --profile This is the part of hoge set by hoge. This is the one that contains Credential information and Region settings. http://docs.aws.amazon.com/ja_jp/cli/latest/userguide/cli-chap-getting-started.html If not specified (write null or delete each item), it follows the behavior of ʻAWS SDK (boto3) (IAM Role on EC2, environment variables such as ʻAWS_SECRET_KEY` on local)
region
That manma region name
versioning
Versioning enabled (true), disabled (false)
default_alias
Alias if not specified as an option
clean_build
When creating the deployment package, move the source code to a temporary directory and then create the package. You don't have to pollute your working directory if you want to hook into a build to install and deploy libraries etc. (hooks will be discussed later)
configuration
Basically, the Lambda setting items are the same.
- name
  The name of the Function. Since it is used to identify the Function in the API, it cannot be duplicated.
- runtime
  Function runtime. Currently it supports python2.7 or nodejs.
- role
  ARN of IAM Role given when Lambda is executed.
- handler
  The namespace of the function that is the entry point for the Function.
- description
  A description of the Function.
- timeoute Maximum execution time of Function (seconds)
- memory_size
  Memory size (MB) given to Function (container)
- vpc_config
  Settings for running in a VPC Specify the subnet ID in subnets and the security group ID in security_groups in the list.

Event (CloudWatch Events) settings

name
Unique rule name for Event
description
Explanation of the rule
targets
Event target (s)
- id
  Unique ID of the target
- input
  Enter the input to be passed to the target (Function's ʻevent` argument)
- input_path
  Cannot be used with ʻinput. Used when you want to handle only any part of the arguments passed by default when ʻinput is not specified in JSONPath format.

Settings for confidential information

key_id
Private key ID of KMS (only the ID part, not ARN)
cipher_texts
Contains a ciphertext in which a specific string is encrypted by KMS. Since it is basically set from a command, it is unlikely that this item will be written manually. The information entered here is passed to the Function, and confidential information can be safely passed (details will be described later).
secret_files
Contains the ciphertext encrypted by KMS for the contents of a particular file. Since it is basically set from a command, it is unlikely that this item will be written manually. The information entered here is passed to the Function, and the file containing confidential information can be safely passed (details will be described later).

Exclusion list

Just enumerate the paths of the files you want to exclude with regular expressions

Action hook

Currently only build is supported. build is the process of creating a deploy package. List the commands you want to execute before pre and after post.

API Gateway

api_id
API ID (rest-api-id) in API Gateway. If you add the -w option when creating the API for the first time with the ʻapi` command, it will be filled in automatically.
stage
The stage name to deploy the API. ʻApi` Used when not specified as an option when executing the command. https://docs.aws.amazon.com/ja_jp/apigateway/latest/developerguide/stages.html
cors
Specify the CORS (Cross-Origin Resource Sharing) option that you want to automatically grant to each resource in the API.
- origin
  Options to specify for ʻAccess-Control-Allow-Origin`
- headers
  Options to specify for ʻAccess-Control-Allow-Headers`
- methods
  Options to specify in ʻAccess-Control-Allow-Methods`
configuration
Describe the API settings in Swagger format. The contents of x-amazon-apigateway-integration required for executing Lambda functions and CORS settings are automatically added unless explicitly stated in the configuration file. Mapping Template also automatically sets Pass all parameters (this Behavior can be suppressed as an option)

command

Only frequently used command line arguments are listed. If you want to know everything, please check README.

init

$ lamvery init -h
usage: lamvery init [-h] [-c CONF_FILE]

optional arguments:
  -h, --help            show this help message and exit
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)

Create a template for various configuration files This is the first command to be executed after installation.

generate

$ lamvery generate -h
usage: lamvery generate [-h] [-c CONF_FILE] -k KIND

optional arguments:
  -h, --help            show this help message and exit
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -k KIND, --kind KIND  The kind of the file # accepts "function"

This command generates templates for various programs. The type of file to be generated is specified by -k, but at present, only the Function template can be generated by function.

build

$ lamvery build -h
usage: lamvery build [-h] [-c CONF_FILE] [-s] [-l] [-e ENV]

optional arguments:
  -h, --help            show this help message and exit
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -s, --single-file     Only use the main lambda function file
  -l, --no-libs         Archiving without all libraries
  -e ENV, --env ENV     Environment variables that pass to the function

Create an archive (ZIP) file of your source code and libraries. For confirmation or when creating an irregular flow. The file name will be Function name.zip. If there is no configuration file, it is execution directory name.zip.

deploy

$ lamvery deploy -h
usage: lamvery deploy [-h] [-a ALIAS] [-c CONF_FILE] [-d] [-s] [-l] [-p]
                      [-e ENV]

optional arguments:
  -h, --help            show this help message and exit
  -a ALIAS, --alias ALIAS
                        Alias for a version of the function
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -d, --dry-run         Dry run
  -s, --single-file     Only use the main lambda function file
  -l, --no-libs         Archiving without all libraries
  -p, --publish         Publish the version as an atomic operation
  -e ENV, --env ENV     Environment variables that pass to the function

Do all of the following together.

--Create source code and library archives --Embed confidential information ciphertext in the archive --Embed environment variables specified as options in the archive --Updated Lambda execution settings --Upload archive --Give another alias to the previous version (if specified) --Alias the new version (if specified)

In addition, I added the Dry run (display only the difference without updating) option that is useful in codenize.tools that I am indebted to. It is -d or --dry-run.

You can specify environment variables with -e or --env.

rollback

$ lamvery rollback -h
usage: lamvery rollback [-h] [-a ALIAS] [-c CONF_FILE] [-v VERSION]

optional arguments:
  -h, --help            show this help message and exit
  -a ALIAS, --alias ALIAS
                        Alias for a version of the function
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -v VERSION, --version VERSION
                        Version of the function

Rewinds deploy. It also supports Dry run. The Rollback specifications are described in here.

configure

$ lamvery configure -h
usage: lamvery configure [-h] [-c CONF_FILE] [-d]

optional arguments:
  -h, --help            show this help message and exit
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -d, --dry-run         Dry run

The source code is not updated, only the Function settings are updated. Of course, it supports Dry run. However, please note that if you are versioning, the settings will not be reflected until you publish a new version. Since the settings are synchronized even when deploying, this command can only be used when there is almost no versioning.

set-alias

$ lamvery set-alias -h
usage: lamvery set-alias [-h] [-a ALIAS] [-c CONF_FILE] [-d] [-v VERSION]
                         [-t TARGET]

optional arguments:
  -h, --help            show this help message and exit
  -a ALIAS, --alias ALIAS
                        Alias for a version of the function
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -d, --dry-run         Dry run
  -v VERSION, --version VERSION
                        Version of the function
  -t TARGET, --target TARGET
                        The alias of the version that is targeted for setting
                        alias

Set an alias. This is also compatible with Dry run. Specify the alias name with -a or --alias, and specify the version to be aliased with -v or --version.

encrypt

$ lamvery encrypt -h
usage: lamvery encrypt [-h] [-c CONF_FILE] [-n SECRET_NAME] [-s] text

positional arguments:
  text                  The text value to encrypt

optional arguments:
  -h, --help            show this help message and exit
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -n SECRET_NAME, --secret-name SECRET_NAME
                        The name of the secret value
  -s, --store           Store encripted value to the configuration file
                        (default: .lamvery.secret.yml)

Encrypt the given value using KMS. Specify the name to be used when retrieving with -n or --name, and if you add -s or --store, the ciphertext will be registered in the configuration file with that name.

encrypt-file

$ lamvery encrypt-file -h
usage: lamvery encrypt-file [-h] [-c CONF_FILE] -p PATH [-s] file

positional arguments:
  file                  The file path to encrypt

optional arguments:
  -h, --help            show this help message and exit
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -p PATH, --path PATH  The path to put the decrypted file in the function
  -s, --store           Store encripted value to the configuration file
                        (default: .lamvery.secret.yml)

Encrypt the specified file using KMS. Specify the file name to be used when handling with Function with -n or --name, and if you add -s or --store, the ciphertext and file name will be registered in the configuration file with that name. Will be done.

decrypt

$ lamvery decrypt -h
usage: lamvery decrypt [-h] [-c CONF_FILE] [-n SECRET_NAME]

optional arguments:
  -h, --help            show this help message and exit
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -n SECRET_NAME, --secret-name SECRET_NAME
                        The name of the secret value

Decrypts the ciphertext in the configuration file. Specify the name decided at the time of encryption with -n or --name. decrypt-file is not implemented because it is a story if you look at the original file.

events Set up CloudWatch Events. Of course, it supports Dry run. It is a differential update, create it if there is no Event setting, and update it if there is one. Events that are not described in the Event associated with the Function will be deleted, but those that are associated with another Function will only be removed from the Target and will not be deleted.

lamvery events [-k]

invoke

$ lamvery invoke -h
usage: lamvery invoke [-h] [-a ALIAS] [-c CONF_FILE] [-v VERSION] json

positional arguments:
  json                  The JSON string or file that pass to the function

optional arguments:
  -h, --help            show this help message and exit
  -a ALIAS, --alias ALIAS
                        Alias for a version of the function
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -v VERSION, --version VERSION
                        Version of the function

Start Function. The resulting Log will be output to standard output. Specify the input with an argument. If you pass it in JSON format, it will be passed as it is, and if you specify the path of the file in which JSON is written, the contents will be passed. You can also specify the version (-v) and alias ( -a).

logs

$ lamvery logs -h
usage: lamvery logs [-h] [-c CONF_FILE] [-f] [-F FILTER] [-i INTERVAL]
                    [-s START]

optional arguments:
  -h, --help            show this help message and exit
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -f, --follow          Watch the log events and updates the display (like
                        `tail -f`)
  -F FILTER, --filter FILTER
                        Filtering pattern for the log messages
  -i INTERVAL, --interval INTERVAL
                        Intervals(seconds) to watch the log events
  -s START, --start START
                        Time to start the log events watching

Browse the function logs that are spit out in CloudWatch Logs. You can use -f to keep playing tail -f, or use -s to specify the start date and time for browsing.

api

$ lamvery api -h
usage: lamvery api [-h] [-c CONF_FILE] [-d] [-n] [-r] [-s STAGE] [-w]

optional arguments:
  -h, --help            show this help message and exit
  -c CONF_FILE, --conf-file CONF_FILE
                        Configuration YAML file (default: .lamvery.yml)
  -d, --dry-run         Dry run
  -n, --no-integrate    Without automatic integration
  -r, --remove          Remove your API
  -s STAGE, --stage STAGE
                        The name of the stage in API Gateway
  -w, --write-id        Write the id of your API to the configuration file
                        (default: .lamvery.api.yml)

Deploy the API to API Gateway. Add -r to remove it. You can suppress the behavior of automatically setting x-amazon-apigateway-integration etc. with -n and deploy as it is written in the configuration file.

Flow of creating Lambda function that uses confidential information (reference)

1. Create a private key with KMS.

https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html

2. Add the following permissions to the IAM Role when executing the Function.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:kms:us-east-1:<your-account-number>:key/<your-key-id>"
            ]
        }
    ]
}

3. Generate a configuration file and Function template

lamvery init
lamvery generate -k function

4. Set the ID of the key created in 1 in the configuration file as shown below.

`yaml:.lamvery.yml`


 profile: default
  region: us-east-1
  configuration:
    name: sample_lambda_function
    runtime: python2.7 # or nodejs
    role: arn:aws:iam::000000000000:role/lambda_basic_execution
    handler: lambda_function.lambda_handler
    description: This is sample lambda function.
    timeout: 10
    memory_size: 128

`yaml:.lamvery.secret.yml`


key_id: a5ed61a9-fa57-4ebf-9b3f-457b95de05ce # <-here! !! !!
cipher_texts: {}

5. Encrypt and store sensitive information

lamvery encrypt -s -n foo "This is a secret"

6. Write the source

`lambda_function.py`


import lamvery

def lambda_handler(event, context):
    print(lamvery.secret.get('foo'))

`lambda_function.js`


var lamvery = require('./lamvery.js');

exports.lambda_handler = function(event, context) {
    lamvery.secret.get('foo', function(err, data) {
        console.log(data);
    });
}

Deploy!!

lamvery deploy

8. Launch Function

lambery invoke {}

You will get the decrypted result as below.

START RequestId: 13829c9c-9f13-11e5-921b-6f048cff3c2d Version: $LATEST
This is a secret
END RequestId: 13829c9c-9f13-11e5-921b-6f048cff3c2d

Finally

It's got a lot of features, but we're planning to add more! If you like, please try it and give us your feedback :-)

Click here for feedback ↓ https://github.com/marcy-terui/lamvery

[^ 1]: A library that provides an isolated virtual environment for Python libraries, etc. https://pypi.python.org/pypi/virtualenv [^ 2]: As of December 16, 2015. There is no API (though it may be internal) or AWS has the saying "Talk with API". https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/getting-started-scheduled-events.html [^ 3]: .py is the raw source, .pyc is the compiled source [^ 4]: I don't have a bench, so it's a shame, but since the compilation is shortened, the startup time is very short, but may it improve? [^ 5]: If you don't use virtualenv, the source that happened to work in the system library will not work if it is local, so there is no option not to use it, right?

[PYTHON] About "Lamvery", a deployment and management tool for AWS Lambda

Introduction

What is Lamvery?

Sauce because it's okay

Overview

How to read

Etymology

Why compare with other similar tools

lambda-uploader, Kappa, etc.

Why did you make Lamvery without using the above?

Rough list of features and things you can do

How to use

Installation

Setting method

basic configuration

yaml:.lamvery.yml

Event (CloudWatch Events) settings

yaml:.lamvery.event.yml

Confidential Information (KMS) Settings

yaml:.lamvery.secret.yml

Exclusion list

yaml:.lamvery.exclude.yml

Action hook

yaml:.lamvery.hook.yml

yaml:.lamvery.api.yml

Setting items

basic configuration

Event (CloudWatch Events) settings

Settings for confidential information

Exclusion list

Action hook

command

Flow of creating Lambda function that uses confidential information (reference)

1. Create a private key with KMS.

2. Add the following permissions to the IAM Role when executing the Function.

3. Generate a configuration file and Function template

4. Set the ID of the key created in 1 in the configuration file as shown below.

yaml:.lamvery.yml

yaml:.lamvery.secret.yml

5. Encrypt and store sensitive information

6. Write the source

lambda_function.py

lambda_function.js

8. Launch Function

Finally

`yaml:.lamvery.yml`

`yaml:.lamvery.event.yml`

`yaml:.lamvery.secret.yml`

`yaml:.lamvery.exclude.yml`

`yaml:.lamvery.hook.yml`

`yaml:.lamvery.api.yml`

`yaml:.lamvery.yml`

`yaml:.lamvery.secret.yml`

`lambda_function.py`

`lambda_function.js`