Try using the services that support serverless development that appeared in 2020 (CDK Pipelines / CodeArtifact / CodeGuru for Python)
This article is the 20th day post of Serverless Advent Calendar 2020. This time, we will introduce the services that appeared in 2020 that mainly support the development of serverless applications using AWS Lambda, which are likely to be useful mainly for development using Python.
Service to touch this time
- CDK Pipelines
- Announced in October 2020
- Developer Preview as of December 20, 2020
- AWS CDK can be used to manage CI/CD pipeline and resource group deployments together
- This makes cross-account deployment ** insanely easy **
- AWS CodeArtifact
- Announced in June 2020
- Used as a cache for official repositories (PyPi, npm, etc.)
- You can exercise control such as using only packages approved within your organization.
- You can treat and manage your own packages like other official repositories.
- Amazon CodeGuru
- Announced at re: Invent in December 2019 and became GA in July 2020.
- Use machine learning to automatically review your code and make recommendations (AWS CodeGuru Reviewer)
- Identify and visualize costly rows in running your application (AWS CodeGuru Profiler)
- In addition to Java that was previously supported at re: Invent in December 2020, Python is also supported.
CDK Pipelines There should be many people who are already operating serverless applications using Lambda etc. in production. Compared to resources such as EC2 used in VPC, resources used serverless such as Lambda and DynamoDB are It may be difficult to operate if resources of multiple stages (development, verification, production, etc.) are contained in one account.
In such a case, the configuration to be created will be as shown in the figure below.
Until now, it was possible to deploy with such an account configuration using CDK etc., but cross-account deployment is a difficult task just by building CI/CD.
So let's take advantage of this CDK Pipeline
Premise
The account structure is as follows (account ID in parentheses) Account A (111111111111): The account where the CI/CD pipeline and Git repository are located. Account B (222222222222): Verification account Account C (333333333333): Production account
When you push to the master branch of the CodeCommit repository located in account A, the Lambda Function is automatically deployed to each environment. Deployment is first done in the Stg environment (account B), then manually approved and then in the Prod environment (account C).
Preparation
Register the AWS CLI profile for accounts A to C as follows Account A: manage Account B: stg Account C: prod
Find out the account number of each account in advance. You can check each with the following commands.
$ aws sts get-caller-identity --profile manage
{
"UserId": "AROAIZMAXJNBKV2ORNQAG:botocore-session-1000000000",
"Account": "111111111111", <--this
"Arn": "arn:aws:iam::111111111111:user/k1nakayama"
}
Also, create a CodeCommit repository in account A. (Here we will create a serverless2020 repository)
CDK init Clone the created repository [^ 1] and perform CDK in it.
$ git clone codecommit://cbd@serverless2020
$ cd serverless2020
$ cdk init --language=python
Bootstrap for deployment with CDK Pipelines
Bootstrap is performed as a process to create an S3 bucket for uploading various assets when deploying using CDK, and to create an IAM role required for deployment.
Before bootstrap, add the settings to cdk.json.
cdk.json
{
"app": "python3 app.py",
"context": {
"@aws-cdk/core:enableStackNameDuplicates": "true",
"aws-cdk:enableDiffNoFail": "true",
"@aws-cdk/core:stackRelativeExports": "true",
"@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
"@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
"@aws-cdk/aws-kms:defaultKeyPolicies": true,
"@aws-cdk/core:newStyleStackSynthesis": "true" <--Add this line
}
}
Then run the bootstrap command. This work command is given to each account, but it is different for the account that builds CI/CD (account A) and the account that places resources (account B, account C).
Account A:
$ cdk bootstrap --profile manage --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://111111111111/ap-northeast-1
Account B/C:
$ cdk bootstrap --profile stg --trust 111111111111 --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://222222222222/ap-northeast-1
$ cdk bootstrap --profile prod --trust 111111111111 --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://333333333333/ap-northeast-1
Installation of CDK module
In order to build CI/CD using CDK Pipelines, you need to load the following modules.
aws-cdk.core
aws-cdk.aws-codepipeline
aws-cdk.aws-codepipeline-actions
aws-cdk.aws-codecommit
aws-cdk.pipelines
We will also use the following modules to deploy the Lambda Function this time.
aws_cdk.aws_lambda
aws-cdk.aws_logs
These are defined in install_requires in setup.py.
setup.py
install_requires=[
"aws-cdk.core",
"aws-cdk.aws-codepipeline",
"aws-cdk.aws-codepipeline-actions",
"aws-cdk.aws-codecommit",
"aws-cdk.pipelines",
"aws_cdk.aws_lambda",
"aws-cdk.aws_logs"
],
After defining the above, let's install
$ pip install -r requirements.txt
CDK for deploying Lambda Functions
This is the same as the CDK code so far, so I will paste only the source code.
serverless2020/serverless2020_stack.py
from aws_cdk import (
core,
aws_lambda as lambda_,
aws_logs as logs,
)
class Serverless2020Stack(core.Stack):
def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
hello_serverless_func = lambda_.Function(self, "HelloServerlessFunction",
code=lambda_.Code.from_asset('functions/app'),
handler="index.lambda_handler",
runtime=lambda_.Runtime.PYTHON_3_8,
tracing=lambda_.Tracing.ACTIVE,
timeout=core.Duration.seconds(29),
memory_size=128,
)
logs.LogGroup(self, 'HelloServerlessFunctionLogGroup',
log_group_name='/aws/lambda/' + hello_serverless_func.function_name,
retention=logs.RetentionDays.TWO_MONTHS
)
CDK for Pipeline
This is the heart of CDK Pipelines, but for a detailed explanation, please refer to other blogs. I will also paste the necessary source code here.
pipeline_lib/pipeline_stage.py
from aws_cdk import (
core
)
from serverless2020.serverless2020_stack import Serverless2020Stack
class PipelineStage(core.Stage):
def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
Serverless2020Stack(self, 'Serverless2020Stack')
pipeline_lib/pipeline_master_stack.py
from aws_cdk import (
aws_codepipeline as codepipeline,
aws_codepipeline_actions as actions,
aws_codecommit as codecommit,
pipelines,
core
)
from pipeline_lib.pipeline_stage import PipelineStage
class PipelineMasterStack(core.Stack):
def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
sourceArtifact = codepipeline.Artifact()
cloudAssemblyArtifact = codepipeline.Artifact()
pipeline = pipelines.CdkPipeline(self, 'Pipeline',
pipeline_name="serverless2020-master-pipeline",
cloud_assembly_artifact=cloudAssemblyArtifact,
source_action=actions.CodeCommitSourceAction(
action_name="CodeCommitgit",
repository=codecommit.Repository.from_repository_name(
self, 'Repo', repository_name="serverless2020"),
output=sourceArtifact,
branch="master"
),
synth_action=pipelines.SimpleSynthAction(
synth_command="cdk synth",
install_commands=[
"pip install --upgrade pip",
"npm i -g aws-cdk",
"pip install -r requirements.txt"
],
source_artifact=sourceArtifact,
cloud_assembly_artifact=cloudAssemblyArtifact,
environment={
'privileged': True
}
)
)
stg = PipelineStage(self, "serverless2020-stg",
env={
'region': "ap-northeast-1", 'account': "222222222222"}
)
stg_stage = pipeline.add_application_stage(stg)
stg_stage.add_actions(actions.ManualApprovalAction(
action_name="Approval",
run_order=stg_stage.next_sequential_run_order()
))
prod = PipelineStage(self, "serverless2020-prod",
env={
'region': "ap-northeast-1", 'account': "333333333333"}
)
pipeline.add_application_stage(app_stage=prod)
app.py
#!/usr/bin/env python3
from aws_cdk import core
from pipeline_lib.pipeline_master_stack import PipelineMasterStack
app = core.App()
PipelineMasterStack(app, "serverless2020-master-pipeline",
env={'region': "ap-northeast-1", 'account': "111111111111"}
)
app.synth()
Deploy
After writing the code so far, it's time to deploy. First of all, let's check if the CDK code can be written without problems
$ cdk synth
If the CloudFormation template is displayed successfully above, it's OK.
Then commit and push these codes into your git repository
$ git add .
$ git commit -m "init"
$ git push
Then it is a long-awaited deployment.
cdk deploy --profile manage
By the time it's complete, the Code Pipeline should already be in progress.
It was so easy to build a cross-account deployment. From now on, just push it to the master branch and CI/CD will work automatically.
AWS CodeArtifact In order to proceed with building serverless applications using Lambda, we often use packages from official public repositories. Efficient use of these caches can be time-consuming for frequent deployments using the CI/CD pipeline. It will also be needed for control. In addition, as we continue to develop serverless applications, we can also create and share libraries that are commonly used for our own projects. I think it is often done.
A service that supports these needs is AWS Code Artifact.
In CodeArtifact, a "domain" is created with one image for the entire organization (a group of multiple AWS accounts bundled together). Separately from this domain, create a "repository" according to the difference in policy for each project and stage. Access to the package is done by connecting to the endpoint of each repository. Therefore, it is possible to manage access rights and package approval for each repository. However, the cache for each version of the package will accumulate for the "domain".
Creating a domain repository
Let's use it now.
For the sake of simplicity, let's create and use one domain and one repository for one account.
After logging in to the AWS Management Console, access AWS Code Artifact.
Select "Create Domain"
For the domain name, enter the normal organization name as described.
The domain was created only by the above.
Then select "Create Repository" to create a repository for production release.
Basically, different packages (versions) may be used in the development environment and the production environment, so it is better to create a repository for each environment and use them properly.
Select the official repository that you may use in your project as the upstream repository.
Select "Create Repository" to complete the repository creation.
Connect to repository
The connection method to the repository can be displayed for each repository by selecting the "Display connection procedure" button on the screen of each repository.
For example, in the case of pip, you can connect with the following command.
- Considering the example of CDK Pipelines, when connecting to an account that is not the default profile, specify the profile with
--profileas shown below.
$ aws codeartifact login --tool pip --repository serverless2020-prod --domain serverless2020 --domain-owner 111111111111 --profile manage
The above command will issue a connection token that is valid for 12 hours, after which pip will connect via Code Artifact.
Install via repository
As a test, try installing the pip package of the project created by CDK Pipelines.
- Since all the packages have already been installed by the above work, the virtual environment is recreated and executed once.
$ pip install -r requirements.txt
After execution, if you check the repository screen on the AWS management console earlier, you can see the information of the installed package as shown below. You now have a cache of each version of the package used in this repository and its artifacts.
Registration of original package
Next, I would like to register my own package.
- It is assumed that the original package has been created in advance and is packaged in dist /.
First, connect to twine to upload the package for Python.
$ aws codeartifact login --tool twine --repository serverless2020-prod --domain serverless2020 --profile manage
As with pip, you can connect with aws codeartifact login --tool twine --repository repository_name --domain domain_name.
Next, install twine and upload it for uploading using twine.
$ pip install -U twine
$ twine upload -r codeartifact dist/*
Above I uploaded my own package called "cp-lambda-utils".
You can see that the upload was successful.
Install your own package
Let's try installing your own package
$ pip install -U cp_lambda_utils
The above installation was successful, and you can see that the installation was successful by checking pip freeze.
Used with CDK
Make sure to get the package used by Lambda deployed by CDK via CodeArtifact.
First of all, the CodeBuild project created with CDK Pipelines needs to have permission to access CodeArtifact.
After adding the installation of aws-cdk.aws_iam to setup.py, rewrite pipeline_master_stack.py as follows.
pipeline_lib/pipeline_master_stack.py
from aws_cdk import (
aws_codepipeline as codepipeline,
aws_codepipeline_actions as actions,
aws_codecommit as codecommit,
aws_iam as iam,
pipelines,
core
)
~~~Omission~~~
pipeline = pipelines.CdkPipeline(self, 'Pipeline',
pipeline_name="serverless2020-master-pipeline",
cloud_assembly_artifact=cloudAssemblyArtifact,
source_action=actions.CodeCommitSourceAction(
action_name="CodeCommitgit",
repository=codecommit.Repository.from_repository_name(
self, 'Repo', repository_name="serverless2020"),
output=sourceArtifact,
branch="master"
),
synth_action=pipelines.SimpleSynthAction(
synth_command="cdk synth",
install_commands=[
"pip install --upgrade pip",
"npm i -g aws-cdk",
"pip install -r requirements.txt"
],
source_artifact=sourceArtifact,
cloud_assembly_artifact=cloudAssemblyArtifact,
environment={
'privileged': True
}
),
role_policy_statements=[
iam.PolicyStatement(
actions=['codeartifact:*'], resources=['*']),
iam.PolicyStatement(
actions=['sts:GetServiceBearerToken'], resources=['*'])
]
)
This will push first. Next, add a description to connect to CodeArtifact in CodeBuild.
pipeline_lib/pipeline_master_stack.py
install_commands=[
"pip install --upgrade pip",
"aws codeartifact login --tool pip --repository serverless2020-prod --domain serverless2020 --domain-owner 111111111111",
"npm i -g aws-cdk",
"pip install -r requirements.txt"
],
This will push again. If you do not divide the stages in this way, an error will occur because you do not have permissions.
Next, place requirements.txt in the package target directory of Lambda Function, describe the package to be installed, and then
Rewrite serverless2020_stack.py as follows.
serverless2020/serverless2020_stack.py
from aws_cdk import (
core,
aws_lambda as lambda_,
aws_logs as logs,
)
from pathlib import Path
~~Omission~~
hello_serverless_func = lambda_.Function(self, "HelloServerlessFunction",
code=lambda_.Code.from_asset(
'functions/app',
asset_hash_type=core.AssetHashType.SOURCE,
bundling=core.BundlingOptions(
image=lambda_.Runtime.PYTHON_3_8.bundling_docker_image,
command=[
"bash",
"-c",
" && ".join(
[
"pip install -r requirements.txt -t /asset-output",
"cp -au . /asset-output",
]
),
],
user="root:root",
volumes=[
core.DockerVolume(
container_path="/root/.config/pip/pip.conf",
host_path=f"{Path.home()}/.config/pip/pip.conf",
),
],
),
),
handler="index.lambda_handler",
runtime=lambda_.Runtime.PYTHON_3_8,
tracing=lambda_.Tracing.ACTIVE,
timeout=core.Duration.seconds(
29),
memory_size=128,
)
As mentioned above, it is currently not possible to pass AWS credentials to the Docker container for packaging Lambda, so It's a little roundabout way. It seems that this is being considered in issue # 10298.
With the above, you have successfully installed the package using CodeArtifact and can now deploy it.
Amazon CodeGuru It's not limited to serverless applications, but there are still a small number of engineers who develop using the cloud, and there are many engineers who do not have an environment in place to review whether they can write appropriate code. Meanwhile, by using Amazon CodeGuru, you can use CodeGuru Reviewer, which finds and points out various code problems by machine learning.
The contents that Amazon CodeGuru Reviewer points out are as follows.
Amazon CodeGuru Reviewer checks for concurrency issues, potential race conditions, malicious or unscrutinized inputs, improper handling of sensitive data such as credentials, resource leaks, and parallel code conflicts. It also detects operational failures due to conditions and collisions. In addition, we propose best practices on AWS and Java to detect consolidable duplicates in your code and improve maintainability.
- Quoted from Code Guru Reviewer FAQ
Associate Amazon CodeGuru Reviewer with CodeCommit repository
Let's use it immediately First, make an association.
Select "Associated Repositories" from the Amazon Code Guru page.
Then select "Repository Association"
Select AWS CodeCommit as your source provider and select your repository as your repository location. You can make an association by selecting "Association".
By the way, this association can also be set by checking the "Enable Amazon CodeGuru Reviewer for Java and Python" item when creating the CodeCommit repository.
Once associated, it will appear in the repository list.
Try to issue a Pull Request
Once you have made the association, let's issue a PR immediately.
After rewriting and pushing the code, select "Create Pull Request" from the CodeCommit console.
Enter a title and select "Create Pull Request".
When you do PR, when you look at the activity tab, you can see that the status is in progress in the job status column of Amazon CodeGuru Reviewer and that the review is being done.
Review results
After a few minutes, the status will be completed and the indication will be displayed in the activity history column.
The source code that issued the PR this time was very simple, and there was only one point.
Personally, I pointed out that list_objects is an old API and it is better to use list_objects_v2, and that it is possible that the object list is not completely cut off in one request, so verify IsTruncated etc. In the first place, I expected that it would point out that the processing was completed without using any obj_list variable, and that os.environ specified a Key that I do not know if it exists, but that is still the case. It seems that there are no indications around.
CodeGuru Reviewer reviews based on machine learning. In other words, as a reviewer who continues to learn and grows, let me tell you what I expected.
It seems that it may be used to improve the service by reacting to the review with pictograms and replying as shown below.
Amazon CodeGuru Profiler Amazon CodeGuru has another feature called Profiler, which helps you to profile the actual execution situation in Lambda etc. and find the most costly line etc.
I haven't tried this time because I can't execute enough for profiling, but it seems that these settings can be set by adding a little to the options of CDK.
Summary
How was that? This time, we introduced the services that support the development of Lambda Functions using Python. It would be great if you could see that the ones that have appeared in the past year have evolved significantly, making it easier to do serverless development.
Personally, I hope that the last Amazon CodeGuru Reviewer pointed out will become more sophisticated and will grow into a reliable presence for doing business. Let's give feedback more and more!
[^ 1]: In the example, I cloned using git-remote-codecommit
Recommended Posts