This factor is a dependency management factor that is both original and beyond. Most modern development languages have package management tools for resolving library references such as npm, pip, and gradle. Resolving application dependencies using such package management ensures that a well-configured set of applications can be built and run in any environment. Also, in terms of dependencies, you need to pay attention not only to application libraries, but also to build scripts and deployment scripts. You need to prevent these scripts from being implicitly used for libraries, commands, etc. installed on a particular system.
Furthermore, when I make an app in a Windows environment like me, if the script is written in a shell, I feel that it is system-dependent. As long as you have a runtime version on any OS. I think we should make efforts with the goal of being able to cover all the development, testing, and execution of applications in equal conditions. ..
Continuing from last time, I will try to practice this factor based on Lambda and Python. I will proceed with the flow from package management using Python pipenv to creating a package for deploying Lambda functions based on it. Python to be used is 3.8, and the runtime installation will start from the point where it is completed.
pipenv pipenv is a tool that integrates package management with pip and virtual environment management with venv. I think it is at a high level as a tool for explicitly separating and managing dependencies.
pip install pipenv
PIPENV_VENV_IN_PROJECT=1
This will make the virtual environment created by virtualenvv project-specific. Considering that multiple projects are handled at the same time, it is recommended to separate them so far from the viewpoint of dependency separation.
pipenv --python 3.8
The Pipfile file and the .venv folder are created directly under the project folder, and you are ready to go.
By using pipenv, it is possible to use it only in the development environment and manage unnecessary packages in the product code. Tools such as autopep8 and pylint correspond to it, so install it as follows.
pipenv install --dev autopep8 pylint
boto3
The Python version of the AWS SDK, boto3, is pre-installed in the Lambda runtime environment. In other words, if you want to use boto3 included in the execution environment, install boto3 as a development package so that it is not included in the product code. This is effective because it reduces the time to load the huge code of boto3 at the cold start of Lambda. However, boto3 included in the execution environment has a weakness that the version cannot be controlled by the user. If AWS changes the Lambda execution environment, there is always the risk that the version of boto3 in the production environment will suddenly change one day. Furthermore, if the version upgrade includes a braking change, it may suddenly stop working. Also, boto3 included in the execution environment is a slightly older version of the module, so it cannot be used in use cases where the latest version of boto3 is required. When operating a system that runs 24/365, it is recommended that boto3 be included in the application, even at the expense of cold start performance.
This time, we will build a project by embracing boto3 in the application. Install the package below.
pipenv install boto3
In order to deploy Lambda functions, you need to zip and upload the required modules, including the dependent packages. The process for ZIP conversion is roughly as follows.
First of all, the prerequisite project structure is as follows.
root
+ src
| + lambda_function.py
+ packing.py
+ Pipfile
+ Pipfile.lock
+ setup.py
__src folder __: An application for Lambda functions. Contains a Lambda handler. packing.py: Code for packaging Lambda functions Pipfile: pipenv configuration file Pipfile.lock: Detailed information about the environment built with pipenv. Use this file to completely reproduce the environment setup.py: Configuration file for deploying Lambda function applications with pip
First of all, setup.py is as follows. This is a configuration file for installing a set of apps under the src folder with the pip command, which is used to package the application. In the following, lambda_function.py and packages under src, if any, will be installed. This is the setting for installing pip.
from setuptools import setup, find_packages
setup(
name="12factor",
version="1.0.0",
packages=find_packages(where='src'),
py_modules=['lambda_function'],
package_dir={'': 'src'},
)
Next, packing.py for actual packaging is as follows. I am writing in python so that I can create ZIP in any environment without depending on the system. The general flow is to collect the necessary files in .dist as a temporary folder and zip them. In addition, this program is not executed in the development environment, but it is assumed to be executed in a clean environment where the source code is obtained from git etc. I am using it for packaging by temporarily creating a virtual environment of .venv. Be careful because if you create a virtual environment of .venv in the development environment, it will be destroyed.
pipfile.lock in the ZIP folder. The pipenv sync command faithfully reproduces the packages, making it the most appropriate solution from a "dependency management" perspective. If you specify a folder for ZIP in the PIP_TARGET environment variable. pipenv sync Only dependent packages will be installed in that folder when you run the command.import os
import shutil
import subprocess
DISTINATION_FOLDER = '.dist'
if __name__ == '__main__':
#Create a folder for ZIP
shutil.rmtree(DISTINATION_FOLDER, ignore_errors=True)
os.mkdir(DISTINATION_FOLDER)
#Change the folder where packages are saved during pipenv sync
os.environ['PIP_TARGET'] = DISTINATION_FOLDER
#Delete if virtual environment already exists
subprocess.call('pipenv --rm')
# pipfile.Virtual environment reproduction from lock
subprocess.call('pipenv sync')
#Install app code with pip
subprocess.call('pip install -t {} .'.format(DISTINATION_FOLDER))
# ZIP
shutil.make_archive('lambda_function', 'zip', DISTINATION_FOLDER)
#Delete virtual environment
subprocess.call('pipenv --rm')
As a third factor, I practiced dependency management methods using Python and Lambda. I think that how sensitive dependency management is managed depends on the project, but I hope that the example introduced this time can be used as an example when it is managed most strictly.
Next time, "(4) Design, Build, Release, Execution / Build, Release, Execution --- Strictly separate the three stages of Build, Release, and Execution". Table of Contents of this series Table of Contents of All Series