Note: This article is a reprint of the note article. If there is an update of the contents, the note will be updated. If the qiita article is out of date, please also check the note.
https://note.com/thiroyoshi/n/nfcb0be8c0a22
Hello. This is thiroyoshi.
This time, I would like to introduce a little technique when using the Python library with Lambda. I've incorporated this into the VueSlsApp. It doesn't look any different, but it's recommended because it simplifies the sauce.
If you are new to VueSlsApp, check out the introductory articles and magazines below. https://note.com/thiroyoshi/n/na16112b4ec1b
You can actually touch what you have deployed from the following. https://vueslsapp.thiroyoshi.com/
The repository of VusSlsApp is as follows. https://github.com/thiroyoshi/vueslsapp
table of contents
--When using a Python library with Lambda, the problem of raising the library together ... A more serious problem that does not pass the path --Add the path to PTHONPATH and you'll be able to see Python --If you have environment-dependent libraries, this method may not be good
The problem that everyone faces when using Python with Lambda is to put together a library, but now there are various tools and it is relatively easy to raise.
The Serverless Framework, which is also used in VueSlsApp, automatically puts together the things in the directory, so there is no problem.
The most troublesome thing is that ** the library cannot be found and cannot be executed because the path does not pass at the time of execution **. It doesn't matter if you're using the Serverless Framework ...
Of course, Lambda, which contains the Python execution environment, doesn't have the path of the library that I added without permission unless I set anything.
Then ** just pass the pass **.
Python has an environment variable called PYTHONPATH that sets the path to search for sources. See below for the official commentary.
https://docs.python.org/ja/3/using/cmdline.html#envvar-PYTHONPATH
The following article describes how to pass PYTHONPATH on Lambda and all the research contents up to that point. informative.
http://uorat.hatenablog.com/entry/2017/08/31/145340
The important points are as follows.
・/var/Also specify runtime
On Lambda, it seems that the Python runtime is also specified in PYTHONPATH by default. In other words, ** If you overwrite all PYTHONPATH by yourself, you will not be able to execute Python in the first place **.
In VueSlsApp, libraries are collected in a directory called pylibs, so I set it as follows.
** ・ serverless.yml (partial excerpt) **
PYTHONPATH: /var/runtime:/var/task/pylibs
As a result, external library references are modified as follows:
** ・ functions / repository / messages_repository.py (before modification) **
import json
import random
import hashlib
from decimal import Decimal
from functions.common import initializer
initializer.add_path()
from pylibs import boto3
from pylibs.boto3.dynamodb.conditions import Key
from functions.common.decimal_encoder import DecimalEncoder
** ・ functions / repository / messages_repository.py (after modification) **
import json
import random
import hashlib
from decimal import Decimal
from pylibs import boto3
from pylibs.boto3.dynamodb.conditions import Key
from functions.common.decimal_encoder import DecimalEncoder
The fix is that the code that dynamically added the path of the external library ** initializer.add_path () ** is no longer needed.
It's not cool and the performance drops, so I was wondering how to stop this method ... I was happy to finally stop! (Rejoicing
The above method is not suitable if the library itself needs to be rebuilt for the execution environment.
This method is only for ** using a library that runs on the OS with pip install **. If you pip install on Windows, you will get all the libraries that run on Windows.
In other words, this method is not suitable when the environment is highly dependent and you need to change the build environment of the library according to when running on Lambda (= when running on Amazon Linux).
In such a case, I think there are the following methods. (I haven't tried it)
You can change the environment when you do pip install. In other words, let's set the OS that runs the CI / CD environment to Amazon Linux.
However, depending on the CI / CD tools and services, you may not be able to change the OS freely, so I think you need to carefully check whether you can use the Amazon Linux image in your own environment.
There is a Serverless Framework plugin that nicely organizes Python libraries.
https://www.npmjs.com/package/serverless-python-requirements
This plugin also uses Docker to build using the same environment as the execution environment.
So why not use this plugin from the beginning?
The reason is simple: ** PYTHONPATH is all you need to specify **.
With the AP I'm making, only simple things can be done with Lambda. In other words, there is almost no opportunity to use a heavy library that depends on the build environment. What's more, using plugins involves a lot of unnecessary processing in my development, such as installing them and running Docker containers, which can increase the execution time of the pipeline.
This time, I introduced some ways to pass Python path in Lambda.
After all, there are various methods, but it is best to choose the method with the highest cost performance for you. Even if there is a useful tool, it is not good to blindly believe it and use it.
What do you need? What do you need? Is it a factor that reduces the efficiency of development? From that perspective, I would like to identify tools and development methods.
Recommended Posts