Investigate the cause when an error is thrown when python3.8 is not found when using lambda-uploader with python3.8
The beginning of the matter
I decided to add a python function that will be complicated to some extent on AWS Lambda, but since pip cannot be used on lambda, when using a function that uses an external module, ** [Sanzugawara-like sneaking work of finding the external modules used one by one from \ Python〇〇 \ Lib \ site-packages ] and zipping them together occurs. ** ** Of course, human beings are unreliable creatures, so if you try to do this kind of work manually every time, an accident will occur like the runaway blue train of a certain Britain who slammed Panjandrum and marmite **
** Will an accident happen? Accidents are caused by humans! ** ** (* In addition, this work takes a lot of time)
Therefore, I planned to improve efficiency and reduce the accident rate as much as possible by using lambda-uploader and entrusting the extraction work of external modules.
Work environment
- Windows10(64bit) --python3.8 (using 64bit installer) 3.8 is because lambda does not support python3.8 or later as of the end of December 2020. --In addition to the above, the installation location of python is (Originally not recommended, but it may be inherited for each terminal, so it is under C: \ Program Files. Originally, each c: \ users ~ or below)
To use lambda-uploader
・ Lambda.json → Configuration file that summarizes the outline of the lambda function to be uploaded ・ Requirements.txt → If you define an external module to be used by the created lambda function, it will refer to this file and automatically pick it up and zip it. In other words, if you create a batch that automatically generates files from pip freeze, it will automate the most risky work. ** Do not cause an accident **
・ Event.json → The content of the POST request to be thrown during the test, which may be empty when testing the GET request So this time it is unnecessary. Other reference contents etc. When there is no description, so it seems that it is not a required file.
・ 〇〇.py → It is an entry point of aws lambda
python
def lambda_handler(event, context):～
A py file that describes the processing of functions.
The procedure of lambda-uploader is roughly divided into the following three points in order based on the configuration files described above.
- build
- zip
- upload (to aws lambda)
Set as follows in the json file (information related to security is hidden) ...
lambda.json
{
"name": "testFunction",
"description": "testFunction description.",
"region": "ap-northeast-1",
"handler": "lambda_function.lambda_handler",
"role": "arn:aws:iam::XXXXXXXXXXXX:role/BlogenistBlogSample",
"timeout": 30,
"memory": 128,
"runtime": "python3.8"← Default if not explicitly specified(Python2 which is being abandoned from the times.7)Upload with.
}
Exo Dedede Dedede
And an (unexpected) accident happened
Carn
PS C:\workspace(python)\testcode\lambda> lambda-uploader Building Package
âï¸ Unexpected error. Please report this traceback.
Uploader: 1.3.0
Botocore: 1.16.38
Boto3: 1.19.38
Traceback (most recent call last):
File "c:\program files\python38\lib\site-packages\lambda_uploader\shell.py", line 194, in main
_execute(args)
File "c:\program files\python38\lib\site-packages\lambda_uploader\shell.py", line 83, in _execute
pkg = package.build_package(pth, requirements,
File "c:\program files\python38\lib\site-packages\lambda_uploader\package.py", line 51, in build_package
pkg.build(ignore)
File "c:\program files\python38\lib\site-packages\lambda_uploader\package.py", line 79, in build
self.install_dependencies()
File "c:\program files\python38\lib\site-packages\lambda_uploader\package.py", line 151, in install_dependencies
self._build_new_virtualenv()
File "c:\program files\python38\lib\site-packages\lambda_uploader\package.py", line 177, in _build_new_virtualenv
python_exe = self._python_executable()
File "c:\program files\python38\lib\site-packages\lambda_uploader\package.py", line 195, in _python_executable
raise Exception('Unable to locate {} executable'
Exception: Unable to locate python3.8 executable
(´ ・ ω ・ `)
Exception: Unable to locate python3.8 executable
(´ ； ω ； `) Boo
In the last line, if python3.8 is not found at build time, an error will be thrown (Oh).
Try to find out the cause
That's why (I think that python.exe is there because the final execution environment python itself is on aws), and the cause investigation started.
-
Isn't your PATH? → No problem. Check both [\ python38 ] and [\ Python38 \ Lib \ site-packages ] in the environment variable settings (or rather, since I have been using commands so far with the setting to pass PATH at the time of installation, PATH is the cause at this point That's strange)
-
Do I have to specify a minor version as well? So I modified the json parameter to "runtime": "python 3.8.6" →Exception: Unable to locate python3.8.6 executable (´ ・ ω ・ \
) (´ ・ ω ・) -
Better yet, remove json's runtime parameter →Exception: Unable to locate python2.7 executable (´ ・ ω ・ \
) (´ ・ ω ・ \) (´ ・ ω ・ `) Solise Soda -
lambda-uploader It may be a bug of the module, so Go to git to see the latest version (** The module that is not reflected in pypi and is updated forever only on git Unexpectedly many **) ↓
↓
(´ ； ω ； `) Boo
Even if I check the error contents in various ways, it does not get caught So, since there is Traceback, I go directly to the source code of lambda-uploader. If you haven't messed with the settings strangely, \ Python38 \ Lib \ site-packages \ lambda_uploader has the target code, so open the directory quickly and near the last processed part
File "c:\program files\python38\lib\site-packages\lambda_uploader\package.py", line 195, in _python_executable
raise Exception('Unable to locate {} executable'
Investigate.
package.py
def _python_executable(self):
if self._pyexec is not None:
python_exe = find_executable(self._pyexec)
if not python_exe:
Line 195 → raise Exception('Unable to locate {} executable'
.format(self._pyexec))
If the processing of the find_executable function is suspicious, I will stare at it and investigate the caller.
- It is recommended to build a python environment with vs code because intelligence is convenient even for complicated code. I want to write an article about building a development environment someday.
spawn.py
def find_executable(executable, path=None):
"""Tries to find 'executable' in the directories listed in 'path'.
A string listing directories separated by 'os.pathsep'; defaults to
os.environ['PATH']. Returns the complete filename or None if not found.
"""
_, ext = os.path.splitext(executable)
if (sys.platform == 'win32') and (ext != '.exe'):
executable = executable + '.exe'
if os.path.isfile(executable):
return executable
if path is None:
path = os.environ.get('PATH', None)
if path is None:
try:
path = os.confstr("CS_PATH")
except (AttributeError, ValueError):
# os.confstr() or CS_PATH is not available
path = os.defpath
# bpo-35755: Don't use os.defpath if the PATH environment variable is
# set to an empty string
# PATH='' doesn't match, whereas PATH=':' looks in the current directory
if not path:
return None
paths = path.split(os.pathsep)
for p in paths:
f = os.path.join(p, executable)
print(f)
if os.path.isfile(f):
print(f)
# the file exists, we have a shot at spawn working
return f
return None
As far as I can see the above code in order, I got the environment variable, divided it (a single line string connected by a semicolon if it is as it is), attached something to the end and checked the file.
for p in paths:
f = os.path.join(p, executable)
print(f) #Print statement in case of trouble
if os.path.isfile(f):
# the file exists, we have a shot at spawn working
return f
Run again.
PS C:\workspace(python)\testcode\lambda> lambda-uploader
Building Package
* ↓ is the content that was spit out by print
C:\Program Files\Python38\Scripts\python3.8.exe
C:\Program Files\Python38\python3.8.exe
・ ・ ・ Python3 at the end of other environment variables below.8.String with exe attached&Traceback
As far as I saw the above, I thought that it was not such a strange thing, but the error was that python3.8 could not be found, so when I went to the corresponding directory ...
(ω ・ \ ) (・ ω ・ \) (´ ・ ω ・ \ ) (´ ； ω ；) Boo
**why? ** **
Then rewrite json.
So it was because I was searching for an exe file that did not exist in the first place. for that reason,
"runtime": "python"
If you write **, it will move. Until the build. ** ** It's obviously a strange movement, so I thought that if I moved it as it was, it would get stuck after that. As a matter of course, it failed in the upload process. The content of the exception
botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the UpdateFunctionConfiguration operation: Value python at 'runtime' failed to satisfy constraint: Member must satisfy enum value set: [java8, java11, nodejs10.x, nodejs12.x, python2.7, python3.6, python3.7, python3.8, dotnetcore2.1, go1.x, ruby2.5] or be a valid ARN
- In summary, write the python version in the ** runtime parameter of lambda.json. ** Or do you have a really valid ARN setting?
Since botocore is a part that handles aws authentication, I checked Is there a package that operates aws services (it was misguided from the conclusion) botocore module In the first place, the source code of the lambda-uploader module issued a SyntaxWarning: "is" with a literal. Did you mean "=="? warning, and ** after all Can't it be used with python3.8? I tried the following that came up while thinking **.
PS C:\workspace(python)\testcode\lambda> lambda-uploader Building Package
Uploading Package
Fin
Even if I checked on aws lambda, I was able to upload normally. I'm sorry that the solution wasn't very refreshing.
However, what kind of flow does python〇〇.exe come in? (I was wondering if vitalenv would refer to python〇〇.exe that was generated by interrupting earlier, but even if I looked at the contents, it was referenced before virtualenv.)
If anyone knows the cause, please point it out in the comments.
Conclusion
** The great thing about python is that you can directly rewrite the module that has been PIPed, debug it, and check its operation. (Don't forget to put it back when you're done) **
Also, lambda-uploader hasn't been updated for nearly 3 years, and existing code is starting to have trouble due to recent changes in python specifications, so it may be better to assume that it will not be usable in the future.
Recommended Posts