[PYTHON] Image processing with Lambda + OpenCV (gray image creation)

Image processing with Lambda + OpenCV (gray image creation)

This article is a hands-on article for reviewing and fixing the knowledge gained by developing Serverless Web App Mosaic. It is one of w2or3w / items / 87b57dfdbcf218de91e2).

It would be nice to read this article after looking at the following.

• Uploading files to S3 and calling Lambda using AWS Amplify

introduction

Implement Lambda function in Python 3.6 that performs image processing on the image triggered by uploading the image to S3. OpenCV is used for image processing.

content

Create a repository for Lambda function projects on Github

Create a repository for your project on GitHub. The repository name is sample_lambda_py_project.

Clone repository to Cloud9

$ git clone https://github.com/{YourGithubID}/sample_lambda_py_project.git

Creating a simple program

Create an appropriate folder (here, source) in sample_vue_project, and create a lambda_function.py file in it. For the time being, let's assume that the code is implemented inline in Previous article. (The content of the log message will be changed a little.)

lambda_function.py

from urllib.parse import unquote_plus
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    logger.info("Function Start (deploy from S3) : Bucket={0}, Key={1}" .format(bucket, key))

I think the Cloud9 environment looks like this.

Zip the function code and deploy it to Lambda via S3

It is not necessary for simple code as it is now, but if you use a library other than the standard library such as OpenCV, you need to upload the library as a package in a zip file.

So, first, let's zip it without using the library and upload it to Lambda via S3.

Zip compression

Compress under the directory where lambda_function.py is located with the name lambda-package.zip. Execute the following command in the directory where lambda_function.py is located.

$ zip -r ../lambda-package.zip *

Upload to S3

Upload the created lambda-package.zip to S3. Execute the following command while keeping the directory where lambda_function.py is located.

$ aws s3 cp ../lambda-package.zip s3://sample-vue-project-bucket-work/deploy/lambda-package.zip

Deploy in Lambda console

AWS Console> Lambda> Functions> S3Trigger ********-work To access. Set the function code as follows. Code entry type: File upload from Amazon S3 Runtime: Python 3.6 Handler: lambda_function.lambda_handler Amazon S3 link URL: The link URL of lambda-package.zip uploaded earlier Click the save button at the top right of the screen.

Operation check

After uploading the file from the sample_vue_project web app, AWS Console> CloudWatch> Log Group> aws / lambda / S3Trigger ********-work Please visit to see the latest log stream. Function Start (deploy from S3) : Bucket=sample-vue-project-bucket-work, Key=public/191231063457-16f5aaa24b1/faceA.jpeg It is OK if an INFO log like this is output.

OpenCV installation

In the source directory where lambda_function.py is located, run the following command.

• Make sure that the Python referenced by pip is 3.6.

$ pip --version
pip 9.0.3 from /usr/lib/python3.6/dist-packages (python 3.6)
$ pip install opencv-python -t .

The Lambda function needs to include the library in the zip package to upload, so -t . Specify this option and install it in the current directory.

Gray image converter

Update lambda_function.py.

lambda_function.py

# coding: UTF-8
import boto3
import os
from urllib.parse import unquote_plus
import numpy as np
import cv2
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
s3 = boto3.client("s3")

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    logger.info("Function Start (deploy from S3) : Bucket={0}, Key={1}" .format(bucket, key))

    fileName = os.path.basename(key)
    orgFilePath = u'/tmp/' + fileName
    processedFilePath = u'/tmp/processed-' + fileName

    if (not key.startswith("public")):
        logger.info("not start with public")
        return

    keyOut = key.replace("public", "processed", 1)
    logger.info("Output local path = {0}".format(processedFilePath))

    try:
        s3.download_file(Bucket=bucket, Key=key, Filename=orgFilePath)

        orgImage = cv2.imread(orgFilePath)
        grayImage = cv2.cvtColor(orgImage, cv2.COLOR_RGB2GRAY)
        cv2.imwrite(processedFilePath, grayImage)

        s3.upload_file(Filename=processedFilePath, Bucket=bucket, Key=keyOut)
        logger.info("Function Completed : processed key = {0}".format(keyOut))

    except Exception as e:
        logger.exception(e)
        raise e
        
    finally:
        if os.path.exists(orgFilePath):
            os.remove(orgFilePath)
        if os.path.exists(processedFilePath):
            os.remove(processedFilePath)

(* I won't touch on the basic syntax of Python here. If you need it, learn somewhere.)

Uploaded Convert a file like public / 191231112635-16f5bb5247f / lenna.png to a gray image, Uploading to a path like processed / 191231112635-16f5bb5247f / lenna.png.

Lambda basic settings

You can specify the memory and timeout in the Lambda basic settings. By default, memory = 128MB, timeout 25 seconds. Photos taken with recent smartphones are large in size, so if you keep the default settings, you may run out of memory and an exception may occur. When you increase the processing performed by Lambda in the future, you may run out of time and time out. Adjust your Lambda preferences as needed. (* Regarding the size of the image, I think it is better to compress it on the client side before uploading.)

Deployment and operation check

Zip it up, upload it to S3, deploy it to Lambda, and then upload the image from your web app. It is OK if the monochrome image is uploaded below the S3 bucket. Check the CloudWatch log as well. (Lena is beautiful)

Afterword

When you operate from Lambda faction to S3 which is the trigger source like this sample program, the Lambda function is called by that operation. If you are not careful, you will end up with an infinite loop of Lambda, so be careful.

If you inadvertently fall into a Lambda infinite loop, you should notice it right away, but if you don't notice it right away, you don't want to think about it. The automatic billing addition machine will continue to do its job without hesitation.

The way to stop the infinite loop Lambda is to either remove the function or upload the function code that raises the exception. I think it's a good idea to upload the function code that raises the exception, because deleting a function can be cumbersome to create and configure again, but is there any other way? Please let me know.

In any case, you will have to operate it with your trembling hands. While sweating strangely.

Also, I haven't debugged the Python code called by Lambda yet. It seems that you can debug with Cloud9, so once you learn it, you will be able to proceed with development more efficiently. Once you've mastered it, try to write an article.