[PYTHON] A story that I had a hard time trying to create an "app that converts images like paintings" with the first web application

Introduction

This is a story in which the author, who has little knowledge of networks, tried to create an image processing web application as the first web application and had trouble handling images. Even if I searched the net, there were surprisingly few articles about ** image processing web application using Python, GCP, Flask **, so ** I struggled with the procedure for creating an image processing web application ** that works for the time being. I will introduce it.

Finished product

** 1. When you open the website, select the image you want to process and send it, ** image.png ** Selected original image ** 処理前.jpg

** 2. An image converted like a painting will be returned. ** ** 処理後.PNG

Web application that converts images like paintings

Web app overview

I tried to summarize the outline of the created web application in the figure. This time, we used Google's cloud ** GCP ** as a web server within the scope of the free service. The actual image processing part was created using ** Python **, and the appearance of the website was easily created using only ** HTML **. Then, I created a link between Python and HTML using ** Flask **, which is a kind of Python API.

image.png

1. GCE (Google Cloud Engine) environment construction

First, we will build an environment to publish web applications using a service called GCE in GCP. ** Google account and credit card are required to use GCP **, so have them ready. Also, in this article, ** SSH connection from Google Chrome to GCE ** is used to remotely control the virtual environment in the cloud. In order to use the GCE service, you need to do the following three tasks, but please refer to the article that explains it in an easy-to-understand manner. Since I connect from Google Cloud Platform Console (browser is Google Chrome), I skipped the SSH authentication key part of the article. Settings are required if you want to make an SSH connection from a Mac terminal. [Reference article] GCP (GCE) to start from now on, safely use the free tier

[Required work] ・ Register for a free trial of GCP ・ Creating a project · Creating a virtual machine instance

2. How to start Cloud shell

After building the GCE environment, open the Instances page (https://console.cloud.google.com/compute/instances) and click Activate cloud shell in the upper right to launch the terminal. .. ** Once the terminal is up and running, you can work with the GCE instance you built using Linux commands. ** By the way, according to the Official Site, Compute Engine will automatically connect to the VM instance for the first time. It seems that an SSH authentication key pair has been generated in.

image.png

3. How to transfer data to and from the local environment

To run websites and image processing created in your local environment on GCE, you need to share your local data with GCE. There is a way to put it in a cool way with commands, but since it is troublesome to set up, I chose ** the method of using Cloud Storage, which is the easiest to understand **. From the GCP tab, select Strage → Browser. After creating a bucket with a suitable name in the storage browser, upload the file you want to share with GCE locally. image.png

And you can save the data saved in the storage to any location of GCE by executing the following code in the cloud shell. gsutil is an application provided to access Cloud Storage from the command line.

gsutil cp -r gs://{Bucket name}/ ./{GCE folder name, etc.}

[Reference article] [Free] The story of the Qiita Hall of Fame [Easy]

4. Web application file structure (when using GCE / Python / Flask)

When creating a web application using Python and Flask on GCE, it is necessary to have the following file and directory structure. ** Conversely, you can save the created program with the following configuration and execute some commands on GCP to create an application immediately. ** If you name each file and folder other than my folder name different from the following, an error will occur, so you need to be careful.

my folder name (any name is OK) /  ├ static/  ├ templates/  │  └ index.html  ├ app.yaml  ├ main.py  └ requirements.txt

I will briefly describe each role. -** static : Enter static data (css, Javascript, images, trained DL model, etc.) that you want to access while the app is running. This time it's empty. - templates : Save the html file. - app.yaml : This is an instruction sheet that describes the behavior of GAE. - main.py : A python file that describes data processing and cooperation with html. - requirements.txt **: A file containing the python library and version required to execute main.py. It will build the environment as it is at the time of deployment.

5. Creating a web app

Once you have created the above file structure on GCE, creating a web application is easy. Execute the following command.

To deploy your app, you must first create it in your region.

gcloud app create

Deploy the app.

gcloud app deploy app.yaml --project {my project name}

Enter your project name in the {my project name} part. I think it's written in the Cloud Shell. If you copy and paste the source code in Chapter 9 of this article to create a program file and the directory structure is as in Chapter 4, it will work. If you get an error, you can check the contents from App Engine Dashboard.

6. Image processing method

I adopted the ** Kuwahara filter ** introduced in the article below and used the code as it is. ** It's amazing that you can get a painting-like image that looks like an elaborate deep learning image processing, even though it can be expressed by a simple formula **. I think it's quite interesting. See the article below for details. I came across an image filter with a clearly Japanese name called Kuwahara filter, and when I tried it, it was amazing, so I will introduce it.

The original image a.jpeg

** After applying the filter ** b.jpe

7. Struggling with how to pass images from Python to HTML

I had a hard time passing the image after image processing in Python to HTML. At first, I usually saved the image once in a static file, passed the path to HTML, and read the image from HTML. ** However, while the app is running in the GCE environment, writing from Python to the folder seems to be restricted, resulting in an error. ** According to Official Site, the following four are recommended for saving data in the runtime environment. Was there. · Cloud Firestore in Datastore mode ・ Cloud SQL for MySQL ・ Cloud SQL for PostgreSQL ・ Cloud Storage

I tried Cloud Storage, which seems to be able to save images, but I couldn't get it to work well with my current knowledge of networks. Besides, I found an article that data can be saved with tempfile, which is a Python API, and tried it, and I was able to save data from Python, but on the contrary, I could not access from HTML and got stuck. It was.

When I think about it, I don't need to save the image somewhere in the first place, so I searched for a way to pass image data directly from Python to HTML **. ** There was a method of encoding image data with base64 on the Python side, passing it to HTML as a character string, and decoding the character string on the HTML side to create an image! ** When I actually tried it with reference to the article below, it worked. Image generated using pillow is displayed in html without saving to a file [Django]

8. Problems due to lack of memory

** When I tried various images, it seems that depending on the image size, the maximum memory (256MB) of the free version GCE was exceeded and an error occurred (it was useless if it was larger than 1000 x 600). ** Maybe I can improve the program and data structure without upgrading the machine, but I haven't done it yet. ** GCP has $ 300 free credit for the first year ** so you might want to upgrade and give it a try. By the way, you can check the details of the error from the App Engine dashboard in the GCP tab. メモリ不足エラー.PNG

9. Source code

app.yaml


runtime: python37

main.py


#Load the required module
#Flask related
from flask import Flask, render_template, request, redirect, url_for, abort
import base64
import tempfile
from PIL import Image
import io
import numpy as np
import cv2

app = Flask(__name__)

def kuwahara(pic,r=5,resize=False,rate=0.5): #Original image, one side of a square area, ratio when resizing or resizing
    h,w,_=pic.shape
    if resize:pic=cv2.resize(pic,(int(w*rate),int(h*rate)));h,w,_=pic.shape
    pic=np.pad(pic,((r,r),(r,r),(0,0)),"edge")
    ave,var=cv2.integral2(pic)
    ave=(ave[:-r-1,:-r-1]+ave[r+1:,r+1:]-ave[r+1:,:-r-1]-ave[:-r-1,r+1:])/(r+1)**2 #Batch calculation of average value
    var=((var[:-r-1,:-r-1]+var[r+1:,r+1:]-var[r+1:,:-r-1]-var[:-r-1,r+1:])/(r+1)**2-ave**2).sum(axis=2) #Bulk calculation of variance
    
    def filt(i,j):
        return np.array([ave[i,j],ave[i+r,j],ave[i,j+r],ave[i+r,j+r]])[(np.array([var[i,j],var[i+r,j],var[i,j+r],var[i+r,j+r]]).argmin(axis=0).flatten(),j.flatten(),i.flatten())].reshape(w,h,_).transpose(1,0,2)
    filtered_pic = filt(*np.meshgrid(np.arange(h),np.arange(w))).astype(pic.dtype) #Color determination
    return filtered_pic
    
@app.route("/", methods=["GET", "POST"])
def upload_file():
    if request.method == "GET":
        return render_template("index.html")
    if request.method == "POST":
        #Save the uploaded file once
        f = request.files["file"]
        filepath = tempfile.mkdtemp()+"/a.png "
        #filepath = "{}/".format(tempfile.gettempdir()) + datetime.now().strftime("%Y%m%d%H%M%S") + ".png "
        
        f.save(filepath)
        image = Image.open(filepath)
        
        #Image processing part
        image = np.asarray(image)
        filtered = kuwahara(image, r=7)        
        filtered = Image.fromarray(filtered)
        
        #Encode with base64
        buffer = io.BytesIO()
        filtered.save(buffer, format="PNG")
        img_string = base64.b64encode(buffer.getvalue()).decode().replace("'", "")
        
        result = "image size {}×{}".format(len(image[0]), len(image))
        return render_template("index.html", filepath=filepath, result=result, img_data=img_string)
    
    
if __name__ == "__main__":
     app.run(host="127.0.0.1", port=8080, debug=True)

The following article was used as a reference when writing Flask code. Python x Flask x PyTorch Easy construction of number recognition web application

requirements.txt


Flask==1.1.2
Pillow==7.2.0
Numpy==1.16.4
opencv-python==4.2.0.34

index.html


<html>
    <body>
        {% if result %}
	<IMG SRC="data:image/png;base64,{{img_data}}" alt="img_data"  id="imgslot"/>
        <div>{{result}}</div>
	<HR>
        {% endif %}
Please select a file and send<BR>
        <form action = "./" method = "POST" 
           enctype = "multipart/form-data">
           <input type = "file" name = "file" />
           <input type = "submit"/>
        </form>
     </body>
</html>

10. Finally

Thank you for reading the article until the end. Creating a web application wasn't straightforward, but I managed to make something that works. ** LGTM will be one of the motivations to write an article, so thank you. ** **

Recommended Posts

A story that I had a hard time trying to create an "app that converts images like paintings" with the first web application
The story that had nothing to do with partitions when I did disk backup with dd for the first time
Note that I was addicted to accessing the DB with Python's mysql.connector using a web application.
A story that I had a hard time displaying graphs on top of each other with matplotlib
I made a web application in Python that converts Markdown to HTML
With LINEBot, I made an app that informs me of the "bus time"
The story of making a web application that records extensive reading with Django
I want to create a web application that uses League of Legends data ①
Create a web app that converts PDF to text using Flask and PyPDF2
I want to create an API that returns a model with a recursive relationship in the Django REST Framework
(Note) A web application that uses TensorFlow to infer recommended song names [Create an execution environment with docker-compose]
A story that failed when trying to remove the suffix from the string with rstrip
I wanted to create a dll to use a function written in C from Python with ctypes, but I had a hard time
I had a hard time trying to access Hadoop3.0.0 from a browser (and Arch Linux)
I played with Single GANs (SinGAN) (+ I also summarized the points I had a hard time trying to implement (through path, Linux command, googlecolab utilization, etc.))
I want to create a lunch database [EP1] Django study for the first time
I want to create a lunch database [EP1-4] Django study for the first time
A story that I wanted to display the division result (%) on HTML with an application using django [Beginner learns python with a reference book in one hand]
A story that I wanted to realize the identification of parking lot fullness information using images obtained with a Web camera and Raspberry Pi and deep learning.
Create a web application that recognizes numbers with a neural network
The first step to creating a serverless application with Zappa
Create a web API that can deliver images with Django
I want to create a Dockerfile for the time being.
I tried to create serverless batch processing for the first time with DynamoDB and Step Functions
A story that didn't work when I tried to log in with the Python requests module
I made a web application that graphs the life log recorded on Google Home like a Gantt chart.
Create a BOT that can call images registered with Discord like pictograms
I made a web application that converts photos into Van Gogh's style
I had a hard time with ImportError: DLL load failed in tensorflow 2.0
Create a web app that can be easily visualized with Plotly Dash
The story of having a hard time introducing OpenCV with M1 MAC
A story I was addicted to trying to get a video url with tweepy
The story of developing a web application that automatically generates catchphrases [MeCab]
Play like a web app with ipywidgets
Create a simple web app with flask
I made a WEB application with Django
Let's make an A to B conversion web application with Flask! From scratch ...
The story of IPv6 address that I want to keep at a minimum
[ES Lab] I tried to develop a WEB application with Python and Flask ②
[kotlin] Create an app that recognizes photos taken with a camera on android
I made an appdo command to execute a command in the context of the app
Create an app that works well with people's reports using the COTOHA API
A story that I was addicted to when I made SFTP communication with python
I tried to create a model with the sample of Amazon SageMaker Autopilot
I made a plug-in from the Japan Meteorological Agency GPV to easily create an animated contour diagram with QGIS.
Create an app that guesses students with python
How to create a multi-platform app with kivy
I made a twitter app that decodes the characters of Pricone with heroku (failure)
A story that required preparation when trying to do a Django tutorial with plain centos7
I tried to make something like a chatbot with the Seq2Seq model of TensorFlow
A story that got stuck when trying to upgrade the Python version on GCE
I can't find the clocksource tsc! ?? The story of trying to write a kernel patch
The story of Linux that I want to teach myself half a year ago
I made a web application that maps IT event information with Vue and Flask
Try to create a waveform (audio spectrum) that moves according to the sound with python
Use Twitter API to reduce the time taken by Twitter (create a highlighting (like) timeline)
How to test the current time with Go (I made a very thin library)
When writing to a csv file with python, a story that I made a mistake and did not meet the delivery date
I made a program in Python that changes the 1-minute data of FX to an arbitrary time frame (1 hour frame, etc.)
A story that I wanted to do a function like before_action used in rails with django [Beginner learns python with a reference book]
A story that I did not know how to load a mixin when making a front with the django app [Beginners learn python with a reference book in one hand]