[PYTHON] I tried to make an image classification BOT by combining TensorFlow Lite and LINE Messaging API

As you study how to use machine learning libraries such as TensorFlow, you will think "I want to make something that works!". This time, I would like to create an image classification BOT by combining TensorFlow Lite, LINE Messaging API, and Heroku.

Why use Heroku

There are various choices such as AWS and GCP for the deployment destination of the created application. Heroku comes with a free tier that makes it relatively easy to get started. The free tier has restrictions such as the number of disguise environments that can be started and the usage time limit per month. It is recommended as an environment for deploying the sample APP.

Reasons to use TensorFlow Lite

This article uses Heroku's free tier as the deployment destination. In the free tier, the storage capacity is limited to about 300MB. Therefore, we use TensorFlow Lite, which can reduce the capacity of the model.

Reasons to use the LINE Messaging API

I think there are various apps that incorporate machine learning, such as Android and iOS smartphone apps and WEB apps. TensorFlow has libraries for Android and iOS environments, Of course, knowledge of application development is required to use them. You can also use TensorFlow.js to create apps in your browser or Node.js.

The app created this time is to "classify what is reflected in the image sent from the user". If you want to create it as a smartphone app, you need to develop both the smartphone app and the backend. I decided to use the LINE Messaging API because it is not an app that runs in a browser and I want to develop an app with a minimum of code.

Try to create a parrot return bot

I would like to make a simple parrot return BOT to check the operation of the BOT. I would like to briefly explain the operation flow of the LINE Messaging API.

1. The user sends a message to the LINE platform.
2. The LINE platform sends an HTTP POST request containing the event object to the BOT server.
3. Verification is performed on the BOT server to confirm that the request was sent from the LINE platform.
4. After clearing the verification, you will be able to acquire the content (text message, image message, etc.) sent by the user.
5. Perform the necessary processing for the user request and send the message.

You may find it difficult to implement just by looking at this flow, but it is relatively easy to implement using the LINE Messaging API SDK. What actually needs to be discontinued is the "processing of messages sent by users" part.

Please check the official document for the sample code of LINE Messaging API SDK and Echolalia BOT. This time I'm using Python, but there are also SDKs for various languages ​​such as JAVA, PHP, GO, and Node.js. LINE Messaging API SDK https://github.com/line/line-bot-sdk-python

The created Echolalia BOT can be used from LINE by deploying it on Heroku. How to deploy to Heroku will be another long one, so I'd like to write another article.

Implementation of image classification BOT

The image classification BOT created this time can be implemented by applying the parrot return BOT. As a part to actually implement,

1. Get the image sent by the user.
2. Label what is shown in the image with the image classification model.
3. Send the label as a text message to the user.

The implemented code is below.

#Model loading
model = Interpreter(model_path='Model path')
model.allocate_tensors()

#Get model input and output shapes
input_details = model.get_input_details()[0]
output_details = model.get_output_details()[0]

@handler.add(MessageEvent, message=ImageMessage)
def handle_image(event):

    #Get the content sent by the user
    image_url = f'https://api.line.me/v2/bot/message/{event.message.id}/content/'
    header = {
        "Content-Type": "application/json",
        "Authorization": "Bearer " + YOUR_CHANNEL_ACCESS_TOKEN
    }
    res = requests.get(image_url, headers=header)

    #Image preprocessing
    image = Image.open(BytesIO(res.content))
    input_data = process_image(input_details['shape'], image)

    #Predict the label of what is in the image
    model.set_tensor(input_details["index"], input_data)
    model.invoke()
    label = model.tensor(output_details["index"])().argmax()

    #Send the image label to the user by text message.
    line_bot_api.reply_message(
            event.reply_token,
            TextSendMessage(text=f'maybe...{label}'))

It can be operated as an image classification BOT by replacing the following part of the parrot return BOT.

@handler.add(MessageEvent, message=TextMessage)
def handle_message(event):
    line_bot_api.reply_message(
        event.reply_token,
        TextSendMessage(text=event.message.text))

sample: