[PYTHON] Creating an API that returns negative-positive inference results using BERT in the Django REST framework

This post is the 20th day article of "Django Advent Calendar 2019 --Qiita".

This is siny.

In this article, I've summarized the challenges of creating a Rest API that returns negative-positive inference results using the ** Django REST framework ** and ** BERT model **.

• Since DRF and BERT were implemented for the first time this time, I would appreciate it if you could point out any mistakes or points such as "It is better to do this!".

Introduction

This article doesn't cover the parts involved in implementing the BERT model because Django is the main purpose. For implementation and learning of the BERT model using the Japanese dataset used in this article, see [** Natural language processing Advent Calendar 2019 Day 25 (Creation of negative / positive classifier using BERT) **] Please see https://qiita.com/ysiny/items/b01250228e0c5cc0e647) ".

First of all, it is a schematic diagram of the overall processing of the DRF environment assumed this time.

What you are doing is simple, it is a REST API that infers (binary classification) whether the sentence given as input data is negative or positive with a BERT model and returns the result to the client side.

• In the schematic diagram, Azure's data science environment (DSVM) is used, but this time we will build it using a local development server.

[API demo video]

The modules implemented in this article can be found in this git repository, so please download and use them as appropriate.

Regarding BERT, [Book "Learn while making! Deep learning by PyTorch"](https://www.amazon.co.jp/%E3%81%A4%E3%81%8F%E3%82%8A % E3% 81% AA% E3% 81% 8C% E3% 82% 89% E5% AD% A6% E3% 81% B6-PyTorch% E3% 81% AB% E3% 82% 88% E3% 82% 8B % E7% 99% BA% E5% B1% 95% E3% 83% 87% E3% 82% A3% E3% 83% BC% E3% 83% 97% E3% 83% A9% E3% 83% BC% E3 % 83% 8B% E3% 83% B3% E3% 82% B0-% E5% B0% 8F% E5% B7% 9D% E9% 9B% 84% E5% A4% AA% E9% 83% 8E / dp / It was created with reference to 4839970254). In the above book, the negative / positive classification of the BERT model was based on English data, so we have improved it so that it can be classified as negative / positive in the Japanese dataset based on the book.

table of contents

1. Premise
2. Environment construction
3. Create a Django REST framework
4. Inference using REST API
5. Simple tool
6. Summary
7. Reference books

1. Premise

The contents of this article have been confirmed to work in the following environments.

Local environment

2. Environment construction

This time we will build a DRF that works in the Ubuntu environment of Windows 10. First of all, create a virtual environment and install the necessary modules with conda.

Installation of various modules

conda create -n drf python=3.6
conda activate drf
conda install pytorch=0.4 torchvision -c pytorch
conda install pytorch=0.4 torchvision cudatoolkit -c pytorch
conda install pandas scikit-learn django

If conda doesn't work, install it with pip.

pip install mojimoji
pip install attrdict
pip install torchtext
pip install pyknp
pip install djangorestframework

Juman ++ installation

The BERT Japanese Pretrained model used this time uses Human ++ (v2.0.0-rc2) for morphological analysis of the input text, so this article also matches the morphological analysis tool to ** Human ++**. The procedure for installing Juman ++ is summarized in a separate article, so please refer to the following.

[** Summary of JUMAN ++ installation procedure **] https://sinyblog.com/deaplearning/juman/

After installing Juman ++, it is OK if the morphological analysis is available in the local environment as follows.

#JUMAN operation check
from pyknp import Juman					
text = "I'm learning about natural language processing."					
juman = Juman()					
result =juman.analysis(text)					
result = [mrph.midasi for mrph in result.mrph_list()]					
print(text)
I'm learning about natural language processing.
print(result)
['Nature', 'language', 'processing', 'To', 'about', 'Learning', 'During ~', 'is', '。']			

3. Create a Django REST framework

Project creation

First, create a django project. (Project name: drf)

django-admin startproject drf

Application creation

Next, create an application (application name: appv1)

cd drf
python manage.py startapp appv1

Customization of settings.py

Add rest_framework and application (appv1) to INSTALLED_APPS in settings.py.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',                #add
    'appv1.apps.Appv1Config',        #add

]

Placement of BERT related modules

Create the following folder directly under the application folder (appv1) and place the module as specified.

• If you want to use the trained model file (bert_fine_tuning_chABSA.pth) for the time being, download the trained model file from here. Please place it.
• Vob.txt, bert_config.json, pytorch_model.bin are [HP published by Kyoto University](http://nlp.ist.i.kyoto-u.ac.jp/index.php?BERT%E6%97%A5 You can download it from% E6% 9C% AC% E8% AA% 9EPretrained% E3% 83% A2% E3% 83% 87% E3% 83% AB).

Place the following files directly under the application (appv1).

Start Django's shell mode and generate a dictionary data file for inference.

python manage.py shell
from appv1.config import *
from appv1.predict import create_vocab_text
TEXT = create_vocab_text()

Executing the above will generate appv1 / data / text.pkl.

The entire directory structure is as follows.

├─drf
│  │  db.sqlite3
│  │  manage.py
│  │
│  ├─appv1
│  │  │  admin.py
│  │  │  apps.py
│  │  │  bert.py           #BERT model definition
│  │  │  config.py         #Various configuration files
│  │  │  dataloader.py     #torchtext data loader generation file
│  │  │  models.py
│  │  │  predict.py        #For reasoning
│  │  │  serializers.py    #Serializer
│  │  │  tests.py
│  │  │  tokenizer.py      #BERT word split related shell
│  │  │  views.py
│  │  ├─data

│  │  │      test_dumy.tsv  #Dummy data
│  │  │      train_dumy.tsv #Dummy data
│  │  │      text.pkl    #Wordbook data used for inference
│  │  │
│  │  ├─vocab
│  │  │      vocab.txt   #Bert lexicon data
│  │  │
│  │  ├─weights
│  │  │      bert_config.json
│  │  │      bert_fine_tuning_chABSA.pth  #Fine-tuned Bert model
│  │  │      pytorch_model.bin
│  │  │
│  ├─drf
│  │  │  settings.py
│  │  │  urls.py
│  │  │  wsgi.py

Inference operation check

Make sure that the inference process works properly using the BERT trained model in the django environment.

After starting Django's shell mode, execute the following command to give sample text data and check that negative or positive judgment can be performed.

• See comments for the processing performed by each command.

python manage.py shell
-----------------------------------------------------------------------------
#Shell mode from here
In [1]: from appv1.config import *
In [2]: from appv1.predict import predict2, create_vocab_text, build_bert_model
In [3]: from appv1.bert import get_config, BertModel,BertForchABSA, set_learned_params
In [4]: import torch
In [5]: config = get_config(file_path=BERT_CONFIG)  #Loading BERT config settings
In [6]: net_bert = BertModel(config)   #BERT model generation
In [7]: net_trained = BertForchABSA(net_bert)   #Combine negative / positive classifier with BERT model
In [8]: net_trained.load_state_dict(torch.load(MODEL_FILE, map_location='cpu'))  #Low learned weight
   ...:Do
Out[8]: IncompatibleKeys(missing_keys=[], unexpected_keys=[])
In [9]: net_trained.eval()   #Set to inference mode
Out[9]:
BertForchABSA(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (word_embeddings): Embedding(32006, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): BertLayerNorm()
      (dropout): Dropout(p=0.1)
    )
    (encoder): BertEncoder(
      (layer): ModuleList(
        (0): BertLayer(
          (attention): BertAttention(
            (selfattn): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1)
            )

 ~~Since there are many output results, some are omitted.

    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (cls): Linear(in_features=768, out_features=2, bias=True)
)

In [10]: input_text = "In terms of profit and loss, ordinary income was due to a decrease in interest on loans and gains on sales of securities.
    ...:It decreased by 7,273 million yen to 67,413 million yen."
In [11]: result = predict2(input_text, net_trained).numpy()[0]  #Executing inference(The return value is negative or positive
    ...:Tib)
['[UNK]', 'surface', 'To', 'Oki', 'not to mention', 'Is', '、', '[UNK]', 'Revenue', 'Is', '、', 'Lending', 'Money', 'Interest', 'Or', 'Valuable', 'Securities', 'sale', 'Profit', 'of', 'Decrease', 'To', 'Than', '、', 'First term', 'ratio', '[UNK]', 'Circle', 'Decrease', 'of', '[UNK]', 'Circle', 'When', 'Nari', 'Was']
[2, 1, 534, 8, 7779, 26207, 9, 6, 1, 7919, 9, 6, 15123, 306, 28611, 34, 27042, 4190, 3305, 8995, 5, 1586, 8, 52, 6, 4523, 2460, 1, 387, 1586, 5, 1, 387, 12, 105, 4561, 3]
In [12]: print(result)
0

As mentioned above, if a negative (0) or positive (1) is returned as a return value in the result variable without any error, the operation is OK.

Create serializer

Next, create a DRF serializer. In addition, serialization and deserialization in DRF are the following processes.

In this case, we don't deal with Django models, so we don't convert them to model objects. Instead, create a serializer that says "** Receive the input text you want to make a negative / positive judgment, input it to the BERT trained model, and output the inference result **".

There are three main types of DRF serializers, but this time we want to implement model-independent customized processing, so we will use the ** Serializer class ** of rest_framework.

|serializer processing|meaning| |:--|:--|:--|　 | ModelSerializer　　　　　　　|Utilize a single model object| |Serializer|Handle a single resource or implement model-independent customized processing| |ListSerializer|Handle multiple resources|

Create a file called serializers.py directly under appv1 and add the following code.

appv1\serializers.py

from rest_framework import serializers
from appv1.config import *	
from appv1.predict import predict2
from appv1.bert import get_config, BertModel,BertForchABSA	
import torch		


class BertPredictSerializer(serializers.Serializer):
    """Serializer to obtain BERT negative / positive classification results"""

    input_text = serializers.CharField()
    neg_pos = serializers.SerializerMethodField()

    def get_neg_pos(self, obj):
        config = get_config(file_path=BERT_CONFIG)  #Loading the bert config file
        net_bert = BertModel(config)  #BERT model generation
        net_trained = BertForchABSA(net_bert) # #Combine negative / positive classifier with BERT model
        net_trained.load_state_dict(torch.load(MODEL_FILE, map_location='cpu'))  #Load learned weights
        net_trained.eval()  #Set to inference mode
        label = predict2(obj['input_text'], net_trained).numpy()[0]  #Get inference results
        return label

First, create a serializer called ** BertPredictSerializer ** by inheriting ** serializers.Serializer ** from rest_framework.

class BertPredictSerializer(serializers.Serializer):
    """Serializer to obtain BERT negative / positive classification results"""

    input_text = serializers.CharField()
    neg_pos = serializers.SerializerMethodField()

A string type CharField is defined as input (** input_text **). Since the output is a dynamic value (inference result), the field value is defined by ** serializers.SerializerMethodField () **, which can determine the value of the field according to the result of the method. ..

When using SerializerMethodField (), define the applied method name as ** get_ + field name **. In this case, we define the ** get_neg_pos ** method.

If you define such a serializer, the get_neg_pos method will be executed when you POST the input text to the API view using BertPredictSerializer, and the processing result will be returned as a JSON string.

This time, we want to give the input sentence data to the BERT learning model and obtain the negative / positive judgment result, so the following processing is described in the get_neg_pos method.

    def get_neg_pos(self, obj):
        config = get_config(file_path=BERT_CONFIG)  #Loading the bert config file
        net_bert = BertModel(config)  #BERT model generation
        net_trained = BertForchABSA(net_bert) # #Combine negative / positive classifier with BERT model
        net_trained.load_state_dict(torch.load(MODEL_FILE, map_location='cpu'))  #Load learned weights
        net_trained.eval()  #Set to inference mode
        label = predict2(obj['input_text'], net_trained).numpy()[0]  #Get inference results
        return label

You can see what we are doing by looking at the comments, but I will add it below.

-** get_config ** method: Read from bert_config.json and convert JSON dictionary variables to object variables -** BertModel ** class: Class that generates BERT model -** BertForchABSA ** Class: A model in which the BERT model (BertModel) is connected to the part that determines the negative / positive of chABSA. -** predict2 ** method: A method that infers whether the input sentence is negative or positive and returns 0 or 1.

The above four method classes are defined in ** bert.py **. Load the trained model parameters with ** net_trained.load_state_dict **. ** predict2 ** method is defined in ** predict.py **, and ** inference when input sentence ** is passed as the first argument and ** trained model instance ** is passed as the second argument. It is a method ** that returns the result (0 or 1).

This completes the serializer creation.

Create view

The DRF view can be implemented as either a class-based view or a function-based view.

This time I tried using a class-based view called GenericAPIView.

from django.shortcuts import render
from rest_framework import generics, status
from rest_framework.response import Response
from appv1.serializers import BertPredictSerializer


class BertPredictAPIView(generics.GenericAPIView):
    """BERT Negative-Positive Classification Prediction Class"""
    serializer_class = BertPredictSerializer

    def post(self, request, *args, **kwargs):     

        serializer = self.get_serializer(request.data)
        return Response(serializer.data, status=status.HTTP_200_OK)

It inherits ** GenericAPIView ** and defines ** BertPredictAPIView **. Specify the serialization class (BertPredictSerialzer) defined in serializer.py in the ** serializer_class ** attribute. Then define the ** post ** method inside the BertPredictAPIView and internally use the GenericAPIView's ** get_serializer ** method to get the serializer instance used for Validation. Specify ** request.data ** as an argument.

serializer = self.get_serializer(request.data)

** request.data ** works with "POST", "PUT", and "PATCH" methods to handle arbitrary data. (Function similar to request.POST) Finally, specify ** serializer.data ** as the argument of ** Response ** to complete the view that returns the inference result.

Definition of urls.py

Finally, add the URL definition in drf \ urlspy.

from django.contrib import admin
from django.urls import path
from appv1 import views  #add


urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/v1/predict/', views.BertPredictAPIView.as_view()),   #add
]

The BertPredict API View defined in views.py is defined as a URL pattern called api / v1 / predict. You have now defined ** http://127.0.0.1/api/v1/predict ** as the Rest API endpoint.

At this point, execute djagno migration once and execute runserver.

python manage.py migrate
python manage.py runserver

After that, when you access ** http://127.0.0.1:8000/api/v1/predict/**, the following screen will be displayed.

Enter the text you want to judge negative / positive in "** input text **" at the bottom of the screen and press the ** POST ** button to make inferences using BERT's fine-tuning trained model and return the result. ..

The screen below shows the result of pressing the POST button after entering the text "In Tokyu Community Corp., management stock has expanded for both condominiums and buildings, resulting in higher sales and profits."

As a return value, ** input_text ** and ** neg_pos (0 or 1) ** representing negative / positive are returned.

In the above example, positive (= 1) is returned as the inference result.

4. Inference using REST API

Next, try to CALL the created REST API from the local client and receive the inference result. This time, the client and server are simply executed on the same PC.

After starting the development server with python manage.py runserver, open the DOS screen etc. in another window and execute the python command to make the python code executable and execute the following code.

import urllib.request
import urllib.parse
import json
def predict(input_text):
    URL = "http://127.0.0.1:8000/api/v1/predict/"
    values = {
        "format": "json",
        "input_text": input_text,
        }
    data = urllib.parse.urlencode({'input_text': input_text}).encode('utf-8')
    request = urllib.request.Request(URL, data)
    response = urllib.request.urlopen(request)
    result= json.loads(response.read())
    return result

input_text = "In terms of profit and loss, ordinary income decreased by 7,273 million yen from the previous fiscal year to 67,413 million yen due to a decrease in interest on loans and gain on sales of securities."
result = predict(input_text)
print(result)
{'input_text': 'In terms of profit and loss, ordinary income decreased by 7,273 million yen from the previous fiscal year to 67,413 million yen due to a decrease in interest on loans and gain on sales of securities.', 'neg_pos': 0}
print(result['input_text'])
In terms of profit and loss, ordinary income decreased by 7,273 million yen from the previous fiscal year to 67,413 million yen due to a decrease in interest on loans and gain on sales of securities.
print(result['neg_pos'])
0

Specify ** "http://127.0.0.1:8000/api/v1/predict/"** as the endpoint. Give json as format and input_text as input data to the dictionary variable values.

After that, if you throw it with the POST method together with the sentence data encoded for the endpoint using urllib.request.Request, the BertPredictAPIView class defined in views.py will be called and the process will move in the flow of serialization → inference execution. To go. I will convert the processing result with json.loads so that it can be handled as dictionary type data on the python side.

Then, it will be converted to dictionary type data as shown below, so you can access the information you want by key name (input_text, neg_pos).

{'input_text': 'In terms of profit and loss, ordinary income decreased by 7,273 million yen from the previous fiscal year to 67,413 million yen due to a decrease in interest on loans and gain on sales of securities.', 'neg_pos': 0}

5. Simple tool

Finally, create a simple command that automatically throws a large amount of input data to the REST API and outputs the inference result. The csv files and programs used below are under django-drf-dl / drf / tools / in the git repository.

Prepare the following test.csv file that you want to predict. (Column name is INPUT)

Place predict.py with the following code in the same folder as test.csv.

import pandas as pd
import numpy as np
import urllib.request
import urllib.parse
import json

def predict(input_text):
    URL = "http://127.0.0.1:8000/api/v1/predict/"
    values = {
        "format": "json",
        "input_text": input_text,
            }
    data = urllib.parse.urlencode({'input_text': input_text}).encode('utf-8')
    request = urllib.request.Request(URL, data)
    response = urllib.request.urlopen(request)
    result= json.loads(response.read())
    return result['neg_pos'][1]	

if __name__ == '__main__':
    print("Start if __name__ == '__main__'")
    print('load csv file ....')
    df = pd.read_csv("test.csv", engine="python", encoding="utf-8-sig")
    df["PREDICT"] = np.nan   #Add prediction column
    print('Getting prediction results ....')
    for index, row in df.iterrows():
        df.at[index, "PREDICT"] = predict(row['INPUT'])
    print('save results to csv file')
    df.to_csv("predicted_test .csv", encoding="utf-8-sig", index=False)
    print('Processing terminated normally.')

When you start the DOS screen etc. and execute the following command, test.csv is read line by line and thrown to the REST API to receive the inference result.

python predict.py
----------------------------------
Start if __name__ == '__main__'
load csv file ....
Getting prediction results ....
save results to csv file
Processing terminated normally.

When the last data is completed, predicted_test.csv with inference result will be generated in the same folder.

6. Summary

This time, I created a REST API that judges simple negatives and positives using a binary classification model of DRF and BERT in the local environment. In the future, I would like to build a REST API on the Azure platform and apply it to tasks such as multi-classification and FAQ instead of binary classification.

7. Reference books

The DRF created in this article was created by applying a little to the content in Django REST Framework textbook that can be used in the field. I learned DRF with this book for the first time this time, but it is a very helpful book, so it is recommended for those who want to learn DRF from now on.

Tomorrow is the 21st day article of "Django Advent Calendar 2019 --Qiita" by ssh22. Thank you!