[PYTHON] [Natural language processing] I tried to visualize the hot topics this week in the Slack community

** Example sentence word-separation (noun only): "Private baseball" **

It would be even better.

• Isn't it better to erase "I" as well? I thought you there. See the "Stopword Removal" chapter

5.3. Implementation of morphological analysis and word-separation

When implementing morphological analysis,

--What to use for the morphological analysis library --What to use as dictionary data

We have to decide that.

This time, I did the following.

--Morphological analysis library: Janome --Dictionary data: Janome default (NEologd for new words is even better)

In addition, the part of speech to be extracted is "morphological analysis tool part of speech system", and what is necessary to achieve the purpose is something? I thought from the viewpoint.

The official mascot, Janome, is cute. (I don't know if Janome is the name)

morphological_analysis.py

from janome.tokenizer import Tokenizer
from tqdm import tqdm
import pandas as pd
import argparse
from pathlib import Path
import sys


exc_part_of_speech = {
    "noun": ["Non-independent", "代noun", "number"]
}
inc_part_of_speech = {
    "noun": ["Change connection", "General", "固有noun"],
}


class MorphologicalAnalysis:

    def __init__(self):
        self.janome_tokenizer = Tokenizer()

    def tokenize_janome(self, line: str) -> list:
        # list of janome.tokenizer.Token
        tokens = self.janome_tokenizer.tokenize(line)
        return tokens

    def exists_pos_in_dict(self, pos0: str, pos1: str, pos_dict: dict) -> bool:
        # Retrurn where pos0, pos1 are in pos_dict or not.
        # ** pos = part of speech
        for type0 in pos_dict.keys():
            if pos0 == type0:
                for type1 in pos_dict[type0]:
                    if pos1 == type1:
                        return True
        return False

    def get_wakati_str(self, line: str, exclude_pos: dict,
                       include_pos: dict) -> str:
        '''
        exclude/include_pos is like this
        {"noun": ["Non-independent", "代noun", "number"], "adjective": ["xxx", "yyy"]}
        '''
        tokens = self.janome_tokenizer.tokenize(line, stream=True)  #Generator to save memory
        extracted_words = []
        for token in tokens:
            part_of_speech0 = token.part_of_speech.split(',')[0]
            part_of_speech1 = token.part_of_speech.split(',')[1]
            # check for excluding words
            exists = self.exists_pos_in_dict(part_of_speech0, part_of_speech1,
                                             exclude_pos)
            if exists:
                continue
            # check for including words
            exists = self.exists_pos_in_dict(part_of_speech0, part_of_speech1,
                                             include_pos)
            if not exists:
                continue
            # append(Acquired surface layer shape to absorb notation fluctuation)
            extracted_words.append(token.surface)
        # wakati string with extracted words
        wakati_str = ' '.join(extracted_words)
        return wakati_str


def make_wakati_for_lines(msg_ser: pd.Series) -> pd.Series:
    manalyzer = MorphologicalAnalysis()
    wakati_msg_list = []
    for msg in tqdm(msg_ser, desc='[mk wakati msgs]'):
        wakati_msg = manalyzer.get_wakati_str(str(msg), exc_part_of_speech,
                                              inc_part_of_speech)
        wakati_msg_list.append(wakati_msg)
    wakati_msg_ser = pd.Series(wakati_msg_list)
    return wakati_msg_ser


def main(input_fname: str):
    input_root = '../../data/020_intermediate'
    output_root = '../../data/030_processed'
    # 1. load messages_cleaned.csv
    msgs_cleaned_fpath = input_root + '/' + input_fname
    df_msgs = pd.read_csv(msgs_cleaned_fpath)
    # 2. make wakati string by record
    ser_msg = df_msgs.msg
    df_msgs['wakati_msg'] = make_wakati_for_lines(ser_msg)
    # 3. save it
    pin = Path(msgs_cleaned_fpath)
    msgs_wakati_fpath = output_root + '/' + pin.stem + '_wakati.csv'
    df_msgs.to_csv(msgs_wakati_fpath, index=False)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("input_fname", help="set input file name", type=str)
    args = parser.parse_args()
    input_fname = args.input_fname
    # input file must been cleaned
    if 'cleaned' not in input_fname:
        print('input file name is invalid.: {0}'.format(input_fname))
        print('input file name must include \'cleaned\'')
        sys.exit(1)
    main(input_fname)

6. Preprocessing: Normalization

6.1. What is normalization?

Normalization in the preprocessing of natural language processing refers to the following processing. It is also called "name identification".

1. Unification of character types
2. Unify Kana to full-width
3. Alphabet is unified to lowercase ... etc.
4. Number replacement
5. Replace all numbers with 0, etc.
6. • There are few situations where numbers are important in natural language processing.
7. Unification of words using a dictionary
8. Judge "Sony" and "Sony" as the same, and unify the notation as "Sony"

The world of normalization is deep, so I'll leave it here.

6.2. Implementation of normalization

This time, for the sake of simplicity, only the following processing was implemented. It's insanely easy.

1. Unify the alphabet to lowercase
2. Replace all numbers with 0

normalization.py

import re
import pandas as pd
from tqdm import tqdm
import argparse
from pathlib import Path
import sys


def normarize_text(text: str):
    normalized_text = normalize_number(text)
    normalized_text = lower_text(normalized_text)
    return normalized_text


def normalize_number(text: str) -> str:
    """
    pattern = r'\d+'
    replacer = re.compile(pattern)
    result = replacer.sub('0', text)
    """
    #Replace consecutive numbers with 0
    replaced_text = re.sub(r'\d+', '0', text)
    return replaced_text


def lower_text(text: str) -> str:
    return text.lower()


def normalize_msgs(wktmsg_ser: pd.Series) -> pd.Series:
    normalized_msg_list = []
    for wktstr in tqdm(wktmsg_ser, desc='normalize words...'):
        normalized = normarize_text(str(wktstr))
        normalized_msg_list.append(normalized)
    normalized_msg_ser = pd.Series(normalized_msg_list)
    return normalized_msg_ser


def main(input_fname: str):
    input_root = '../../data/030_processed'
    output_root = input_root
    # 1. load wakati messages
    msgs_fpath = input_root + '/' + input_fname
    df_msgs = pd.read_csv(msgs_fpath)
    # 2. normalize wakati_msg (update)
    ser_msg = df_msgs.wakati_msg
    df_msgs.wakati_msg = normalize_msgs(ser_msg)
    # 3. save it
    pin = Path(msgs_fpath)
    msgs_ofpath = output_root + '/' + pin.stem + '_norm.csv'
    df_msgs.to_csv(msgs_ofpath, index=False)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("input_fname", help="set input file name", type=str)
    args = parser.parse_args()
    input_fname = args.input_fname
    # input file must been cleaned
    if 'wakati' not in input_fname:
        print('input file name is invalid.: {0}'.format(input_fname))
        print('input file name must include \'wakati\'')
        sys.exit(1)
    main(input_fname)

7. Pretreatment: Stop word removal

7.1. What is Stopword Removal?

"Stop word removal" is the process of removing stop words. (Too much as it is ...)

So what is a "** stop word **"?

According to the goo Japanese dictionary

Words that are excluded from the search by themselves because they are so common in full-text search. Refers to "ha", "no", "desu" in Japanese, and a, the, of in English. Stopwords.

Natural language processing tasks often have the purpose of "understanding the context of a sentence." Stopwords are unnecessary for this purpose and should be removed.

7.2. How to decide the stop word?

There are two main ones. This time we will use 1.

1. Method using a dictionary (** ← This time is adopted **)
2. Method using frequency of appearance

For dictionary data, use here.

There are several reasons for choosing method 1.

--When using a dictionary Existing dictionary makes it easy to install. --When using the frequency of occurrence, it is necessary to extract only "words that occur frequently in ** general conversation ** and therefore should be removed". If you use only the Slack data you have at hand to determine the frequency of appearance, ** there is a risk that "words related to exciting topics" will be unintentionally removed **, so I thought that it was necessary to prepare separate data and aggregate it. .. ...... It's a little difficult

7.3. Implementation of stopword removal

Removes the words registered in the dictionary data introduced in the previous section. In addition, the following characters have also been removed. It's a word that I thought was annoying while tuning various things.

• 「-」 ―― “ー”
• 「w」
• 「m」
• "Lol"

stopword_removal.py

import pandas as pd
import urllib.request
from pathlib import Path
from tqdm import tqdm
import argparse
from pathlib import Path
import sys


def maybe_download(path: str):
    stopword_def_page_url = 'http://svn.sourceforge.jp/svnroot/slothlib/CSharp/Version1/SlothLib/NLP/Filter/StopWord/word/Japanese.txt'
    p = Path(path)
    if p.exists():
        print('File already exists.')
    else:
        print('downloading stop words definition ...')
        # Download the file from `url` and save it locally under `file_name`:
        urllib.request.urlretrieve(stopword_def_page_url, path)
    #stop word added
    sw_added_list = [
        '-',
        '-',
        'w',
        'W',
        'm',
        'Lol'
    ]
    sw_added_str = '\n'.join(sw_added_list)
    with open(path, 'a') as f:
        print(sw_added_str, file=f)


def load_sw_definition(path: str) -> list:
    with open(path, 'r', encoding='utf-8') as f:
        lines = f.read()
        line_list = lines.split('\n')
        line_list = [x for x in line_list if x != '']
    return line_list


def remove_sw_from_text(wktstr: str, stopwords: list) -> str:
    words_list = wktstr.split(' ')
    words_list_swrm = [x for x in words_list if x not in stopwords]
    swremoved_str = ' '.join(words_list_swrm)
    return swremoved_str


def remove_sw_from_msgs(wktmsg_ser: pd.Series, stopwords: list) -> pd.Series:
    swremved_msg_list = []
    for wktstr in tqdm(wktmsg_ser, desc='remove stopwords...'):
        removed_str = remove_sw_from_text(str(wktstr), stopwords)
        swremved_msg_list.append(removed_str)
    swremved_msg_ser = pd.Series(swremved_msg_list)
    return swremved_msg_ser


def main(input_fname: str):
    input_root = '../../data/030_processed'
    output_root = input_root
    # 1. load stop words
    sw_def_fpath = 'stopwords.txt'
    maybe_download(sw_def_fpath)
    stopwords = load_sw_definition(sw_def_fpath)
    # 2. load messages
    msgs_fpath = input_root + '/' + input_fname
    df_msgs = pd.read_csv(msgs_fpath)
    # 3. remove stop words
    ser_msg = df_msgs.wakati_msg
    df_msgs.wakati_msg = remove_sw_from_msgs(ser_msg, stopwords)
    # 4. save it
    pin = Path(msgs_fpath)
    msgs_ofpath = output_root + '/' + pin.stem + '_rmsw.csv'
    df_msgs.to_csv(msgs_ofpath, index=False)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("input_fname", help="set input file name", type=str)
    args = parser.parse_args()
    input_fname = args.input_fname
    # input file must been cleaned
    if 'norm' not in input_fname:
        print('input file name is invalid.: {0}'.format(input_fname))
        print('input file name must include \'norm\'')
        sys.exit(1)
    main(input_fname)

8. Preprocessing: Important word extraction (tf-idf)

8.1. What is tf-idf?

There are many great commentary articles. I referred to this article. TF-IDF | Qiita

Here's a rough explanation.

• tf (Term Frequency) --Frequency of word occurrence in a document ――If it is big, "the word often appears in the document"
• idf (Inverse Document Frequency) --Reciprocal of ** (to all documents) of documents in which a word appears ** ――If it's big, it doesn't appear much in other documents.

tf-idf is the product of tf and idf. In other words ** Large tf-idf ** = ** Frequently appearing in one document ** & ** Not often appearing in other documents ** = Important for understanding the ** context of the document **

8.2. Implementation of word scoring processing by tf-idf

8.2.1. What should be a document / all documents?

The purpose of this time is to see the characteristics of remarks for a certain period (1 week). Therefore, I thought that it should be possible to understand ** what characteristics the remarks of a certain period (1 week) have for all the posts so far **.

Therefore,

-** All documents : All posts so far - 1 document **: A mass of posts for a certain period (1 week)

I calculated tf-idf as.

8.2.2. Implementation

Write the process flow easily

1. Read the message mart table (preprocessed)
2. Grouping by message period
3. Divide and group past data in 7-day units starting from the time of processing execution
4. Calculate tf-idf with one group of messages as one document
5. Extract words whose tf-idf score is above the threshold (output as a dictionary)

important_word_extraction.py

import pandas as pd
import json
from datetime import datetime, date, timedelta, timezone
from pathlib import Path
from sklearn.feature_extraction.text import TfidfVectorizer
JST = timezone(timedelta(hours=+9), 'JST')

#Grouping messages by a specified period
def group_msgs_by_term(df_msgs: pd.DataFrame, term: str) -> dict:
    # set term
    term_days = 8
    if term == 'lm':
        term_days = 31
    print('group messages every {0} days'.format(term_days))
    # analyze timestamp
    now_in_sec = (datetime.now(JST) - datetime.fromtimestamp(0, JST)).total_seconds()
    interval_days = timedelta(days=term_days)
    interval_seconds = interval_days.total_seconds()
    oldest_timestamp = df_msgs.min().timestamp
    oldest_ts_in_sec = (datetime.fromtimestamp(oldest_timestamp, JST) - datetime.fromtimestamp(0, JST)).total_seconds()
    loop_num = (abs(now_in_sec - oldest_ts_in_sec) / interval_seconds) + 1
    # extract by term
    dict_msgs_by_term = {}
    df_tmp = df_msgs
    now_tmp = now_in_sec
    for i in range(int(loop_num)):
        # make current term string
        cur_term_s = 'term_ago_{0}'.format(str(i).zfill(3))
        print(cur_term_s)
        # current messages
        df_msgs_cur = df_tmp.query('@now_tmp - timestamp < @interval_seconds')
        df_msgs_other = df_tmp.query('@now_tmp - timestamp >= @interval_seconds')
        # messages does not exist. break.
        if df_msgs_cur.shape[0] == 0:
            break
        # add current messages to dict
        dict_msgs_by_term[cur_term_s] = ' '.join(df_msgs_cur.wakati_msg.dropna().values.tolist())
        # update temp value for next loop
        now_tmp = now_tmp - interval_seconds
        df_tmp = df_msgs_other
    return dict_msgs_by_term

# tf-Extract important words and return them as a dictionary while referring to the idf score
def extract_important_word_by_key(feature_names: list, bow_df: pd.DataFrame, uids: list) -> dict:
    # >Look at each line and extract important words(top X words in tfidf)
    dict_important_words_by_user = {}
    for uid, (i, scores) in zip(uids, bow_df.iterrows()):
        #Create a table of the user's words and tfidf scores
        words_score_tbl = pd.DataFrame()
        words_score_tbl['scores'] = scores
        words_score_tbl['words'] = feature_names
        #Sort in descending order by tfidf score
        words_score_tbl = words_score_tbl.sort_values('scores', ascending=False)
        words_score_tbl = words_score_tbl.reset_index()
        # extract : tf-idf score > 0.001
        important_words = words_score_tbl.query('scores > 0.001')
        #Creating a dictionary for the user'uid0': {'w0': 0.9, 'w1': 0.87}
        d = {}
        for i, row in important_words.iterrows():
            d[row.words] = row.scores
        #Add to table only if the user's dictionary has at least one word
        if len(d.keys()) > 0:
            dict_important_words_by_user[uid] = d
    return dict_important_words_by_user

#Extract important words in specified period units
def extraction_by_term(input_root: str, output_root: str, term: str) -> dict:
    # ---------------------------------------------
    # 1. load messages (processed)
    # ---------------------------------------------
    print('load msgs (all of history and last term) ...')
    msg_fpath = input_root + '/' + 'messages_cleaned_wakati_norm_rmsw.csv'
    df_msgs_all = pd.read_csv(msg_fpath)
    # ---------------------------------------------
    # 2. group messages by term
    # ---------------------------------------------
    print('group messages by term and save it.')
    msgs_grouped_by_term = group_msgs_by_term(df_msgs_all, term)
    msg_grouped_fpath = input_root + '/' + 'messages_grouped_by_term.json'
    with open(msg_grouped_fpath, 'w', encoding='utf-8') as f:
        json.dump(msgs_grouped_by_term, f, ensure_ascii=False, indent=4)
    # ---------------------------------------------
    # 3.Tf for all documents-idf calculation
    # ---------------------------------------------
    print('tfidf vectorizing ...')
    # >Words in all documents are columns, the number of documents (=A matrix with user) as the row is created. Tf for each element-There is an idf value
    tfidf_vectorizer = TfidfVectorizer(token_pattern=u'(?u)\\b\\w+\\b')

    bow_vec = tfidf_vectorizer.fit_transform(msgs_grouped_by_term.values())
    bow_array = bow_vec.toarray()
    bow_df = pd.DataFrame(bow_array,
                        index=msgs_grouped_by_term.keys(),
                        columns=tfidf_vectorizer.get_feature_names())
    # ---------------------------------------------
    # 5. tf-Extract important words based on idf
    # ---------------------------------------------
    print('extract important words ...')
    dict_word_score_by_term = extract_important_word_by_key(
        tfidf_vectorizer.get_feature_names(),
        bow_df, msgs_grouped_by_term.keys())
    return dict_word_score_by_term

9. Visualization process with Wordcloud

9.1. What is Wordcloud?

Words with high scores are large images, and words with low scores are small images. Various values such as "appearance frequency" and "importance" can be freely set for the score.

Official repository: amueller / word_cloud: octocat:

9.2. Prepare Wordcloud fonts

This time, I will use this. What is Homemade Rounded M +

9.3. Implementation of Wordcloud

In the previous chapter "8. Preprocessing: Extracting important words (tf-idf)", the following JSON file was output.

important_word_tfidf_by_term.json

{
  "term_ago_000": {
    "data": 0.890021,
    "game": 0.780122,
    "article": 0.720025,
    :
  },
  "term_ago_001": {
    "translation": 0.680021,
    "data": 0.620122,
    "deepl": 0.580025,
    :
  },
  :
}

Load this and create a Wordcloud image. Use the method WordCloud.generate_from_frequencies ().

wordcloud_from_score.py

from wordcloud import WordCloud
import matplotlib.pyplot as plt
import json
from pathlib import Path
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from tqdm import tqdm
import sys
import argparse


def main(input_fname: str):
    input_root = '../../data/031_features'
    output_root = './wordcloud_by_user' if 'by_user' in input_fname else './wordcloud_by_term'
    p = Path(output_root)
    if p.exists() is False:
        p.mkdir()
    # -------------------------------------
    # 1. load tf-idf score dictionary
    # -------------------------------------
    d_word_score_by_user = {}
    tfidf_fpath = input_root + '/' + input_fname
    with open(tfidf_fpath, 'r', encoding='utf-8') as f:
        d_word_score_by_user = json.load(f)
    # -------------------------------------
    # 2. gen word cloud from score
    # -------------------------------------
    fontpath = './rounded-l-mplus-1c-regular.ttf'
    for uname, d_word_score in tqdm(d_word_score_by_user.items(), desc='word cloud ...'):
        # img file name is user.png
        uname = str(uname).replace('/', '-')
        out_img_fpath = output_root + '/' + uname + '.png'
        # gen
        wc = WordCloud(
            background_color='white',
            font_path=fontpath,
            width=900, height=600,
            collocations=False
            )
        wc.generate_from_frequencies(d_word_score)
        wc.to_file(out_img_fpath)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("input_fname", help="set input file name", type=str)
    args = parser.parse_args()
    input_fname = args.input_fname
    main(input_fname)

10. Bonus

Articles that I especially referred to

-Types of preprocessing in natural language processing and their power | Qiita

• NLP Pipeline 101 With Basic Code Example — Feature Extraction

Other reference materials (large amount) are summarized in here: octocat:.

Promotion

This time, we are using data from the Slack community called Data Learning Guild. The Data Learning Guild is an online community of data analytics talent. If you are interested, please check here.

Data Learning Guild Official Homepage

Data Learning Guild 2019 Advent Calendar