[PYTHON] Text analysis of pub rice on Yokohama City IR

0. Introduction

The August issue of "Public Relations Yokohama" was posted, so read it immediately. I always enjoyed reading it because I was always referred to by household consultations, such as "Well, the monthly food expenses for two people were too high for 50,000 yen !?" This month, there was an article about Yokohama City IR pub rice (I was doing it around April of this year), so when I looked at it, I heard that about 10,000 pub rice were received. It is unlikely that pub rice will receive such comments, so I wondered if there was any raw data, and when I looked at it it was!

The raw text data was released after processing information such as slander and slander. The city of Yokohama categorized the issues of pub rice into about five,

• I think there are more issues,
• What kind of opinion was reflected in the "Yokohama IR (Integrated Resort) Direction (Draft)" revised based on pub rice? I thought, so I decided to do a basic analysis for the time being.

1. Aggregation results described in the pub rice summary report

For the time being, you can see the overall result.

1.1 Number of submitters of opinions

• 5,040 people / organizations submit 9,509 opinions

1.2 Classification of opinions

1.3 Status of response to opinions

2 Take a look

• Read data (Data is extracted from the above report and sent to tsv) It was a little difficult to extract from the pdf data, but the following processed data.

• In addition, [Publish as tsv file on google spread sheet](https://docs.google.com/spreadsheets/d/e/2PACX-1vSzhD4TYx7nQ1a73zQW6ufe9w9-jEuwgYcwPLeM-ey25XbSTwYQUkw8802UM2DMLzpnS92vhn If you want to analyze it yourself, but the processing is troublesome, please. Is it okay if pub rice corresponds to when copyright is restricted? ?? ??

df = pd.read_csv("https://docs.google.com/spreadsheets/d/e/2PACX-1vSzhD4TYx7nQ1a73zQW6ufe9w9-jEuwgYcwPLeM-ey25XbSTwYQUkw8802UM2DMLzpnS92vHn8jQdkj/pub?gid=1919825840&single=true&output=tsv",sep='\t')

2.1 Word aggregation

Aggregate only nouns. It seems that what you are saying is quite transmitted.

2.1.1 (Miscellaneous) Pros and cons

If the word "agree / disagree" is included, a line of pros and cons will be created because it will be for / disagree with this proposal.

df ['sanpi'] = df ['opinion']. Apply (lambda x:'agree' if'agree' in x or'agree' in x else ('opposite' if'opposite' in x or'withdraw' in x else'unprocessed'))) df['sanpi'].value_counts()

result It's quite divided. Since there is a mixture of pros and cons for unprocessed, processing is required.

for pos in ['Agree',' Disagree','Unprocessed']: print ("---- {} Opinion ----". format (pos)) print (df [df ['sanpi'] == pos] ['opinion']. Head (5))

---- Agree ---- 1 I agree with you. Please proceed positively. By the way, I was born and raised in Yokohama. When I was little, Yamashita Park was ... 4 I agree with IR I think Yokohama will be better. I will do my best to support you! !! 5 I agree with IR. I think Yokohama will be better. good luck! !! 6 I agree with IR. I think this is a great opportunity to utilize the potential of Yokohama. I support you! !! 8 I agree with IR. I want Yokohama to be 100 years old and say "I'm glad I lived in Yokohama all the time" ... Name: Opinion, dtype: object ----opposite opinion---- 2 Absolutely the opposite. 3 The introduction of IR attraction should be withdrawn immediately. 7 I am against attracting IRs. I don't think it is necessary in Japan, of course, in Yokohama. 11 The draft direction of Yokohama IR states that it is beautiful, but the biggest Nerai is to build a casino facility in it ... 12 Absolutely no questions asked No casino required Name: Opinion, dtype: object ---- Unprocessed Opinion ---- 0 We should proceed for the future of Yokohama. 18 With the consumption of Japan as a whole declining in Corona and the economy stagnating, the mayor is encouraged to finance from Yokohama at IR Casino ... 23 The spread of the new coronavirus is having a major impact on the world economy. Even in Japan ... 25 Ｉ ＡＧＲＥＥ ＴＯ ＩＮＶＩＴＩＮＧ ＹＯＫＯＨＡＭＡ ＩＲ！！ ＹＯＫＯＨＡＭＡ ＩＲ ... 26 I think it's very good

2.2 Word cloud

2.2.1 Overall

2.2.2 By "correction" and "reference"

I thought the topic had changed drastically, but it didn't change much at first glance. ..

2.3 Characteristic word group for each category

2.3.1 Differentiation of "opinion" and "correction"

Calculate the importance by creating 5 models centered on the wood system, which are classified by 2 values, whether they end with "reference" or are reflected up to "correction". The closer it is to yellow, the higher the importance (words that appear biased toward either side). This one seems to better see the topics reflected in the "correction" opinion.

A confusion matrix is created and the F value is about 0.96. It seems that the fact that the number of data is unbalanced but can be divided cleanly reflects a specific opinion. This time around, the economic effect on the "shopping" district, measures against "infectious diseases", and "over tourism". Is "code" a dress code?

• With validation data [[ 84 14] [ 0 2058]] 0.9598435462842242

Let's look at some specific comments from the most important ones.

• Pub rice related to "shops"

word ='shop' df [df ['opinion'] .str.contains (word)]

• "Infection" related pub rice

word ='infection' df [df ['opinion'] .str.contains (word)]

• Pub rice related to "over tourism" I heard this word for the first time.

word ='overtourism' df [df ['opinion'] .str.contains (word)]

2.3.2 Distinguishing between "agree" and "disagree"

• Calculate the importance of words that often appear (do not appear) in "agree" or "disagree". I can't paste the image, so markdown. .. Writing "casino" is the opposite, and writing IR seems to be in favor.

• 1 Confusion matrix with validation data [[513 4] [ 17 243]] 0.9692612875332733

• 2 Please note that the judgment will be incorrect unless the words "agree", "disagree", etc.) that are the basis of the classification are excluded.

df ['wakati'] = df ['opinion'] .apply (wakati_mecab) df['wakati'] = df['wakati'].replace({'Agree|Agree|Opposition|Withdrawal':''},regex=True) matrix, names, y_tra = get_sparce_matrix (df [df ['sanpi']! ='Unprocessed'],'wakati','sanpi')

2.4 LDA (topic model)

• What topics are there in {Agree-Disagree / Each category}?

2.5 Co-occurrence network

Under construction

2.6 Sentence vector generation with transformer

making. If you make a two-dimensional map, you can see a mass of opinions.

3. Any comment

making···