Tweet analysis with Python, Mecab and CaboCha

Introduction

This article is the 22nd day article of Python Advent Calendar 2014 --Qiita.

Last week, I wrote a blog called BigQuery x Perfume x tweet analysis on the company's Advent Calendar. There, I tried to analyze the tweets about Perfume collected during one week from 12/12 (Friday) to 12/18 (Thursday) by giving them to BigQuery.

This time, as a development, I will do tweet analysis with natural language processing using Mecab and CaboCha m (_ _) m

Referenced

That's why I look up the tweets of the House of Representatives election --Qiita https://github.com/mima3/stream_twitter

Rather than reference, most of what I'm doing is for sale by mima_ita ...

environment

· Mac OSX 10.9.5 -Python 2.7.8

Twitter data collection

This time, I used a service called mention. In mention, you can easily pull data from SNS with the keywords set on the management screen. Export the data acquired here to csv from the management screen, and you are ready to go.

The conditions specified this time are as follows.

Search keyword: Perfume|| prfm || perfume_um (prfm and perfume_um are hashtags used in posts about Perfume)

Negative keywords: RT

Target SNS: Twitter Language: Japanese

Period: 12/12 (Friday) -12/18 (Thursday)

Install Mecab and Cabocha

Mecab：http://salinger.github.io/blog/2013/01/17/1/ Cabocha：http://qiita.com/ShingoOikawa/items/ef4ac2929ec19599a3cf

If you follow this article, there was no problem (`・ ω ・ ´) ゞ

Code used for collection and analysis

https://github.com/mima3/stream_twitter

However, here we use the Streaming API to collect Twitter data and store it in the DB. This time, unlike that, it is necessary to store what was collected by mention and spit out by csv in the DB.

So, please store it in SQLite with the following source.

create_database.py

#!/usr/bin/python
# -*- coding: utf-8 -*-
import sqlite3
import csv

if __name__ == '__main__':
    con = sqlite3.connect("twitter_stream.sqlite")

    c = con.cursor()
    # Create table
    c.execute('''CREATE TABLE "twitte" ("id" INTEGER NOT NULL PRIMARY KEY, "createAt" DATETIME NOT NULL, "idStr" VARCHAR(255) NOT NULL, "contents" VARCHAR(255) NOT NULL);
    ''')
    c.execute('''CREATE INDEX "twitte_createAt" ON "twitte" ("createAt");''')
    c.execute('''CREATE INDEX "twitte_idStr" ON "twitte" ("idStr")''')

    # Insert data
    i = 0
    data = []
    reader = csv.reader(open("./perfume_tweet.csv"))
    for row in reader:
        id = i+1
        createAt = row[4]
        idStr = unicode(row[0],'utf-8')
        contents = unicode(row[1],'utf-8')
        t = (id,createAt,idStr,contents)
        data.append(t)
        i += 1
    con.executemany(u"insert into twitte values(?,?,?,?)",data)

    # Save (commit) the changes
    con.commit()

    # We can also close the cursor if we are done with it
    con.close()

You have successfully created twitter_stream.sqlite in your current directory.

Histogram by time

Here, I will pick up 12/17 (Wednesday). We will count the number of tweets by hour in the day.

 python twitter_db_hist.py "2014/12/16 15:00" "2014/12/17 15:00" 3600

• Because the time that can be obtained from Twitter is UTC, it is 9 hours off in Japan time.

Looking at this, we can analyze the following.

・ The time zone from 05:00 to 06:00 is the least. It's all low from 02:00 to 06:00, so everyone is probably sleeping at this time. ・ The most frequent hours are from 23:00 to 24:00. The number of tweets is generally high from 18:00 to 24:00, but the highest is from 23:00 to 24:00 at the end of the day. Many people have been sleeping since they tweeted about Perfume?

That's a rough idea, but you can understand it somehow.

Frequent word extraction

Next, we will perform morphological analysis using Mecab. Here, we will sift the collected tweets for one week.

python twitter_db_mecab.py "2014/12/11 15:00" "2014/12/17 15:00" > mecab.txt

Below is a list of the top 100.

The top words are included in the bot's tweet, so it's not significant data. As a sensory value, it seems that meaningful words appear when analyzing from a total of 500 tweets or less.

During this data collection period, there were many tweets about the event "Masked Chowder ~ YAJIO CRAZY ~ Chowder University International Collagen High School" that was held on Saturday, December 20th. was.

Event name related: "Mask" "Chowder" Ticket information related: "Bid" "Yahoo auction" "Price" "yauc" "Ticket" "Sheet" (I wasn't familiar with yauc, but it seems to be a hashtag of Yahoo Auction)

After that, I was able to roughly analyze the following.

Perfume song title related: "Cling" (Cling Cling) "Natural" "Love" (In love with Natural) Singer related: "Arashi" "Momo" "Kuro" "Sakanaction" "Sekaowa" "Ringo" "Shiina" Perfume Music Player related: "Listen" "Music" "Player" "mp"

• Perfume Music Player is Perfume's official app for smartphones. From this app, you can tweet the song you are currently listening to as follows. "I'm listening to Cling Cling for the 252,533th person on Perfume Music Player. #Prfm_mp http://perfume-app.com/"

Dependency analysis

Next, let's use CaboCha to aggregate the dependency relationships of clauses. Again, we are analyzing a week's worth of tweets.

python twitter_db_cabocha.py "2014/12/11 15:00" "2014/12/17 15:00" > cabocha.txt

Let's look at the top 100 in the same way.

Again, a total of 500 or more dependencies are noisy bots, so let's go through. By the way, the bot has the following two tweets.

Perfume's Nocchi A-chan During the live ●● and rage www http://t.co/4Q0fmhel2l Perfume "Nocchi" Ayano Omoto Married to a general man? http://t.co/k6cFhLZwnZ

I don't know how to look at the dependency table, so I grep it with the names of the three members. Among the dependent Phrase, the ones that include "A-chan", "Kashiyuka", and "Nocchi" are shown below. (* Extract only those with neatly separated phrases)

As you can see, there were many things related to Kashino in the dependency analysis! Looking at this dependency, the characteristics of each of the three people are revealed, which is wonderful ...

Finally

As mentioned above, with the help of mima_ita, we have analyzed the tweets about Perfume. Looking back, it's a pity that we have collected a large amount of bottweets that are unnecessary for analysis under the conditions we performed this time ...

I still have a lot of data analysis and Python skills, so I will buy this book to be released next week and study it (`・ ω ・ ´) ゞ Introduction to programming in Python language: World Standard MIT textbook