[PYTHON] Analyzing Twitter Data | Trend Analysis

This article is a trend analysis version of Twitter Data analysis. This series will be deepened later. Since this is my first article, I will explain this case in detail after a brief analysis of Twitter Data. Japanese is hard, so I will change it to English. Git Repository

Why Analyze Twitter Data? Twitter data analysis is used in a wide range of areas including but not restricted to analyzing the mentions of each political party in an election, detecting the reactions to the introduction of a new product, understanding the geographical scope of discussion of a news story.

How to analyze twitter data? Twitter data can be retrieved from twitter API. Data can be wrangled and manipulated according to the purpose of analysis. For example, if we need to get insights on a certain product from twitter data, there're a few strategies we can apply:

Collecting data via twitter API

This is a slightly complicated procedure if it's your first time to use twitter API. Here I'm going to explain how to collect data through twitter API.

  1. Log in your twitter account
  2. Apply for a developer account. Apply Here!
  3. Create a new application.
  4. Go to the application and find keys and tokens. (We'll need them later for authentication purposes.)


from tweepy import OAuthHandler
from tweepy import API
import json

consumer_key = ...
consumer_secret = ...
access_token = ...
access_token_secret = ...
# Consumer key authentication
auth = OAuthHandler(consumer_key, consumer_secret)

# Access key authentication
auth.set_access_token(access_token, access_token_secret)

# Set up the API with the authentication handler
api = API(auth)

Getting the trend Here I'm gonna use UK as an example. Information regarding WOE ID can be found here.

UK_WOE_ID = 23424975
UK_trends = api.trends_place(UK_WOE_ID)

trends = json.loads(json.dumps(UK_trends, indent=1))

Local and global thought patterns

# Loading json module
import json

# Load WW_trends and US_trends data into the the given variables respectively
WW_trends = json.loads(open('datasets/WWTrends.json').read())
US_trends = json.loads(open('datasets/USTrends.json').read())

# Inspecting data by printing out WW_trends and US_trends variables

Prettifying the output

# Pretty-printing the results. First WW and then US trends.

print("WW trends:")
print(json.dumps(WW_trends, indent=1))

print("\n", "US trends:")
print(json.dumps(US_trends, indent=1))

Finding common trends

# Extracting all the WW trend names from WW_trends
world_trends = set([trend['name'] for trend in WW_trends[0]['trends']])

# Extracting all the US trend names from US_trends
us_trends = set([trend['name'] for trend in US_trends[0]['trends']]) 

# Getting the intersection of the two sets of trends
common_trends = world_trends.intersection(us_trends)

# Inspecting the data
print(world_trends, "\n")
print(us_trends, "\n")
print (len(common_trends), "common trends:", common_trends)

Exploring the hot trend

# Loading the data
tweets = json.loads(open('datasets/WeLoveTheEarth.json').read())

# Inspecting some tweets

Digging deeper

# Extracting the text of all the tweets from the tweet object
texts = [tweet['text'] for tweet in tweets]

# Extracting screen names of users tweeting about #WeLoveTheEarth
names = [user_mention['screen_name'] for tweet in tweets for user_mention in tweet['entities']['user_mentions']]

# Extracting all the hashtags being used when talking about this topic
hashtags = [hashtag['text'] for tweet in tweets for hashtag in tweet['entities']['hashtags']]

# Inspecting the first 10 results
print (json.dumps(texts[0:10], indent=1),"\n")
print (json.dumps(names[0:10], indent=1),"\n")
print (json.dumps(hashtags[0:10], indent=1),"\n")

Frequency analysis

# Importing modules
from collections import Counter

# Counting occcurrences/ getting frequency dist of all names and hashtags
for item in [names, hashtags]:
    c = Counter(item)
    # Inspecting the 10 most common items in c
    print (c.most_common(10), "\n")

Activity around the trend

# Extracting useful information from retweets
retweets = [(tweet['retweet_count'], 
             tweet['text']) for tweet in tweets if 'retweeted_status' in tweet]

A table that speaks a 1000 words

# Importing modules
import matplotlib.pyplot as plt
import pandas as pd

# Create a DataFrame and visualize the data in a pretty and insightful format
df = pd.DataFrame(retweets, columns=['Retweets','Favorites', 'Followers', 'ScreenName', 'Text'])

df = df.groupby(['ScreenName','Text','Followers']).sum().sort_values(by=['Followers'], ascending= False)

Analyzing used languages

# Extracting language for each tweet and appending it to the list of languages
tweets_languages = []
for tweet in tweets: 

# Plotting the distribution of languages
%matplotlib inline
Screen Shot 2020-02-20 at 11.05.21 pm.png

That's the end of our project. In the next project of this series we'll look into more complicated use of twitter API.

Recommended Posts

Analyzing Twitter Data | Trend Analysis
Starbucks Twitter Data Location Visualization and Analysis
Program for Twitter Trend Analysis (Personal Note)
Data analysis python
Data analysis Titanic 1
Data analysis with python 2
Data analysis using xarray
Data analysis parts collection
Data analysis using Python 0
Negative / Positive Analysis 2 Twitter Negative / Positive Analysis (1)
Negative / Positive Analysis 3 Twitter Negative / Positive Analysis (2)
Data analysis with Python
Try rudimentary sentiment analysis on Twitter Stream API data.
[CovsirPhy] COVID-19 Python package for data analysis: S-R trend analysis
My python data analysis container
Multidimensional data analysis library xarray
Python for Data Analysis Chapter 4
Extract Twitter data with CSV
[Python] Notes on data analysis
Python data analysis learning notes
Python for Data Analysis Chapter 2
Wrap analysis part1 (data preparation)
Data analysis using python pandas
Tips for data analysis ・ Notes
Python for Data Analysis Chapter 3
First satellite data analysis by Tellus
Preprocessing template for data analysis (Python)
November 2020 data analysis test passing experience
Data analysis for improving POG 3-Regression analysis-
Time series analysis 3 Preprocessing of time series data
Get data from Twitter using Tweepy
Data analysis starting with python (data visualization 1)
Data analysis starting with python (data visualization 2)
Data handling 2 Analysis of various data formats
Multidimensional data analysis library xarray Part 2