[PYTHON] The first time a programming beginner tried simple data analysis by programming

Excuse I didn't update

The update has been delayed. There are many reasons (or excuses), but I think most of the time I didn't write much code. At the beginning of the year, the amount of lessons and assignments settled down a little, and at the same time, I was wandering around in my mind, thinking about my graduation work and my career path after graduation, so I didn't move much. So I updated my my blog every day, but my honest impression is that there wasn't enough content to write on Qiita. I'm still a little lost, but for the time being, my graduation work has been decided and I'm working hard to make it.

I learned a little python in class

During the class, I played with some data on the cloud. I also learned about an amazing site called Kaggle. This is the first time I tried it while taking data from Kaggle myself after class and checking it.

Get data from kaggle

https://www.kaggle.com/unsdsn/world-happiness#2019.csv I thought that I could see various correlations, so I chose this.

Mount from google drive

from google.colab import drive
drive.mount('/content/drive')

Upload the required CSV to google drive in advance

Import and data acquisition of libraries that may be needed

import numpy as np 
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("/content/drive/My Drive/2019.csv")

I was impressed that there are so many libraries besides this!

Check the number of data and the presence of missing values

df.count()

Number of data 156, no missing values If I don't do this, I don't know whether to display all the data or only the first few, so I tried it. I also wanted to avoid data with many missing values because it seemed to be confusing. (I think we need to take on the challenge in the future, but for the time being, this is the first time)

Try to display only the first 20

df.head(20)

Japan is not included

Let's roughly display what seems to be necessary

df.describe()

Is this just the nature of the data? I can understand how.

Try to find the correlation coefficient between score (happiness) and social support (social welfare)

#Library preparation
import numpy as np
import pandas as pd

#Data set preparation

##Happiness to array
happy = df["Score"]

##Arrange social welfare
social = df["Social support"]

#Get the correlation coefficient!
correlation = np.corrcoef(social, happy)
print(correlation)

[[1. 0.77705779] [0.77705779 1. ]]

It came out ~~ Since the correlation coefficient is 0.7, social welfare has a strong correlation with happiness! !!

Try to put out a heat map

#Library preparation
import pandas as pd
import numpy as np

#You should be able to get the correlation coefficient between columns!
corr_df =df.corr()
print(corr_df)

Overall rank ... Perceptions of corruption Overall rank 1.000000 ... -0.351959 Score -0.989096 ... 0.385613 GDP per capita -0.801947 ... 0.298920 Social support -0.767465 ... 0.181899 Healthy life expectancy -0.787411 ... 0.295283 Freedom to make life choices -0.546606 ... 0.438843 Generosity -0.047993 ... 0.326538 Perceptions of corruption -0.351959 ... 1.000000

[8 rows x 8 columns]

I'm sorry it's hard to see, I'm out for the time being!

#Library preparation
import seaborn as sns
sns.heatmap(corr_df, cmap= sns.color_palette('cool', 5), annot=True,fmt='.2f', vmin = -1, vmax = 1)

I forgot to completely overtake the overall rank, but I managed to get it!

I will study for myself in the future

Impressions of the class

I've always been interested in python. Although the number of lessons was limited to 4 times in total, it was interesting to learn various things about "data", not just python. The rest was purely fun. I remember doing a little statistics at SPSS when I was a graduate student. I didn't use statistics in my master's thesis, so I only touched on it a little, but it was purely interesting at that time as well. I remembered that time when I was young lol

There are many things I want to try

For the time being, this was my first time, so I tried it roughly without thinking about deep things. Instead of thinking about strict statistics, simply try to figure out or visualize. If you look at the specialists, I think there are a lot of things to do. There are many things I would like to do with python, such as factor analysis, principal component analysis, and logistic analysis that I used to do with SPSS. I couldn't do it at all due to lack of time and knowledge during the class ...

Try it steadily

The class itself is over, but at the same time I felt the potential of machine learning, and at the same time, I realized that I had a faint feeling because I was in this industry. I'm sure it's not just fun or interesting things while studying, but in the future I would like to study as much as I can. I'm not planning to include machine learning in the product I'm making, so how much can I do in a different frame from the complete graduation work? It seems that it will be finite non-execution, and I'm already worried, but I wrote Qiita to warn myself, so I'd like to try it little by little. I hope to be able to describe such contents little by little in the future.