[PYTHON] [Hands-on for beginners] Read kaggle's "Forecasting Home Prices" line by line (4th: Complementing Missing Values (Complete))

theme

• Click here for the first content
• Click here for the contents of Part 2
• Click here for the contents of Part 3

This is the 4th project to make a note of the contents of hands-on, where everyone will challenge the famous "House Price" problem of kaggle. It's more of a memo than a commentary, but I hope it helps someone somewhere. The impression that it gradually accumulated when it was the 4th time.

• Original theme: https://www.kaggle.com/c/house-prices-advanced-regression-techniques
• Referenced article: https://yolo-kiyoshi.com/2018/12/17/post-1003/

Today's work

Complementing missing values (I'll do it here this week)

What I did up to the last time was the one that "gets the index including the missing value as an array". (By the way, I feel like I'm confused that "python has various concepts of arrays and it's annoying")

#Complement missing values according to data type
#0 for float
#In the case of object'NA'
na_float_cols = alldata[na_col_list].dtypes[alldata[na_col_list].dtypes=='float64'].index.tolist() #float64
na_obj_cols = alldata[na_col_list].dtypes[alldata[na_col_list].dtypes=='object'].index.tolist() #object
#Substitute 0 if float64 type is missing
for na_float_col in na_float_cols:
    alldata.loc[alldata[na_float_col].isnull(),na_float_col] = 0.0
#If the object type is missing'NA'Substitute
for na_obj_col in na_obj_cols:
    alldata.loc[alldata[na_obj_col].isnull(),na_obj_col] = 'NA'

Indexes that are numeric and have missing values

• Details of this are omitted because it was done in Part 3.

alldata[na_col_list].dtypes[alldata[na_col_list].dtypes=='float64'].index.tolist()

Indexes with missing values in categorical variables

• The details of this are almost the same as the one I did in Part 3, so I will omit it.

alldata[na_col_list].dtypes[alldata[na_col_list].dtypes=='object'].index.tolist()

Substitute a missing value of numeric type

for na_float_col in na_float_cols:
    alldata.loc[alldata[na_float_col].isnull(),na_float_col] = 0.0

Now, but repeatedly about for in

I will read about for now. The order of variables and objects is reversed from that written in PHP (I don't know if it's correct).

• Reference about for: https://udemy.benesse.co.jp/development/web/python-for.html
• This variable: no_float_col
• Objects in this iteration: no_float_cols

It's persistent, but .isnull ()

Try to output na_float_col and ʻalldata [na_float_col]`. For the time being, let's take a look at the iron plate for checking the operation of the iterative process.

• First time of na_float_col You can see the name of "index containing missing values".

• ʻAlldata [na_float_col] `first time You will see an array of "index containing missing values" values.

• Result of .isnull () ʻIsnull ()is used to determine if each value is null. Output ʻalldata [na_float_col] .isnull ()

About .loc

alldata.loc[alldata[na_float_col].isnull(),na_float_col]

• .log: Apparently it can be used to see and output values in rows and columns in a matrix
• See .loc: https://note.nkmk.me/python-pandas-at-iat-loc-iloc/
• ʻAlldata.loc [alldata [na_float_col] .isnull (), na_float_col] `output result

Set a value for the missing value

Specify in a matrix and enter "0.0" only for missing values. alldata.loc[alldata[na_float_col].isnull(),na_float_col] = 0.0

Completion of missing values for categorical variables

• Since it is almost the same, I will omit it.

Missing value completion result

Each item is too detailed to see, but you should be able to go with this.

Output result of ʻall data`

Dummy of categorical variables

I thought I'd do it, but the time has run out, so I'd like to finish it as a preparation for "dummy categorical variables". Is it like quantifying it so that it can be analyzed? .. .. ??

• Dummy variable reference: https://markezine.jp/article/detail/20790

That's it.

It took longer than I expected to complete the missing values. I wonder if this is a Python trap that packs everything in one line (hopefully it's not a trap or anything).

It's almost time for the actual treatment to come closer and I'm excited to smell the clothes I'm wearing.