[PYTHON] Full disclosure of methods used in machine learning

Introduction

There are more opportunities for anyone to learn machine learning, such as kaggle and learning sites. On the other hand, I feel that there are many people who do not get a sense of the whole because the amount of information is too much.

Therefore, I would like to publish ** the methods used in machine learning introduced in various reference books and articles **. Please refer to the link for detailed coding.

We also publish various information on SNS, so if you feel good reading the article I would be grateful if you could follow Twitter account "Saku731".

A series of machine learning steps

First, the skills required to learn machine learning are as follows. In practice, it requires a lot of detailed work, but at first it is enough to study the following.

―― 1) Data visualization: Grasp the overall feeling of the data and decide the preprocessing policy -2) Data preprocessing: Clean the data so that the prediction accuracy is high. ―― 3) Algorithm selection: Determine the appropriate algorithm for the data -4) Model learning: Let the computer learn the rules of data -5) Model verification: Confirm the prediction accuracy of the completed model

1) Data visualization

-Confirm basic statistics

• Average value
• standard deviation --Aggregate data -Cross tabulation -Binning -Pivot table -Group -Graph (Detailed version is here) --Bar graph
• Scatter plot --Histogram --Box plot -Heat map of correlation matrix

2) Data preprocessing

-Handling Missing Values

• Removal --Complement -Align data scale * Since the math part is difficult, skip it and read it.
• Standardization
• Normalization --Categorical variable encoding -one-hot encoding -Label Encoding --Click here if you want to know the difference in encoding -Join table (join DataFrame)
  • merge
  • concat

3) Algorithm selection

Link is here

--Regression --Linear regression (simple regression, multiple regression) --Regression tree --Random forest regression --Classification --Logistic regression --Decision tree --Random forest --Support Vector Machine (SVM) --Can be used for both --Neural network (deep learning) - XGBoost - LightGBM

4) Model learning

-Validation --Holdout method --Cross validation -Parameter tuning --Grid search --Random search -(Application) Bayesian optimization

5) Model verification

-Evaluation index (regression) - RMSE ( Root Mean Squared Error ) - RMSLE ( Root Mean Squared Logarithmic Error ) - MAE ( Mean Absolute Error ) --Coefficient of determination -Evaluation index (2 class classification) --Confusion matrix (TP, TN, FP, FN) - Accuracy - Precision - Recall --F value --LogLoss (cross entropy error) --AUC (ROC curve area) -Evaluation index (multi-class classification) - multi-class accuracy - multi-class logloss - mean-F1, macro-F1, micro-F1

at the end

I will increase the information from time to time. If you have any requests for additions or corrections, please contact us and it will be greatly appreciated.

I also wrote an article in "A series that uses machine learning for work". Please use all means.

-Part 1: Understanding the purpose of machine learning -Part 2: Overview of AI Development Project -Part 3: Python coding procedure

We also publish various information on SNS, so if you feel good reading the article I would be grateful if you could follow Twitter account "Saku731".

~~ Also, at the end of the sentence, we are doing "** Team Development Experience Project **" for a limited time. ~~ ~~ If you are interested, please check [Application Sheet] for details. ~~ (Addition) The deadline has been closed because it is full. The next time is scheduled for March 2019, so if you would like to be informed, please fill in [Reservation Form].