Beginning with Python machine learning

I came to the Open Developers Conference in Kamata.

Let's create a live article while listening to the story.

** 2017/12/21 postscript ** Click here for presentation slides → https://speakerdeck.com/terapyon/python-ji-jie-xue-xi-kotohazime-at-odc

Theme "Beginning with Python Machine Learning"

Predict epidemics from the number of reported influenza in Chiba City and temperature

procedure

1. Select data and get data ~ ~ ~ Search for material
2. Data processing, data connection ~ ~ ~ Easy to use
3. Visualization ~ ~ ~ Somehow something can be seen. What kind of algorithm should I use?
4. Algorithm selection
5. Machine learning model selection
6. Machine learning model evaluation

Select data and get data

Received from Chiba City and the Japan Meteorological Agency.

Data processing

• Character code
• Make up for the missing part

Visualization

• Try to make a graph
• Can you see the correlation?
• Determine the amount of features to use

Algorithm selection

• Is all the features solid in deep learning?
• Data reprocessing in some cases
• Create dimension number? ?? ??
• Determine explanatory variables and objective variables

Machine learning model selection

Easy if you come here Immediately

Machine learning model evaluation

• Forecast
• Evaluate the prediction results
• Cross-validation
• Isn't it overfitting because it fits the model too much?

Come on python

Use Jupyter notebook

[Shift]-[Enter] will execute

Convenient

You can publish it to github as it is

View data with Pandas

I can read the bad CSV and process it somehow

Make full use of the data frame function

If you index by date, you can also concatenate weekly data and daily data with a single command.

Serialization

This is the most time consuming

Visualization with matplotlib

It doesn't work if Japanese is included

You can easily get a histogram from pandas using matplotlib

To define what a "fashion" is

Visualize multivariable calculus at once with pandas The graph that goes straight up from the lower left to the upper right is correlation 1

Consider algorithm candidates

• Logistic regression

And

Split training data and test data using scikit-learn

Explanatory variable x

Objective variable y Popular 1 Not popular 0

Save test data to avoid overfitting. 80% for learning, 20% for testing Percentage depends on model and amount of data

Algorithm initialization

use scikit-learn

Prediction clf.predict (X_val)

Use a confusion matrix because it's almost a hit

True positive False positive True negative False negative

Cross-validation

Was the ratio of learning and testing OK? Repeat the division and recalculation

Grid search (parameter adjustment)

(By the way, time runs out