[PYTHON] How to use machine learning for work? 02_Overview of AI development project

Introduction

I am writing a series on the theme of "How to use machine learning for work". In this second article, I will organize how the AI development project will proceed.

The chapters are as follows.

――What is the difference between AI development and machine learning? --Overview of AI development ――Where do you use machine learning in AI development? --Procedures for building a model

What is machine learning? If you feel that your understanding of is suspicious, please also read "Part 1: Understanding the purpose of machine learning".

What is the difference between AI development and machine learning?

As I introduced in Part 1: Understanding the purpose of machine learning, there is a comprehensive relationship of "AI> machine learning". Specifically, it will be positioned as ** using machine learning technology while advancing AI development **.

(Source: https://rikei-danshi.work/entry/ai-chigai)

Overview of AI development

AI development is organized in the figure below, but we will proceed with the process of "planning-PoC-development-operation".

--Planning: Clarify what you want to do with AI --PoC: Is the planned AI technically feasible? Is the project profitable? Check --Development: Build a "model" that is the heart of AI and implement it in the system --Operation: Perform maintenance and operation so that the developed system operates stably.

Where to use machine learning?

Now that you have a look at the overall process of AI development, let's think about where to use machine learning. The answer is to use machine learning in "PoC" and "development".

The "model" shown in the figure is AI itself, and machine learning skills are required to build a good model. For those who have forgotten what a model is, see Part 1: Understanding the Purpose of Machine Learning.

Steps to build a model

To master machine learning, you need three main skills. Specific programming will be organized in Part 3. First, let's understand only the big picture this time.

--Data preprocessing --Model learning --Model validation

Data preprocessing

The purpose of data preprocessing is ** to prepare the data so that the model can be trained **.

Specifically, we will carry out three things, but in essence, the main analysis is the ** "preparation" ** of machine learning. Articles written by people with practical experience are often written in the field of ** data science ** rather than ** machine learning **.

--EDA (Exploratory Data Analysis): Grasp the statistical overall trend of data --Processing of missing and outliers: removing obstructive data that causes noise, etc. --Feature extraction: Analyze where ** the law of data ** appears

Model learning

From here, machine learning technology will appear in earnest. As introduced in "Part 1: Understanding the purpose of machine learning" ** Finding the law from a large amount of prepared data ** Is the main purpose.

The machine learning method provided for that purpose is the ** algorithm **. If you compare it to cooking, it will be easier to imagine if you think of ** "data = ingredients" ** and ** "algorithm = recipe" **.

If you cook with the appropriate recipe according to the ingredients, you will get a delicious dish. Machine learning is the same, and if you use an appropriate algorithm for data, you can create a model with good prediction accuracy (= good AI).

Model validation

After training the model, you need a ** performance check ** to see how accurate the model predicts.

• Sometimes ** AI predicts everything correctly, right? Some people mistake it for **, so please be sure to tell us the importance of verification.

There are two things to do in the verification of this model.

• Regarding the second, I will omit the details because it is enough to understand in practice

--Measurement of prediction accuracy: When I read the reference book, various difficult explanations (holdout method, cross-validation, etc.) are introduced. ** Just try it out and check if you can predict it correctly **. --Consider the continuous maintainability of the model: I will omit it because the content will be deep, but if you are interested, please read Re-learning the model.

at the end

In the next article, I will organize the concrete implementation procedure of preprocessing-learning-verification based on the programming language Python. Please look forward to it as it covers and organizes a series of programming methods and specific study methods.

Also, various information is posted on SNS, so if you feel that you can read the article, I would be grateful if you could follow Twitter account "Saku731".

~~ Also, at the end of the sentence, we are doing "** Team Development Experience Project **" for a limited time. ~~ ~~ If you are interested, please check [Application Sheet] for details. ~~ (Addition) The deadline has been closed because it is full. The next time is scheduled for March 2019, so if you would like to be informed, please fill in [Reservation Form].

Until the end Thank you for reading.