[PYTHON] I tried FX technical analysis by AI "scikit-learn"

1. at first
Document structure
1. Document contents (excerpt)
Impressive process
Where to get "PDF version document and data for execution"

1. 1. at first

This is a water strider. (Of course, pen name) Actually, the author has been continuing the EA development of MT4 (MQL4) for a long time, hoping to connect it to actual profit if it can be done as a hobby, and for the past 1-2 years, especially linking MT4 and AI (especially non-technical analysis) ) Was the main focus, but the results were not so good and I was stuck.

So, although it is not a short break, I tried to apply AI to "technical analysis" that I thought that the functions and performance of MT4 were sufficient, "AI" scikit-learn " FX technical analysis by ".

Since I am a beginner in AI, I have researched many articles and materials that try to utilize AI for FX technical analysis. (Net and books) However, unfortunately, I couldn't find anything that could be used properly. Maybe some private business ones can be used, but they aren't available to the general public like me.

So, I tried my own method (probably) to use MT4 (MQL4) to select "features" to be used in AI. In fact, I found other tools (especially Python-based) that could be used. This is a measure I took because I didn't have it, but I think it's probably a difference in the development process from other developers.

Although it is currently a prototype version, we have obtained "results that will shine if polished", so we decided to release it to everyone. Documents (reports) are serialized and divided into two. (Currently there are two, but we plan to increase them in the future)

2. Document structure

This series reports the test results when AI (scikit-learn) is used for prediction by technical analysis of FX (USDJPY; dollar / yen). (Currently the prototype version)

** Attempt of FX technical analysis by AI "scikit-learn"; ** It is divided into [Part 1] and [Part 2].

◎ Please read [Part 1] first, and obtain [Part 2] if you think it is worth reading. (How to get it is described in [Part 1].)

Both [Part 1] and [Part 2] are prepared in PDF format, and the data is prepared in ZIP format. I haven't listed everything in Markdown format on Qiita, and I haven't used GitHub because I can't master it yet.

3. 3. Document contents (excerpt)

** [Goal this time] ** ・・ Refer to the block diagram It is to extract the "technical analysis result" that is worth passing to the MT4 (MQL4) side on the AI side. To be honest, I think that MT4 (MQL4) is sufficient for the technical analysis function. When I tried it, I got quite interesting results, I think the try was not wasted. (AI program is written in Python)

"DLL" for passing trading information (data) from Python code side to MT4 (MQL4) and its Please refer to the author's website for how to use it. (DLL is also placed)

** Explanation contents in [Part 1] and [Part 2]; ** First of all, the "procedure for applying AI to FX technical analysis" adopted by the author this time is as follows.

The author uses Spyder for the Python development environment.

(1) [Part 1] is An example of the last stage ・ DataFrame (features and labels) generated from the Forex (FX) chart Explains an example of performing machine learning / prediction using a Python program.

(2) In [Part 2] I will explain from the very first stage to creating a DataFrame with an example -Detailed explanation of how to generate DataFrame (features and labels) used in [Part 1](separate article).

** <Indicator used (example)> ** ・・ [Part 2] posted The indicator used to extract the features is common as the author's original (pine needle type filter). Bollinger band.

** ** ・・ Posted in [Part 1] The example DataFrame has up to 1049 rows. -Column elements are features and labels, -Row elements are "data" extracted from the FX chart.

** ** ・・ [Part 1] posted, "2 examples" The figure below is an image cut from the description (Spyder output) Example 1; Example 2;

** [Win / Loss Judgment Method] **

Win / loss judgment uses the same concept as when creating an MT4 EA. In other words, "buy or sell" by specifying profit (profit level), loss (loss level), and spread. Learn the result of winning or losing judgment when entering the market from, and make predictions for test data. In addition, winning and losing "learning and prediction" is performed by "classification" by random forest.
The development environment (all available free of charge) required to execute the contents is described in [Part 1]. However, it does not provide technical explanations on how to obtain Python, scikit-learn and MT4, which are necessary for beginners. Some of the knowledge required for beginners can be found on the author's website (below). ・・ All the knowledge necessary to understand the contents is available on the WEB such as Qiita.

4. Impressive process

Initially, "SVC (SVM)" was used as the algorithm, but the result (example of initial implementation) is It was the following situation. ◆ What a phenomenon that all the prediction results are "-1", no matter how many times I try I couldn't improve. ("-1" is when the forecast result is "loss cut") However, the overall correct answer rate was "138/210" (66%), which made it difficult to understand.

The cause is that SVC (SVM) deals with "unbalanced data", that is, the number of data between classes is large. I learned that various advance adjustments are essential when dealing with objects that are highly biased. (Scaling, normalization, etc. need to be adjusted) After all, I changed the classification algorithm to "Random Forest" to solve this annoying problem. As a result of avoiding it, the result was quite convincing.

5. Where to get "PDF version document and data for execution"

** ◎ Where to get [Part 1] URL ** ** ① Document (PDF) **; http://mql4.s1002.xrea.com/ai_tec/ai_tec_01.pdf ** ② Data set (.zip) **; http://mql4.s1002.xrea.com/ai_tec/ai_tec_01.zip

The data set contains materials necessary for executing "Python code, DataFrame, etc." 　　 ** ◎ How to link Python and MT4 **
This site is being updated, and the basic contents have also been uploaded. [New page of water striders]; http://mql4.s1002.xrea.com/ai_mql4/index.html

** ◎ About MT4 (MQL4) and EA **

It's a little old, but I think it's still worth reading. We are improving the basics of MT4 and MQL4, but before (or less than) "Build_600" MQL4 code (script) now needs a lot of modification to work, Please be careful. [Water striders page]; http://www.green.dti.ne.jp/sdimension/mql/

that's all