[PYTHON] Introduction of DataLiner ver.1.3 and how to use Union Append

Introduction

We have implemented all the main functions that were expected for the release of DataLiner 1.3.1. In the future, the development pace will be about adding bug fix and preprocessing frequently.

Release article: https://qiita.com/shallowdf20/items/36727c9a18f5be365b37 GitHub: https://github.com/shallowdf20/dataliner Document: https://shallowdf20.github.io/dataliner/preprocessing.html

Installation

! pip install -U dataliner

Changes in ver1.3

There are the following four.

--UnionAppend implementation --StandardizeData abolished (renamed to Standard Scaling) --ArithmeticFeatureGenerator abolished (renamed to AppendArithmeticFeatures) --load_titanic implementation

Then, I will introduce the specific usage.

How to use

First, import the package to be used this time.

import dataliner as dl
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import RandomForestClassifier

From this version, titanic data is included in the package, so it is easy to try. Use the load_titanic method to load the sample titanic data.

X, X_test, y = dl.load_titanic()

Now X is the data excluding'Survived'in train.csv, X_test is test.csv, and y is the'Survived' column in train.csv.

How to use

This time, I will introduce the process called ** Union Append **.

In DataLiner pre-processing, basically all pre-processing that adds new features from existing features is given the class name Append 〇〇.

That's why we changed ArithmeticFeatureGenerator to AppendArithmeticFeatures in this version. The exceptions are BinarizeNaN and CountRowNaN, but these are the processes that are performed before the missing value completion / category processing in principle, so we have given this name.

Here, for example, let's say you want to add features as a whole, and suppose you build a pipeline as follows.

process = make_pipeline(
    dl.ImputeNaN(),
    dl.TargetMeanEncoding(),
    dl.StandardScaling(),
    dl.AppendCluster(),
    dl.AppendAnomalyScore(),
    dl.AppendPrincipalComponent(),
    dl.AppendClusterTargetMean(),
    dl.AppendClassificationModel(model=RandomForestClassifier(n_estimators=100, max_depth=5)),
    dl.AppendClusterDistance(),
    dl.AppendArithmeticFeatures(),
)
process.fit_transform(X, y)

In this method, for example, the features added by Append Cluster are used as the original data of the next Append Anomaly Score. (And the features will increase steadily in all Append 〇〇 below)

You may want to process in parallel instead of serial processing like this, and make all the features that are the basis of Append 〇〇 the same. You can use Union Append in that case.

process = make_pipeline(
    dl.ImputeNaN(),
    dl.TargetMeanEncoding(),
    dl.StandardScaling(),
    dl.UnionAppend([
        dl.AppendCluster(),
        dl.AppendAnomalyScore(),
        dl.AppendPrincipalComponent(),
        dl.AppendClusterTargetMean(),
        dl.AppendClassificationModel(model=RandomForestClassifier(n_estimators=100, max_depth=5)),
        dl.AppendClusterDistance(),
        dl.AppendArithmeticFeatures(),
    ]),
)
process.fit_transform(X, y)

By giving the class of Append 〇 〇 that you want to apply to UnionAppend as an array, all the base features of the processing in UnionAppend are unified, and the processing result of each Append 〇〇 is combined and returned. The execution result is as follows.

Processing to test data is as usual.

process.transform(X_test)

in conclusion

This completes the implementation of the initially expected functions and the preprocessing used. In the future, if I find a bug fix and a new pre-processing / I will add it if I can think of it.