[PYTHON] Since DataLiner 1.2.0 has been released, we will introduce the newly added preprocessing.

Introduction

We have released DataLiner 1.2.0, a pre-processing library for machine learning. This time, I have added about 6 new preprocessing, so I would like to introduce it.

GitHub: https://github.com/shallowdf20/dataliner PyPI: https://pypi.org/project/dataliner/ Document: https://shallowdf20.github.io/dataliner/preprocessing.html

Installation

Install using pip.

! pip install -U dataliner

Data preparation

Use Titanic data as usual.

import pandas as pd
import dataliner as dl

df = pd.read_csv('train.csv')
target_col = 'Survived'

X = df.drop(target_col, axis=1)
y = df[target_col]

Let's take a look now.

AppendArithmeticFeatures Four arithmetic operations are performed on the features included in the data, and a new feature with a higher evaluation index than the features used in the calculation is newly added. Evaluation is done by logistic regression. By default, multiplication and evaluation index are AUC, but addition, subtraction and division, and Accuracy are also available. It is necessary to fill in the missing values before using.

process = make_pipeline(
    dl.ImputeNaN(),
    dl.AppendArithmeticFeatures(metric='roc_auc', operation='multiply')
)
process.fit_transform(X, y)

In this way, new features are added.

RankedEvaluationMetricEncoding After making each category a dummy variable, logistic regression is performed with each category column and objective variable. Create a ranking using the resulting metric (AUC by default) and encode the original category with that ranking. Since 5 folds of logistic regression are fitted to each category, the amount of calculation will be enormous for features with high cardinality, so in advance It is recommended to lower the cardinality by using Drop High Cardinality or Group Rare Category.

process = make_pipeline(
    dl.ImputeNaN(),
    dl.RankedEvaluationMetricEncoding()
)
process.fit_transform(X, y)

You can also check how important each category in the categorical variable is by outputting the ranking.

process['rankedevaluationmetricencoding'].dic_corr_['Embarked']

AppendClassificationModel The classifier is trained based on the input data, and the prediction result is added as a new feature. The model can be any sklearn compliant model. Also, if the predict_proba method is implemented You can add a score instead of a label by giving the argument probability = True. Since the model is trained, missing value completion and categorical variable processing are basically required.

process = make_pipeline(
    dl.ImputeNaN(),
    dl.TargetMeanEncoding(),
    dl.AppendClassificationModel(model=RandomForestClassifier(n_estimators=300, max_depth=5),
                                 probability=False)
)
process.fit_transform(X, y)

This is the case when probability = True. A score for Class 1 will be awarded.

AppendEncoder The various Encoders included in the DataLiner directly replace the category columns with encoded numbers. However, in some cases, you may want to use it as a new feature without replacing it. (TargetMeanEncoder, etc.) In that case, it will be added as a feature by wrapping the Encoder in this class.

process = make_pipeline(
    dl.ImputeNaN(),
    dl.AppendEncoder(encoder=dl.TargetMeanEncoding())
)
process.fit_transform(X, y)

AppendClusterTargetMean Cluster the data and assign a cluster number. (Same as Append Cluster so far) Then replace each cluster number with the average of the objective variables in the cluster and add it as a new feature. Missing value completion and categorical variable processing are required.

process = make_pipeline(
    dl.ImputeNaN(),
    dl.TargetMeanEncoding(),
    dl.AppendClusterTargetMean()
)
process.fit_transform(X, y)

PermutationImportanceTest This is a type of feature selection method. With or without randomly shuffling data for a feature Feature selection is performed from the viewpoint of how much the evaluation index of the model prediction result deteriorates. If shuffling the data randomly does not have much effect on the metric, the feature is considered ineffective and deleted.

process = make_pipeline(
    dl.ImputeNaN(),
    dl.TargetMeanEncoding(),
    dl.PermutationImportanceTest()
)
process.fit_transform(X, y)

Name, PassengerId and Parch have been removed. You can also check the deleted features as follows.

process['permutationimportancetest'].drop_columns_

['PassengerId', 'Name', 'Parch']

You can also adjust the sensitivity by adjusting the threshold threshold. See Document for details.

in conclusion

The above is the newly added preprocessing. RankedEvaluationMetricEncoding is sometimes more accurate than TargetMeanEncoding, so I often try it. Also, the Permutation Importance Test can be executed faster than the Boruta and Step-wise methods, but there is no difference unexpectedly. I think that it may be used when you want to select the (?) Feature more seriously than DropLowAUC.

Release article: [Updated Ver1.1.9] I made a data preprocessing library DataLiner for machine learning

Pre-processing before 1.2 is introduced below. Try processing Titanic data with the preprocessing library DataLiner (Drop) Try processing Titanic data with the preprocessing library DataLiner (Encoding) Try processing Titanic data with the preprocessing library DataLiner (conversion) Try processing Titanic data with the preprocessing library DataLiner (Append)