[PYTHON] I tried Auto Gluon

Introduction

I tried using an AutoML tool called AutoGluon. This time, I will describe the contents of trying the tabular data and sample code prepared by AutoGluon.

What is Auto Gluon

Overview

It is an open source automatic machine learning tool that automatically handles simple feature engineering, model selection, hyperparameter tuning, and ensembles if you prepare data in tabular format.

https://autogluon.mxnet.io/

As of May 2, 2020, version 0.0.6 has been released. For structured data, it seems that classification, regression, image classification, object detection, and sentence classification can be performed.

Accuracy evaluation for tabular data (paper)

https://arxiv.org/pdf/2003.06505.pdf

The accuracy is evaluated in this paper. For AutoGluon, TPOT, H2O, AutoWEKA, auto-sklearn, and Google AutoML Tables, when calculating the prediction accuracy using 50 types of data sets, AutoGluon was the most accurate.

This paper cites Multi-layer Stack Ensemble as a feature of Auto Gluon. Please refer to the paper for details.

environment

Conducted at Google Colaboratory.

Contents

Preparation

!pip install --upgrade mxnet
!pip install autogluon

The installation is now complete. At this stage, errors and warnings appeared in the pandas version etc., but it was finally usable.

import autogluon as ag
from autogluon import TabularPrediction as task

I got an error here, but when I restarted the runtime, it ran without error. Use the same data as in the tutorial.

train_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv')
train_data = train_data.head(10000)
print(train_data.head())

The data used was limited to 10,000 rows to reduce execution time. There are 15 columns, including numbers and strings, as shown below.

Loaded data from: https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv | Columns = 15 / 15 | Rows = 39073 -> 39073
   age   workclass  fnlwgt  ... hours-per-week  native-country   class
0   25     Private  178478  ...             40   United-States   <=50K
1   23   State-gov   61743  ...             35   United-States   <=50K
2   46     Private  376789  ...             15   United-States   <=50K
3   55           ?  200235  ...             50   United-States    >50K
4   36     Private  224541  ...             40     El-Salvador   <=50K

[5 rows x 15 columns]

The objective variable also uses class as in the tutorial.


label_column = 'class'
print("Summary of class variable: \n", train_data[label_column].describe())
Summary of class variable: 
 count      10000
unique         2
top        <=50K
freq        7568
Name: class, dtype: object

When built by default

I will actually build a machine learning model, but since it only fits the above data, It's very easy to build a model. First, let's run the fit argument with almost the defaults.


%%time
dir = 'agModels-predictClass' # specifies folder where to store trained models
predictor = task.fit(train_data=train_data, label=label_column, output_directory=dir)

It seems that 14 explanatory variables are numerical values and the type of object is determined and feature engineering is performed. I didn't see what happened in the log. By default, it seems that accuracy is used for optimization and early stopping. I checked about 10 models and it took 2 minutes to get to the assemble, so I thought it was very fast. However, as described in the API reference (https://autogluon.mxnet.io/api/autogluon.task.html#autogluon.task.TabularPrediction.fit), hyperparameter tuning is not performed by default. ..

Beginning AutoGluon training ...
AutoGluon will save models to agModels-predictClass/
Train Data Rows:    10000
Train Data Columns: 15
Preprocessing data ...
Here are the first 10 unique label values in your data:  [' <=50K' ' >50K']
AutoGluon infers your prediction problem is: binary  (because only two unique label-values observed)
If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])

Selected class <--> label mapping:  class 1 =  >50K, class 0 =  <=50K
Feature Generator processed 10000 data points with 14 features
Original Features:
	int features: 6
	object features: 8
Generated Features:
	int features: 0
All Features:
	int features: 6
	object features: 8
	Data preprocessing and feature engineering runtime = 0.15s ...
AutoGluon will gauge predictive performance using evaluation metric: accuracy
To change this, specify the eval_metric argument of fit()
AutoGluon will early stop models using evaluation metric: accuracy
Fitting model: RandomForestClassifierGini ...
	0.842	 = Validation accuracy score
	2.44s	 = Training runtime
	0.22s	 = Validation runtime
Fitting model: RandomForestClassifierEntr ...
	0.846	 = Validation accuracy score
	3.04s	 = Training runtime
	0.22s	 = Validation runtime
Fitting model: ExtraTreesClassifierGini ...
	0.83	 = Validation accuracy score
	1.93s	 = Training runtime
	0.22s	 = Validation runtime
Fitting model: ExtraTreesClassifierEntr ...
	0.836	 = Validation accuracy score
	2.25s	 = Training runtime
	0.22s	 = Validation runtime
Fitting model: KNeighborsClassifierUnif ...
	0.772	 = Validation accuracy score
	0.07s	 = Training runtime
	0.11s	 = Validation runtime
Fitting model: KNeighborsClassifierDist ...
	0.748	 = Validation accuracy score
	0.04s	 = Training runtime
	0.11s	 = Validation runtime
Fitting model: LightGBMClassifier ...
	0.854	 = Validation accuracy score
	1.21s	 = Training runtime
	0.02s	 = Validation runtime
Fitting model: CatboostClassifier ...
	0.864	 = Validation accuracy score
	8.85s	 = Training runtime
	0.02s	 = Validation runtime
Fitting model: NeuralNetClassifier ...
	0.854	 = Validation accuracy score
	93.04s	 = Training runtime
	0.26s	 = Validation runtime
Fitting model: LightGBMClassifierCustom ...
	0.852	 = Validation accuracy score
	2.2s	 = Training runtime
	0.03s	 = Validation runtime
Fitting model: weighted_ensemble_k0_l1 ...
	0.864	 = Validation accuracy score
	0.82s	 = Training runtime
	0.0s	 = Validation runtime
AutoGluon training complete, total runtime = 121.37s ...
CPU times: user 1min 27s, sys: 6.04 s, total: 1min 33s
Wall time: 2min 1s

You can see a summary of the model built with fit_summary. This time, the result is that the ensemble model has the highest accuracy = 0.864.


results = predictor.fit_summary()

*** Summary of fit() ***
Estimated performance of each model:
                         model  score_val   fit_time  pred_time_val  stack_level
10     weighted_ensemble_k0_l1      0.864   0.822482       0.002695            1
7           CatboostClassifier      0.864   8.849893       0.016495            0
8          NeuralNetClassifier      0.854  93.040479       0.263205            0
6           LightGBMClassifier      0.854   1.208874       0.018122            0
9     LightGBMClassifierCustom      0.852   2.201691       0.032273            0
1   RandomForestClassifierEntr      0.846   3.039360       0.218410            0
0   RandomForestClassifierGini      0.842   2.443804       0.224741            0
3     ExtraTreesClassifierEntr      0.836   2.247025       0.215692            0
2     ExtraTreesClassifierGini      0.830   1.930307       0.217770            0
4     KNeighborsClassifierUnif      0.772   0.072491       0.113025            0
5     KNeighborsClassifierDist      0.748   0.035974       0.112589            0
Number of models trained: 11
Types of models trained:
{'CatboostModel', 'LGBModel', 'RFModel', 'KNNModel', 'WeightedEnsembleModel', 'TabularNeuralNetModel'}
Bagging used: False 
Stack-ensembling used: False 
Hyperparameter-tuning used: False 
User-specified hyperparameters:
{'NN': {'num_epochs': 500}, 'GBM': {'num_boost_round': 10000}, 'CAT': {'iterations': 10000}, 'RF': {'n_estimators': 300}, 'XT': {'n_estimators': 300}, 'KNN': {}, 'custom': ['GBM']}
Plot summary of models saved to file: agModels-predictClass/SummaryOfModels.html
*** End of fit() summary ***

Now let's make a prediction for the test data and calculate the accuracy. Perform the test data in the same way as in the tutorial.

test_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv')
y_test = test_data[label_column]  # values to predict
test_data_nolab = test_data.drop(labels=[label_column],axis=1) # delete label column to prove we're not cheating
print(test_data_nolab.head())
Loaded data from: https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv | Columns = 15 / 15 | Rows = 9769 -> 9769
   age          workclass  fnlwgt  ... capital-loss  hours-per-week  native-country
0   31            Private  169085  ...            0              20   United-States
1   17   Self-emp-not-inc  226203  ...            0              45   United-States
2   47            Private   54260  ...         1887              60   United-States
3   21            Private  176262  ...            0              30   United-States
4   17            Private  241185  ...            0              20   United-States

[5 rows x 14 columns]

If the prediction and training are separate scripts, use load to load the model. There is also an API for accuracy calculation, so use that.

predictor = task.load(dir) # unnecessary, just demonstrates how to load previously-trained predictor from file

y_pred = predictor.predict(test_data_nolab)
print("Predictions:  ", y_pred)
perf = predictor.evaluate_predictions(y_true=y_test, y_pred=y_pred, auxiliary_metrics=True)
Evaluation: accuracy on test data: 0.8722489507626164
Evaluations on test data:
{
    "accuracy": 0.8722489507626164,
    "accuracy_score": 0.8722489507626164,
    "balanced_accuracy_score": 0.796036984340255,
    "matthews_corrcoef": 0.6303973015503467,
    "f1_score": 0.8722489507626164
}
Predictions:   [' <=50K' ' <=50K' ' >50K' ... ' <=50K' ' <=50K' ' <=50K']
Detailed (per-class) classification report:
{
    " <=50K": {
        "precision": 0.8965605421301623,
        "recall": 0.9410817339954368,
        "f1-score": 0.9182818229439498,
        "support": 7451
    },
    " >50K": {
        "precision": 0.7746406570841889,
        "recall": 0.6509922346850734,
        "f1-score": 0.7074542897327709,
        "support": 2318
    },
    "accuracy": 0.8722489507626164,
    "macro avg": {
        "precision": 0.8356005996071756,
        "recall": 0.796036984340255,
        "f1-score": 0.8128680563383603,
        "support": 9769
    },
    "weighted avg": {
        "precision": 0.8676312460367478,
        "recall": 0.8722489507626164,
        "f1-score": 0.8682564137942402,
        "support": 9769
    }
}

The result was an Accuracy of 0.872. I thought it was convenient because the f1 score etc. are also displayed in a list.

When hyper parameter tuning is enabled

Now, I would like to test how much the accuracy changes when hyperparameter tuning is performed. The code is based here.

https://autogluon.mxnet.io/tutorials/tabular_prediction/tabular-indepth.html#

%%time
nn_options = { # specifies non-default hyperparameter values for neural network models
    'num_epochs': 10, # number of training epochs (controls training time of NN models)
    'learning_rate': ag.space.Real(1e-4, 1e-2, default=5e-4, log=True), # learning rate used in training (real-valued hyperparameter searched on log-scale)
    'activation': ag.space.Categorical('relu', 'softrelu', 'tanh'), # activation function used in NN (categorical hyperparameter, default = first entry)
    'layers': ag.space.Categorical([100],[1000],[200,100],[300,200,100]),
      # Each choice for categorical hyperparameter 'layers' corresponds to list of sizes for each NN layer to use
    'dropout_prob': ag.space.Real(0.0, 0.5, default=0.1), # dropout probability (real-valued hyperparameter)
}

gbm_options = { # specifies non-default hyperparameter values for lightGBM gradient boosted trees
    'num_boost_round': 100, # number of boosting rounds (controls training time of GBM models)
    'num_leaves': ag.space.Int(lower=26, upper=66, default=36), # number of leaves in trees (integer hyperparameter)
}

hyperparameters = {'NN': nn_options, 'GBM': gbm_options}  # hyperparameters of each model type
# If one of these keys is missing from hyperparameters dict, then no models of that type are trained.

time_limits = 2*60  # train various models for ~2 min
num_trials = 5  # try at most 3 different hyperparameter configurations for each type of model
search_strategy = 'skopt'  # to tune hyperparameters using SKopt Bayesian optimization routine
output_directory = 'agModels-predictOccupation'  # folder where to store trained models

dir = 'agModels-predictClass-tuned' # specifies folder where to store trained models
predictor = task.fit(
    train_data = train_data,         #Training data
    label = label_column,            #Objective variable
    tuning_data = None,              #Verification data for hyperparameter tuning
    output_directory = dir,          #Storage folder path for models and intermediate files
    time_limits=time_limits,         #Execution time limit(Unit: seconds)
    num_trials=num_trials,           #Maximum number of hyperparameter sets to try
    hyperparameter_tune=True,        #Whether to tune hyperparameters. Default is False
    hyperparameters=hyperparameters, #Specifying hyper parameters
    search_strategy=search_strategy  #How to search for hyper parameter tuning
)

Since the number of hyperparameters that are tuned by default for each model is small, it is necessary to specify the parameters to be tuned for each model. See the API reference (https://autogluon.mxnet.io/api/autogluon.task.html) for details. As the sample, this time we specify the Neural Network and LightGBM parameters.

Warning: `hyperparameter_tune=True` is currently experimental and may cause the process to hang. Setting `auto_stack=True` instead is recommended to achieve maximum quality models.
Beginning AutoGluon training ... Time limit = 120s
AutoGluon will save models to agModels-predictClass-tuned/
Train Data Rows:    10000
Train Data Columns: 15
Preprocessing data ...
Here are the first 10 unique label values in your data:  [' <=50K' ' >50K']
AutoGluon infers your prediction problem is: binary  (because only two unique label-values observed)
If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])

Selected class <--> label mapping:  class 1 =  >50K, class 0 =  <=50K
Feature Generator processed 10000 data points with 14 features
Original Features:
	int features: 6
	object features: 8
Generated Features:
	int features: 0
All Features:
	int features: 6
	object features: 8
	Data preprocessing and feature engineering runtime = 0.12s ...
AutoGluon will gauge predictive performance using evaluation metric: accuracy
To change this, specify the eval_metric argument of fit()
AutoGluon will early stop models using evaluation metric: accuracy
Starting Experiments
Num of Finished Tasks is 0
Num of Pending Tasks is 5
100%
5/5 [01:06<00:00, 13.34s/it]
Finished Task with config: {'feature_fraction': 1.0, 'learning_rate': 0.1, 'min_data_in_leaf': 20, 'num_leaves': 36} and reward: 0.859
Finished Task with config: {'feature_fraction': 0.7568829632019213, 'learning_rate': 0.12424356064985256, 'min_data_in_leaf': 3, 'num_leaves': 55} and reward: 0.858
Finished Task with config: {'feature_fraction': 0.7564494924988235, 'learning_rate': 0.03775715772264416, 'min_data_in_leaf': 8, 'num_leaves': 27} and reward: 0.859
Finished Task with config: {'feature_fraction': 0.9870314769908329, 'learning_rate': 0.1442122206611739, 'min_data_in_leaf': 4, 'num_leaves': 35} and reward: 0.857

Finished Task with config: {'feature_fraction': 0.9339535543959474, 'learning_rate': 0.0050486520525173835, 'min_data_in_leaf': 10, 'num_leaves': 56} and reward: 0.815
	0.8645	 = Validation accuracy score
	1.44s	 = Training runtime
	0.04s	 = Validation runtime
	0.86	 = Validation accuracy score
	1.54s	 = Training runtime
	0.03s	 = Validation runtime
	0.862	 = Validation accuracy score
	1.4s	 = Training runtime
	0.04s	 = Validation runtime
	0.8605	 = Validation accuracy score
	1.33s	 = Training runtime
	0.03s	 = Validation runtime
	0.8155	 = Validation accuracy score
	1.61s	 = Training runtime
	0.04s	 = Validation runtime
Starting Experiments
Num of Finished Tasks is 0
Num of Pending Tasks is 5
80%
4/5 [00:57<00:13, 13.95s/it]
Finished Task with config: {'activation.choice': 0, 'dropout_prob': 0.1, 'embedding_size_factor': 1.0, 'layers.choice': 0, 'learning_rate': 0.0005, 'network_type.choice': 0, 'use_batchnorm.choice': 0, 'weight_decay': 1e-06} and reward: 0.8465
Finished Task with config: {'activation.choice': 0, 'dropout_prob': 0.4983419677765833, 'embedding_size_factor': 1.1583623866868233, 'layers.choice': 0, 'learning_rate': 0.0068602771070335935, 'network_type.choice': 0, 'use_batchnorm.choice': 0, 'weight_decay': 0.0004137874852187468} and reward: 0.847
	Ran out of time, stopping training early.
Finished Task with config: {'activation.choice': 2, 'dropout_prob': 0.23434709621043653, 'embedding_size_factor': 1.157370849535098, 'layers.choice': 1, 'learning_rate': 0.0024035089234857193, 'network_type.choice': 0, 'use_batchnorm.choice': 0, 'weight_decay': 3.3978628271485844e-09} and reward: 0.806
Please either provide filename or allow plot in get_training_curves
	0.848	 = Validation accuracy score
	23.43s	 = Training runtime
	0.57s	 = Validation runtime
	0.8475	 = Validation accuracy score
	21.99s	 = Training runtime
	0.31s	 = Validation runtime
	0.806	 = Validation accuracy score
	9.18s	 = Training runtime
	0.65s	 = Validation runtime
Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 119.88s of the 51.12s of remaining time.
	0.8645	 = Validation accuracy score
	0.8s	 = Training runtime
	0.0s	 = Validation runtime
AutoGluon training complete, total runtime = 69.75s ...
CPU times: user 53.8 s, sys: 6.07 s, total: 59.9 s
Wall time: 1min 9s

hp_tune.png

The result of the fit is as follows.

results = predictor.fit_summary()

*** Summary of fit() ***
Estimated performance of each model:
                          model  score_val   fit_time  pred_time_val  stack_level
8       weighted_ensemble_k0_l1     0.8645   0.804953       0.001838            1
0   LightGBMClassifier/trial_15     0.8645   1.438680       0.037270            0
2   LightGBMClassifier/trial_17     0.8620   1.400933       0.039980            0
3   LightGBMClassifier/trial_18     0.8605   1.325390       0.030934            0
1   LightGBMClassifier/trial_16     0.8600   1.536028       0.026618            0
5  NeuralNetClassifier/trial_20     0.8480  23.431652       0.566203            0
6  NeuralNetClassifier/trial_21     0.8475  21.988767       0.310336            0
4   LightGBMClassifier/trial_19     0.8155   1.610676       0.038171            0
7  NeuralNetClassifier/trial_22     0.8060   9.177221       0.650193            0
Number of models trained: 9
Types of models trained:
{'WeightedEnsembleModel', 'LGBModel', 'TabularNeuralNetModel'}
Bagging used: False 
Stack-ensembling used: False 
Hyperparameter-tuning used: True 
User-specified hyperparameters:
{'NN': {'num_epochs': 10, 'learning_rate': Real: lower=0.0001, upper=0.01, 'activation': Categorical['relu', 'softrelu', 'tanh'], 'layers': Categorical[[100], [1000], [200, 100], [300, 200, 100]], 'dropout_prob': Real: lower=0.0, upper=0.5}, 'GBM': {'num_boost_round': 100, 'num_leaves': Int: lower=26, upper=66}}
Plot summary of models saved to file: agModels-predictClass-tuned/SummaryOfModels.html
Plot summary of models saved to file: agModels-predictClass-tuned/LightGBMClassifier_HPOmodelsummary.html
Plot summary of models saved to file: LightGBMClassifier_HPOmodelsummary.html
Plot of HPO performance saved to file: agModels-predictClass-tuned/LightGBMClassifier_HPOperformanceVStrials.png

Let's calculate the accuracy for the test data.

y_pred = predictor.predict(test_data_nolab)
print("Predictions:  ", y_pred)
perf = predictor.evaluate_predictions(y_true=y_test, y_pred=y_pred, auxiliary_metrics=True)
Evaluation: accuracy on test data: 0.8688709182106664
Evaluations on test data:
{
    "accuracy": 0.8688709182106664,
    "accuracy_score": 0.8688709182106664,
    "balanced_accuracy_score": 0.7895131714141826,
    "matthews_corrcoef": 0.6195102922288485,
    "f1_score": 0.8688709182106664
}
Predictions:   [' <=50K' ' <=50K' ' >50K' ... ' <=50K' ' <=50K' ' <=50K']
Detailed (per-class) classification report:
{
    " <=50K": {
        "precision": 0.8931939841957686,
        "recall": 0.9405448933029124,
        "f1-score": 0.9162580898215336,
        "support": 7451
    },
    " >50K": {
        "precision": 0.7696307852314093,
        "recall": 0.6384814495254529,
        "f1-score": 0.6979485970290026,
        "support": 2318
    },
    "accuracy": 0.8688709182106664,
    "macro avg": {
        "precision": 0.8314123847135889,
        "recall": 0.7895131714141826,
        "f1-score": 0.8071033434252681,
        "support": 9769
    },
    "weighted avg": {
        "precision": 0.8638747606110224,
        "recall": 0.8688709182106664,
        "f1-score": 0.8644573523567893,
        "support": 9769
    }

When AutoStack is enabled

If you enable the option of ʻauto_stack, Multi-layer stack ensemble will be executed. A note is given at the bottom of the sample code for this ʻauto-stack option. The most worrisome thing is that I don't enable the hyper parameter tuning option. Actually, I thought that I would like to ensemble individual models while tuning hyper parameters, but it seems that it is not recommended at present.

https://autogluon.mxnet.io/tutorials/tabular_prediction/tabular-indepth.html#maximizing-predictive-performance

%%time

long_time = 30*60 # for quick demonstration only, you should set this to longest time you are willing to wait
dir = 'agModels-predictClass-autostack' # specifies folder where to store trained models
predictor = task.fit(
    train_data=train_data, 
    label=label_column, 
    auto_stack=True, 
    output_directory = dir,
    time_limits=long_time)
Beginning AutoGluon training ... Time limit = 1800s
AutoGluon will save models to agModels-predictClass-autostack/
Train Data Rows:    10000
Train Data Columns: 15
Preprocessing data ...
Here are the first 10 unique label values in your data:  [' <=50K' ' >50K']
AutoGluon infers your prediction problem is: binary  (because only two unique label-values observed)
If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])

Selected class <--> label mapping:  class 1 =  >50K, class 0 =  <=50K
Feature Generator processed 10000 data points with 14 features
Original Features:
	int features: 6
	object features: 8
Generated Features:
	int features: 0
All Features:
	int features: 6
	object features: 8
	Data preprocessing and feature engineering runtime = 0.1s ...
AutoGluon will gauge predictive performance using evaluation metric: accuracy
To change this, specify the eval_metric argument of fit()
AutoGluon will early stop models using evaluation metric: accuracy
Fitting model: RandomForestClassifierGini_STACKER_l0 ... Training model for up to 899.95s of the 1799.89s of remaining time.
	0.8571	 = Validation accuracy score
	20.36s	 = Training runtime
	1.13s	 = Validation runtime
Fitting model: RandomForestClassifierEntr_STACKER_l0 ... Training model for up to 876.77s of the 1776.72s of remaining time.
	0.859	 = Validation accuracy score
	25.58s	 = Training runtime
	1.34s	 = Validation runtime
Fitting model: ExtraTreesClassifierGini_STACKER_l0 ... Training model for up to 847.93s of the 1747.87s of remaining time.
	0.8486	 = Validation accuracy score
	16.65s	 = Training runtime
	2.13s	 = Validation runtime
Fitting model: ExtraTreesClassifierEntr_STACKER_l0 ... Training model for up to 824.45s of the 1724.4s of remaining time.
	0.8478	 = Validation accuracy score
	18.99s	 = Training runtime
	2.14s	 = Validation runtime
Fitting model: KNeighborsClassifierUnif_STACKER_l0 ... Training model for up to 799.25s of the 1699.2s of remaining time.
	0.7694	 = Validation accuracy score
	0.33s	 = Training runtime
	1.11s	 = Validation runtime
Fitting model: KNeighborsClassifierDist_STACKER_l0 ... Training model for up to 797.76s of the 1697.7s of remaining time.
	0.751	 = Validation accuracy score
	0.3s	 = Training runtime
	1.11s	 = Validation runtime
Fitting model: LightGBMClassifier_STACKER_l0 ... Training model for up to 796.29s of the 1696.24s of remaining time.
	0.874	 = Validation accuracy score
	11.52s	 = Training runtime
	0.17s	 = Validation runtime
Fitting model: CatboostClassifier_STACKER_l0 ... Training model for up to 784.49s of the 1684.44s of remaining time.
	0.8735	 = Validation accuracy score
	54.41s	 = Training runtime
	0.13s	 = Validation runtime
Fitting model: NeuralNetClassifier_STACKER_l0 ... Training model for up to 729.9s of the 1629.85s of remaining time.
	Ran out of time, stopping training early.
	Ran out of time, stopping training early.
	Ran out of time, stopping training early.
	Ran out of time, stopping training early.
	0.8591	 = Validation accuracy score
	641.59s	 = Training runtime
	3.02s	 = Validation runtime
Fitting model: LightGBMClassifierCustom_STACKER_l0 ... Training model for up to 85.08s of the 985.02s of remaining time.
	0.8673	 = Validation accuracy score
	23.17s	 = Training runtime
	0.28s	 = Validation runtime
Completed 1/20 k-fold bagging repeats ...
Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 360.0s of the 960.22s of remaining time.
	0.874	 = Validation accuracy score
	3.57s	 = Training runtime
	0.0s	 = Validation runtime
Fitting model: RandomForestClassifierGini_STACKER_l1 ... Training model for up to 956.64s of the 956.63s of remaining time.
	0.873	 = Validation accuracy score
	32.32s	 = Training runtime
	1.76s	 = Validation runtime
Fitting model: RandomForestClassifierEntr_STACKER_l1 ... Training model for up to 921.46s of the 921.45s of remaining time.
	0.8745	 = Validation accuracy score
	45.88s	 = Training runtime
	1.36s	 = Validation runtime
Fitting model: ExtraTreesClassifierGini_STACKER_l1 ... Training model for up to 873.32s of the 873.31s of remaining time.
	0.871	 = Validation accuracy score
	17.69s	 = Training runtime
	2.16s	 = Validation runtime
Fitting model: ExtraTreesClassifierEntr_STACKER_l1 ... Training model for up to 850.66s of the 850.65s of remaining time.
	0.8703	 = Validation accuracy score
	20.02s	 = Training runtime
	2.15s	 = Validation runtime
Fitting model: KNeighborsClassifierUnif_STACKER_l1 ... Training model for up to 825.65s of the 825.64s of remaining time.
	0.7685	 = Validation accuracy score
	0.49s	 = Training runtime
	1.13s	 = Validation runtime
Fitting model: KNeighborsClassifierDist_STACKER_l1 ... Training model for up to 823.94s of the 823.93s of remaining time.
	0.7494	 = Validation accuracy score
	0.41s	 = Training runtime
	1.13s	 = Validation runtime
Fitting model: LightGBMClassifier_STACKER_l1 ... Training model for up to 822.3s of the 822.3s of remaining time.
	0.8783	 = Validation accuracy score
	12.55s	 = Training runtime
	0.18s	 = Validation runtime
Fitting model: CatboostClassifier_STACKER_l1 ... Training model for up to 809.49s of the 809.48s of remaining time.
	0.878	 = Validation accuracy score
	46.66s	 = Training runtime
	0.15s	 = Validation runtime
Fitting model: NeuralNetClassifier_STACKER_l1 ... Training model for up to 762.64s of the 762.63s of remaining time.
	Ran out of time, stopping training early.
	0.873	 = Validation accuracy score
	543.08s	 = Training runtime
	3.31s	 = Validation runtime
Fitting model: LightGBMClassifierCustom_STACKER_l1 ... Training model for up to 216.03s of the 216.03s of remaining time.
	0.8741	 = Validation accuracy score
	25.99s	 = Training runtime
	0.21s	 = Validation runtime
Completed 1/20 k-fold bagging repeats ...
Fitting model: weighted_ensemble_k0_l2 ... Training model for up to 360.0s of the 189.19s of remaining time.
	0.8784	 = Validation accuracy score
	3.56s	 = Training runtime
	0.0s	 = Validation runtime
AutoGluon training complete, total runtime = 1614.4s ...
CPU times: user 22min 10s, sys: 1min 8s, total: 23min 19s
Wall time: 26min 54s

Similarly, when calculating the accuracy for the test data, the accuracy was about 0.871.

y_pred = predictor.predict(test_data_nolab)
print("Predictions:  ", y_pred)
perf = predictor.evaluate_predictions(y_true=y_test, y_pred=y_pred, auxiliary_metrics=True)

Evaluation: accuracy on test data: 0.8705087521752483
Evaluations on test data:
{
    "accuracy": 0.8705087521752483,
    "accuracy_score": 0.8705087521752483,
    "balanced_accuracy_score": 0.7893980679524981,
    "matthews_corrcoef": 0.6232120872465466,
    "f1_score": 0.8705087521752483
}
Predictions:   [' <=50K' ' <=50K' ' >50K' ... ' <=50K' ' <=50K' ' <=50K']
Detailed (per-class) classification report:
{
    " <=50K": {
        "precision": 0.8926123381568926,
        "recall": 0.9437659374580594,
        "f1-score": 0.9174766781916628,
        "support": 7451
    },
    " >50K": {
        "precision": 0.7784241142252777,
        "recall": 0.635030198446937,
        "f1-score": 0.6994535519125684,
        "support": 2318
    },
    "accuracy": 0.8705087521752483,
    "macro avg": {
        "precision": 0.8355182261910852,
        "recall": 0.7893980679524981,
        "f1-score": 0.8084651150521156,
        "support": 9769
    },
    "weighted avg": {
        "precision": 0.8655176198568123,
        "recall": 0.8705087521752483,
        "f1-score": 0.8657438901156119,
        "support": 9769
    }
}

When the index is changed

You can select any metric you like for the argument of ʻeval_metric in fit`. The default is accuracy for binary classification.


%%time
dir = 'agModels-predictClass-defaultlogloss' # specifies folder where to store trained models
predictor = task.fit(train_data=train_data, label=label_column, eval_metric='log_loss', output_directory=dir)

When I tried log_loss, I got the following Runtime Warning. It looks like it's working fine for some models, but I'm not sure if it's working in the end ...

/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:2295: RuntimeWarning: divide by zero encountered in log
  loss = -(transformed_labels * np.log(y_pred)).sum(axis=1)
/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:2295: RuntimeWarning: invalid value encountered in multiply
  loss = -(transformed_labels * np.log(y_pred)).sum(axis=1)

When feature quantity is selected

One of the arguments of fit was feature_prune, so I tried it. Below is the code.

%%time
dir = 'agModels-predictClass-defaultprune' # specifies folder where to store trained models
predictor = task.fit(train_data=train_data, label=label_column, feature_prune=True, output_directory=dir)

Immediately after executing it, the following warning was issued. It doesn't seem to work yet.

Warning: feature_prune does not currently work, setting to False.

Summary

I've tried the official sample code and some argument options. There seems to be some things that aren't working, so I'm looking forward to future updates.

Recommended Posts

I tried Auto Gluon
I tried scraping
I tried PyQ
I tried AutoKeras
I tried papermill
I tried django-slack
I tried Django
I tried spleeter
I tried cgo
I tried using parameterized
I tried using argparse
I tried using mimesis
I tried competitive programming
I tried running pymc
I tried ARP spoofing
I tried using aiomysql
I tried using Summpy
I tried Python> autopep8
I tried using Pipenv
I tried using matplotlib
I tried using ESPCN
I tried PyCaret2.0 (pycaret-nightly)
I tried using openpyxl
I tried deep learning
I tried AWS CDK!
I tried using Ipython
I tried to debug.
I tried using PyCaret
I tried using cron
I tried Kivy's mapview
I tried using ngrok
I tried using face_recognition
I tried to paste
I tried using Jupyter
I tried using PyCaret
I tried moving EfficientDet
I tried shell programming
I tried using Heapq
I tried using doctest
I tried Python> decorator
I tried running TensorFlow
I tried using folium
I tried using jinja2
I tried AWS Iot
I tried Bayesian optimization!
I tried using folium
I tried using time-window
I tried Value Iteration Networks
I tried fp-growth with python
I tried scraping with Python
I tried AutoGluon's Image Classification
I tried to learn PredNet
I tried Learning-to-Rank with Elasticsearch!
[I tried using Pythonista 3] Introduction
I tried using easydict (memo).
I tried to organize SVM.
I tried face recognition using Face ++
I tried using Random Forest
I tried clustering with PyCaret
I tried using BigQuery ML
I tried "K-Fold Target Encoding"