Any Advice on Avoiding 'NaN' errors

[dfossl]

Sometimes when I run this:

test_model = HyperoptEstimator(regressor=gradient_boosting_regressor('test_regres'),
                            preprocessing=[],
                            algo=tpe.suggest,
                            max_evals=50,
                            trial_timeout=30)
test_model.fit(X_train.to_numpy(), y_train.to_numpy().ravel())

I get an error ValueError: Input contains NaN. during training. It doesn't happen every time and I know that the data has no nan's, infinites, or duplicates. This leads me to believe one of the operations is creating a NaN. Is there anyway to skip these operations or deduce what operation is causing this?

[mandjevant]

Hey there, could you provide me with a sample of your training data?
I am unable to reproduce this issue.

[RaistlinTAO]

I just got same error during HyperoptEstimator search, hope the following log helps

2|hpsklearnBlack  | job exception: Input contains NaN.
 98%|█████████▊| 48/49 [00:02<?, ?trial/s, best loss=?]
2|hpsklearnBlack  | Traceback (most recent call last):
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/Test/playground_gbm.py", line 48, in <module>
2|hpsklearnBlack  |     find_best_model(X_train, y_train, X_test, y_test)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/Test/playground_gbm.py", line 29, in find_best_model
2|hpsklearnBlack  |     estimator.fit(x, y)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hpsklearn/estimator/estimator.py", line 464, in fit
2|hpsklearnBlack  |     fit_iter.send(increment)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hpsklearn/estimator/estimator.py", line 339, in fit_iter
2|hpsklearnBlack  |     hyperopt.fmin(_fn_with_timeout,
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 540, in fmin
2|hpsklearnBlack  |     return trials.fmin(
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/base.py", line 671, in fmin
2|hpsklearnBlack  |     return fmin(
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 586, in fmin
2|hpsklearnBlack  |     rval.exhaust()
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 364, in exhaust
2|hpsklearnBlack  |     self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 300, in run
2|hpsklearnBlack  |     self.serial_evaluate()
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 178, in serial_evaluate
2|hpsklearnBlack  |     result = self.domain.evaluate(spec, ctrl)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/base.py", line 892, in evaluate
2|hpsklearnBlack  |     rval = self.fn(pyll_rval)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hpsklearn/estimator/estimator.py", line 311, in _fn_with_timeout
2|hpsklearnBlack  |     raise fn_rval[1]
2|hpsklearnBlack  | ValueError: Input contains NaN.

According the result from df.isnull().sum() there is no NaN in data, hence, the NaH error occurs during parameter injection.

[mandjevant]

@RaistlinTAO Could I get a snippet of your code please?
I understand that your data does not contain nan values. But I need to see what code exactly is causing the issue.

[RaistlinTAO]

@RaistlinTAO Could I get a snippet of your code please? I understand that your data does not contain nan values. But I need to see what code exactly is causing the issue.

Of course, happy to help

def find_best_model(x, y, test_x, test_y):
    estimator = HyperoptEstimator(
        regressor=gradient_boosting_regressor("T"),
        algo=tpe.suggest,
        max_evals=800,
        trial_timeout=300)
    estimator.fit(x, y)
    print('HyperoptEstimator Score: ')
    print(estimator.score(test_x, test_y))
    print('Best Model: ')
    print(estimator.best_model())

find_best_model(X_train, y_train, X_test, y_test)

[RaistlinTAO]

Update:

With same code and same settings, it will bypass the error after give it another 3-5 tries.

I think it related to hyperparameter combination. sometimes it just skip the wrong combination or preprocessing

[mandjevant]

Thanks a lot @RaistlinTAO that is what I was thinking as well. I've noticed this behaviour before.
Unfortunately, I cannot reproduce the error in our testing environment so it is a difficult issue to fix.
Your code snippet will help me to point down the problem and focus my testing.

I will work on a fix for this. But for the time being, retrying a few times should bypass the error.

[RaistlinTAO]

After several attempts on debug in local env I have located the root cause.

In sklearn\ensemble_gb_losses.py
update_terminal_regions() statement:

    def _update_terminal_region(...):
        ...
        diff_minus_median = diff - median
        ...

# and 

def update_terminal_regions(...):
        raw_predictions[:, k] += learning_rate * tree.predict(X).ravel()

SAMPLE OUTPUT

E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:231: RuntimeWarning: overflow encountered in square
  * np.sum(sample_weight * ((y - raw_predictions.ravel()) ** 2))
E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:231: RuntimeWarning: overflow encountered in square
  * np.sum(sample_weight * ((y - raw_predictions.ravel()) ** 2))
E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:288: RuntimeWarning: overflow encountered in multiply
  raw_predictions[:, k] += learning_rate * tree.predict(X).ravel()
E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:288: RuntimeWarning: invalid value encountered in add
  raw_predictions[:, k] += learning_rate * tree.predict(X).ravel()
 97%|█████████▋| 32/33 [00:06<?, ?trial/s, best loss=?]
job exception: Input contains NaN.

There are two different outputs indicate that diff/median/learning_rate is NaN under certain circumstances.
I ll just leave this here since my hands are tied atm.

[crieux]

Hello,
I get the same issue, is there a fix for this one ?

 self.estim = HyperoptEstimator(
            regressor = any_regressor("my_clf"),
            preprocessing = [],
            algo = tpe.suggest,
            max_evals = 65,
            trial_timeout = 120
            )

Stack trace :

100%|██████████| 1/1 [00:00<00:00,  5.23trial/s, best loss: 0.05266970464190446]
100%|██████████| 2/2 [00:00<00:00,  2.18trial/s, best loss: 0.033195277678000346]
100%|██████████| 3/3 [00:00<00:00,  3.74trial/s, best loss: 0.033195277678000346]
100%|██████████| 4/4 [00:02<00:00,  2.30s/trial, best loss: 0.03245292544638456]
100%|██████████| 5/5 [00:00<00:00, 16.75trial/s, best loss: 0.03245292544638456]
100%|██████████| 6/6 [00:00<00:00, 13.88trial/s, best loss: 0.03245292544638456]
100%|██████████| 7/7 [00:00<00:00,  1.38trial/s, best loss: 0.03245292544638456]
100%|██████████| 8/8 [00:01<00:00,  1.57s/trial, best loss: 0.03245292544638456]
100%|██████████| 9/9 [00:00<00:00, 13.97trial/s, best loss: 0.03245292544638456]
100%|██████████| 10/10 [00:00<00:00, 16.76trial/s, best loss: 0.03245292544638456]
100%|██████████| 11/11 [00:00<00:00,  7.96trial/s, best loss: 0.03245292544638456]
100%|██████████| 12/12 [00:00<00:00,  1.48trial/s, best loss: 0.03245292544638456]
100%|██████████| 13/13 [00:00<00:00,  2.03trial/s, best loss: 0.030216599846460745]
100%|██████████| 14/14 [00:00<00:00, 16.23trial/s, best loss: 0.030216599846460745]
100%|██████████| 15/15 [00:00<00:00,  2.10trial/s, best loss: 0.030216599846460745]
100%|██████████| 16/16 [00:00<00:00, 17.83trial/s, best loss: 0.030216599846460745]
100%|██████████| 17/17 [00:00<00:00,  1.30trial/s, best loss: 0.030216599846460745]
100%|██████████| 18/18 [00:00<00:00, 15.57trial/s, best loss: 0.030216599846460745]
100%|██████████| 19/19 [00:00<00:00,  1.66trial/s, best loss: 0.030216599846460745]
100%|██████████| 20/20 [00:00<00:00, 17.59trial/s, best loss: 0.030216599846460745]
100%|██████████| 21/21 [00:00<00:00, 10.58trial/s, best loss: 0.030216599846460745]
100%|██████████| 22/22 [00:00<00:00,  7.55trial/s, best loss: 0.030216599846460745]
100%|██████████| 23/23 [00:02<00:00,  2.01s/trial, best loss: 0.030216599846460745]
100%|██████████| 24/24 [02:00<00:00, 120.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 25/25 [00:02<00:00,  2.64s/trial, best loss: 0.030216599846460745]
100%|██████████| 26/26 [00:00<00:00,  7.15trial/s, best loss: 0.030216599846460745]
100%|██████████| 27/27 [00:01<00:00,  1.15s/trial, best loss: 0.030216599846460745]
100%|██████████| 28/28 [00:00<00:00,  1.77trial/s, best loss: 0.030216599846460745]
100%|██████████| 29/29 [00:00<00:00, 10.25trial/s, best loss: 0.030216599846460745]
100%|██████████| 30/30 [00:09<00:00,  9.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 31/31 [00:00<00:00, 10.90trial/s, best loss: 0.030216599846460745]
100%|██████████| 32/32 [00:01<00:00,  1.49s/trial, best loss: 0.027541941347137833]
 97%|█████████▋| 32/33 [00:00<?, ?trial/s, best loss=?]
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

[RaistlinTAO]

Hello, I get the same issue, is there a fix for this one ?

 self.estim = HyperoptEstimator(
            regressor = any_regressor("my_clf"),
            preprocessing = [],
            algo = tpe.suggest,
            max_evals = 65,
            trial_timeout = 120
            )

Stack trace :

100%|██████████| 1/1 [00:00<00:00,  5.23trial/s, best loss: 0.05266970464190446]
100%|██████████| 2/2 [00:00<00:00,  2.18trial/s, best loss: 0.033195277678000346]
100%|██████████| 3/3 [00:00<00:00,  3.74trial/s, best loss: 0.033195277678000346]
100%|██████████| 4/4 [00:02<00:00,  2.30s/trial, best loss: 0.03245292544638456]
100%|██████████| 5/5 [00:00<00:00, 16.75trial/s, best loss: 0.03245292544638456]
100%|██████████| 6/6 [00:00<00:00, 13.88trial/s, best loss: 0.03245292544638456]
100%|██████████| 7/7 [00:00<00:00,  1.38trial/s, best loss: 0.03245292544638456]
100%|██████████| 8/8 [00:01<00:00,  1.57s/trial, best loss: 0.03245292544638456]
100%|██████████| 9/9 [00:00<00:00, 13.97trial/s, best loss: 0.03245292544638456]
100%|██████████| 10/10 [00:00<00:00, 16.76trial/s, best loss: 0.03245292544638456]
100%|██████████| 11/11 [00:00<00:00,  7.96trial/s, best loss: 0.03245292544638456]
100%|██████████| 12/12 [00:00<00:00,  1.48trial/s, best loss: 0.03245292544638456]
100%|██████████| 13/13 [00:00<00:00,  2.03trial/s, best loss: 0.030216599846460745]
100%|██████████| 14/14 [00:00<00:00, 16.23trial/s, best loss: 0.030216599846460745]
100%|██████████| 15/15 [00:00<00:00,  2.10trial/s, best loss: 0.030216599846460745]
100%|██████████| 16/16 [00:00<00:00, 17.83trial/s, best loss: 0.030216599846460745]
100%|██████████| 17/17 [00:00<00:00,  1.30trial/s, best loss: 0.030216599846460745]
100%|██████████| 18/18 [00:00<00:00, 15.57trial/s, best loss: 0.030216599846460745]
100%|██████████| 19/19 [00:00<00:00,  1.66trial/s, best loss: 0.030216599846460745]
100%|██████████| 20/20 [00:00<00:00, 17.59trial/s, best loss: 0.030216599846460745]
100%|██████████| 21/21 [00:00<00:00, 10.58trial/s, best loss: 0.030216599846460745]
100%|██████████| 22/22 [00:00<00:00,  7.55trial/s, best loss: 0.030216599846460745]
100%|██████████| 23/23 [00:02<00:00,  2.01s/trial, best loss: 0.030216599846460745]
100%|██████████| 24/24 [02:00<00:00, 120.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 25/25 [00:02<00:00,  2.64s/trial, best loss: 0.030216599846460745]
100%|██████████| 26/26 [00:00<00:00,  7.15trial/s, best loss: 0.030216599846460745]
100%|██████████| 27/27 [00:01<00:00,  1.15s/trial, best loss: 0.030216599846460745]
100%|██████████| 28/28 [00:00<00:00,  1.77trial/s, best loss: 0.030216599846460745]
100%|██████████| 29/29 [00:00<00:00, 10.25trial/s, best loss: 0.030216599846460745]
100%|██████████| 30/30 [00:09<00:00,  9.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 31/31 [00:00<00:00, 10.90trial/s, best loss: 0.030216599846460745]
100%|██████████| 32/32 [00:01<00:00,  1.49s/trial, best loss: 0.027541941347137833]
 97%|█████████▋| 32/33 [00:00<?, ?trial/s, best loss=?]
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

Well my suggestions are:

1. Use Optuna (https://github.com/optuna/optuna),
   however they have different issues, but it's rare.
2. Use GridSearchCV, but considering the time/resource consumption.

Pick any solution that fit your needs and have a good one

[zhukuixi]

I think this have sth to do with the gradient. Try to use gradient clip to avoid this problem when it comes to some extreme hyperparameter combination.

Apply gradient clipping to prevent exploding gradients, which can cause NaN values.