[PYTHON] What to do if you get the error Input contains NaN, infinity or a value too large for dtype ('float64'). In machine learning

What to do if you get the error "Input contains NaN, infinity or a value too large for dtype ('float64')." In machine learning

Cause of error

As stated in the error above, the data is mixed with non-NaN, inf or float64 types. I wrote a column containing a character string in pandas as follows, and even though I filled in Na, an error occurred.

df=df.drop(columns=df.select_dtypes(include='object').columns)

It seems that it is not possible to convert values other than float that are partly included in the column. So, after converting to ndarray so that it can be used with numpy, those with values other than float were converted to float type, those that could not be changed to float type were converted to Na, and then the missing values were converted.

X = df.iloc[:, 1:].values
y = df.iloc[:, 0].values
for i in range(X.shape[1]):
    X[:,i]= pd.to_numeric(X[:,i], errors='coerce')
X1=np.nan_to_num(X)

I'm not sure if this is the correct process, but for the time being, the error when training the model is gone. Please comment if there is any good way.

Recommended Posts

What to do if you get the error Input contains NaN, infinity or a value too large for dtype ('float64'). In machine learning
How to deal with scikit-learn's "ValueError: Input contains NaN, infinity or a value too large for dtype ('float64')."
What to do if you get a Cannot retrieve metalink for repository error in yum
What to do if you get a Call with too many input arguments error at DoAndReturn in a golang test
What to do if you get a "No versions found" error in pipenv
What to do if you get a minus zero in Python
What to do if you get a must override `get_config` error when trying to model.save in Keras
What to do if you get angry with "Value Error: unknown local: UTF-8" in python manage.py syncdb
What to do if a version error occurs in the selenium Chrome driver
What to do if you get an error when trying to send a message in tasks.loop () immediately after startup
What to do if you get the error RuntimeError: Python is not installed as a framework when trying to use matplitlib and pylab in Python 3.3
What to do if you get `No kernel for language python found` in Hydrogen
What to do if you get a memory error when converting from PySparkDataFrame to PandasDataFrame
What to do if you get "(35,'SSL connect error')" in pycurl (one of them)
What to do if you get an error when importing matplotlib in Python (Mac)
What to do if you get the error ʻERR_FEATURE_UNAVAILABLE_ON_PLATFORM` when using ts-node-dev on Linux
What to do if you get "coverage unknown" in Coveralls
What to do if a 0xC0000005 error occurs in tf.train.start_queue_runners ()
What to do if you get Swagger-codegen in python and Import Error: No module named
What to do if you get an error when running "certbot renew" in CakePHP environment
What to do if you get a UnicodeDecodeError with pip install
What to do if you can't use the trash in Lubuntu 18.04.
What to do if you cat or tail a binary file and the terminal is garbled
[Django] What to do if the model you want to create has a large number of fields
What to do if you get an error saying c compiler cannot create executables in configure
What to do if you get `locale.Error: unsupported locale setting` when getting the day of the week from a date in Python
What to do if you can't find PDO in Laravel or CakePHP
What to do if you get lost in file reference with FileNotFoundError
What to do if you get angry in TensorFlow v2 without attribute'app'
What to do if you get a TypeError with numpy min, max
What to do if you get an error when trying to load mnist
What to do if you get a Permission denied (public key) error when trying to pull on Github
[Python] What to do if you get a ModuleNotFoundError when importing pandas using Jupyter Notebook in Anaconda
What to do if you get the error Target WSGI script'/var/www/xxx/xxx.wsgi' cannot be loaded as python module
What to do if you get an error when installing Dlib (Ubuntu)
If you get a no attribute error in boto3, check the version
What to do if a Unicode Encode Error occurs in Sublime Text Python
What to do if you get an error when installing python with pyenv
What to do if you get "Python not configured." Using PyDev in Eclipse
[Python] What to check when you get a Unicode Decode Error in Django
What to do if you get the error "Error: opencv3: Does not support building both Python 2 and 3 wrappers" when installing openCV 3
What to do if you get an "unknown service" error from your gRPC server
What to do if your Jupyter Notebook for beginners asks for a password or token
What to do if you get an OpenSSL error when installing Python 2 with pyenv
What to do if the print command itself causes an error in Maya python
If you get the error "basis matrix is singular to working precision" in GLPK
What to do if you get an Import Error when importing matplotlib with Jupyter
What to do if you can't hit the arrow keys in the Python interactive console
What to do if you run python in IntelliJ and end with an error
What to do if a UnicodeDecodeError occurs in pip
What to do if you get an error when vagrant up when you enable public_network or private_network on Vagrant + Arch Linux → Install netctl
Make a note of what you want to do in the future with Raspberry Pi
What to do if you get an Undefined error when trying to use pip with pyenv
EC2 / Amazon Linux2: What to do if you get an "unable to execute'gcc': No such file or directory" error with pip install
What to do if pip gives a DistributionError in Homebrew
How to get the last (last) value in a list in Python
What to do if you can't log in as root
What to do if you grep a text file and it becomes Binary file (standard input) matches
What to do if pip --user returns an error in a virtual environment created with pyenv
What to do when you get "I can't see the site !!!!"
What to do when the value type is ambiguous in Python?