Building an environment for natural language processing with Python

This is a memo for myself, but for your reference.

1. Version after environment construction

2-1. [For Mac] Installing Python with Homebrew

Install "Homebrew"   ↓ Install "pyenv" using "Homebrew"   ↓ Install "Python" using "pyenv"

(1) Installation of Homebrew

Official Homebrew documentation: https://brew.sh/index_ja

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

You will be asked to press the return / enter key on the way, so press it according to the instructions. You will also be asked to enter the PW when logging in to your PC on the way. Enter the password according to the instructions (however, the entered contents are not displayed on the screen), and press the return / enter key.

After the installation is complete, check the version with the following command.

$ brew -v
Homebrew 2.2.2
Homebrew/homebrew-core (git revision d7286b; last commit 2020-01-01)

(2) Installation of pyenv

Install pyenv (a tool for building and managing multiple versions of Python environment in the same terminal) using Homebrew. pyenv's GitHub repository: https://github.com/pyenv

$ brew install pyenv

After the installation is complete, check the version with the following command.

$ pyenv -v
pyenv 1.2.13

pyenv settings: Added setting code to .bash_profile. Execute the following command line by line (it looks like nothing is happening, but that's OK).

$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
$ echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
$ source ~/.bash_profile

(3) Python installation

Install Python using pyenv.

First, execute the following command to check the version of Python that can be installed with pyenv.

$ pyenv install --list
#Python version that can install the execution result without the first alphabet
 2.1.3
 2.2.3
 2.3.7
・ ・ ・
 3.6.5
 3.6.6
 3.6.7
・ ・ ・

Of these, I installed the relatively new Python 3.6.5.

$ pyenv install 3.6.5

After the installation is complete, check the version of Python installed with the following command.

$ pyenv versions
system
* 3.6.5 (set by /Users/hoge/.pyenv/version)

Execute the following command to change the setting to use the version of Python installed above (otherwise this will continue to use the version of Python that is installed by default on your Mac).

$ pyenv global 3.6.5

Execute the following command to check the version of Python you are using.

$ Python --version
Python 3.6.5

2-2. [For Windows] Install Python from the official website

Select Release version = Python 3.6.5 from the following site, download it, and install it according to the instructions.

Python 3.6.5 download page (official): https://www.python.org/downloads/release/python-365/

It seems that any file to download is for Windows, but unless you are particular about it, select Windows x86-64 executable installer.

When you open the downloaded file, there is a checkbox at the bottom of the screen that says ʻAdd Python 3.6 to PATH. Check this and click ʻInstall Now.

After the installation is complete, execute the following command at the command prompt (so-called black screen) to check the version.

$ Python --version
Python 3.6.5

settings.json


{
    "terminal.integrated.shell.windows": "C:\\WINDOWS\\System32\\cmd.exe"
}

3. Installation of required libraries [Common to Mac / windows]

(1) Install Numpy

$ pip install numpy===1.15.0

(2) Installation of SciPy

$ pip install scipy===1.1.0

(3) Install Pandas

$ pip install pandas===0.23.3

(4) Installation of Scikit-Learn

$ pip install scikit-learn===0.19.2

(5) Installation of MeCab

$ pip install mecab-python3===0.996.2
$ pip install ipykernel
$ pip install mecab-python-windows

(6) Installation of neplogdn

$ pip install neologdn===0.3.2
$ git clone --depth 1 https://github.com/neologd/mecab-ipadic-neologd.git
$ cd mecab-ipadic-neologd
$ ./bin/install-mecab-ipadic-neologd -n -a

[install-mecab-ipadic-NEologd] : Start..
[install-mecab-ipadic-NEologd] : Check the existance of libraries
・ ・ ・
[test-mecab-ipadic-NEologd] : Please check difference between default system dictionary and mecab-ipadic-NEologd

default system dictionary        |     mecab-ipadic-NEologd
Shiro Sano|Shiro Sano
Police academy|Police academy
Mizore|Sleet
Around|Colon
Tarako|Tarakochi
Ichijodani Castle|Ichijodani Castle
Radirer!|Rajira
Kento Hayashi|Kento Hayashi
Matsuko meeting|Matsuko meeting
Ad street|Ad street

[test-mecab-ipadic-NEologd] : Finish..
[install-mecab-ipadic-NEologd] : Please check the list of differences in the upper part.
[install-mecab-ipadic-NEologd] : Do you want to install mecab-ipadic-NEologd? Type yes or no.
yes #Here "Enter yes"
[install-mecab-ipadic-NEologd] : OK. Let's install mecab-ipadic-NEologd.
[install-mecab-ipadic-NEologd] : Start..
[install-mecab-ipadic-NEologd] : /usr/local/lib/mecab/dic is current user's directory
・ ・ ・
[install-mecab-ipadic-NEologd] : Install completed.
[install-mecab-ipadic-NEologd] : When you use MeCab, you can set '/usr/local/lib/mecab/dic/mecab-ipadic-neologd' as a value of '-d' option of MeCab.
[install-mecab-ipadic-NEologd] : Usage of mecab-ipadic-NEologd is here.
Usage:
    $ mecab -d /usr/local/lib/mecab/dic/mecab-ipadic-neologd ...

[install-mecab-ipadic-NEologd] : Finish..
[install-mecab-ipadic-NEologd] : Finish..

#Installation is complete. Specify NEologd as the dictionary used by mecab with the following command
$ mecab -d /usr/local/lib/mecab/dic/mecab-ipadic-neologd
In recent years, research on natural language processing has been active.
Nouns in recent years,Adverbs possible,*,*,*,*,recent years,Kinnen,Kinnen
Is a particle,Particle,*,*,*,*,Is,C,Wow
Natural language processing nouns,Proper noun,General,*,*,*,Natural language processing,Shizengen Goshori,Shizengen Goshori
Particles,Attributive,*,*,*,*,of,No,No
Research nouns,Change connection,*,*,*,*,the study,Kenkyu,Kenkyu
Is a particle,Case particles,General,*,*,*,But,Moth,Moth
Popular nouns,Adjectival noun stem,*,*,*,*,Thriving,Sakan,Sakan
Auxiliary verb,*,*,*,Special Death,Uninflected word,is,death,death
EOS
#The default IPAdic is divided into three words, "natural", "language", and "processing", but NEologd processes in one word.
#When using from Python code, MeCab.Tagger()To the argument of'-d /usr/local/lib/mecab/dic/mecab-ipadic-neologd'Is specified.

(7) Installation of TensorFlow

$ pip install tensorflow===1.14.0

(8) Installation of Keras

$ pip install Keras===2.1.6

(9) Installation of Gensim

$ pip install gensim===3.5.0

(10) Installation of Hyperopt

$ pip install hyperopt===0.1.1

(11) Installation of Matplotlib

$ pip install matplotlib===3.1.2

Check the installed libraries

Check the installed library and its version with the following command.

$ pip list

Capture: pip SSL error

When trying to install each library with the pip command, the following error message may appear.

# pip3 install hoge
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Collecting hoge
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/hoge/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/hoge/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/hoge/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/hoge/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/hoge/
  Could not fetch URL https://hoge/hoge/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='hoge', port=hoge): Max retries exceeded with url: /hoge/hoge/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)) - skipping
  Could not find a version that satisfies the requirement hoge (from versions: )
No matching distribution found for hoge

This is because openssl is not installed. In this case, if you install openssl as below and then install Python again, it will work (only for Mac, Windows is unconfirmed).

# 1.Install openssl.
$ brew install openssl

# 2.Changed the preferred SSL from LibreSSL to OpenSSL installed above.
$ echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bash_profile

# 3.Exit the terminal and restart it. After that, check the version and OK when OpneSSL is displayed.
$ openssl version
OpenSSL 1.1.1d  10 Sep 2019

# 4.Reinstall Python.
$ pyenv install 3.6.5

Recommended Posts

Building an environment for natural language processing with Python
Building an Anaconda environment for Python with pyenv
3. Natural language processing with Python 4-1. Analysis for words with KWIC
3. Natural language processing with Python 2-1. Co-occurrence network
3. Natural language processing with Python 1-1. Word N-gram
Dockerfile with the necessary libraries for natural language processing in python
3. Natural language processing with Python 2-2. Co-occurrence network [mecab-ipadic-NEologd]
Building an environment for executing Python scripts (for mac)
[Python] I played with natural language processing ~ transformers ~
Various commands for building an environment with Apache
Image Processing with Python Environment Setup for Windows
Try building an environment for MayaPython with VisualStudioCode
100 Language Processing with Python Knock 2015
Set up a development environment for natural language processing
Notes from installing Homebrew to building an Anaconda environment for Python with pyenv
Study natural language processing with Kikagaku
Natural language processing for busy people
[Natural language processing] Preprocessing with Japanese
100 Language Processing Knock with Python (Chapter 3)
Building a virtual environment with Python 3
3. Natural language processing with Python 5-1. Concept of sentiment analysis [AFINN-111]
Let's get started with Python ~ Building an environment on Windows 10 ~
3. Natural language processing with Python 5-2. Emotion intensity analysis tool VADER
[Python] Building an environment for competitive programming with Atom (input () can be used!) [Mac]
Quick batch text formatting + preprocessing for Aozora Bunko data for natural language processing with Python
[Pyenv] Building a python environment with ubuntu 16.04
Building a Python3 environment with Amazon Linux2
Build an environment for Blender built-in Python
Building a Windows 7 environment for getting started with machine learning with Python
Note when creating an environment with python
Building a Python 3.6 environment with Windows + PowerShell
[Mac] Building a virtual environment for Python
100 Language Processing Knock with Python (Chapter 2, Part 1)
I tried natural language processing with transformers.
Natural Language Processing Case Study: Word Frequency in'Anne with an E'
Building a Python development environment for AI development
Building an environment for "Tello_Video" on Raspbian
Building an environment for "Tello_Video" on Windows
[Definitive Edition] Building an environment for learning "machine learning" using Python on Windows
[Definitive Edition] Building an environment for learning "machine learning" using Python on Mac
3. Natural language processing with Python 1-2. How to create a corpus: Aozora Bunko
Selenium + WebDriver (Chrome) + Python | Building environment for scraping
Create execution environment for each language with boot2docker
Building a python environment with virtualenv and direnv
Building an environment that uses Python in Eclipse
Building a Python environment with WLS2 + Anaconda + PyCharm
Python: Deep Learning in Natural Language Processing: Basics
Commands for creating a python3 environment with virtualenv
Introduction to Python "Re" 1 Building an execution environment
Let's enjoy natural language processing with COTOHA API
RNN_LSTM2 Natural language processing
Python environment with docker-compose
Image processing with Python
Virtual environment with Python 3.6
Python environment for projects
3. Natural language processing with Python 3-1. Important word extraction tool TF-IDF analysis [original definition]
Building an environment to comply with the Python coding standard (PEP8) with Eclipse + PyDev
3. Natural language processing with Python 3-3. A year of corona looking back at TF-IDF
I read an introductory book on natural language processing
Building a Python environment for pyenv, pyenv-virtualenv, Anaconda (Miniconda)
Write about building a Python environment for writing Qiita Qiita