Install Scrapy on python3

When I was reading "Web scraping with Python", I decided to use scrapy for some, but I could not install it well, so I will briefly summarize the solution. (By the way, the above books are mainly scraped using Beautiful Soup)

Set environment

MacBook Air (13-inch, Mid 2011) Processor: 1.8 GHz Intel Core i7 Memory: 4 GB 1333 MHz DDR3 Version: 10.11.5 Python: 3.6.2

Try putting Scrapy

1, This time I will put it in the environment of Python3.

$ pip3 install scrapy

Then ... I got the following error. Apparently, pip should be upgraded.

You are using pip version 8.1.1, however version 8.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

2, Follow the instructions in the error statement to upgrade pip.

$ pip3 install --upgrade pip

Again there is an error ... This time it seems to be around xcode.

1 error generated.


Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?

Perhaps try: xcode-select --install


error: command '/usr/bin/clang' failed with exit status 1

3, Next, just follow the instructions, so type the command.

$ xcode-select --install

xcode-select: note: install requested for command line developer tools

Then a dialog will be displayed, so agree and download the developer tools.

スクリーンショット 2016-07-02 23.17.27.png

4, After downloading, execute the following again to complete the installation.

$ sudo pip3 install scrapy


Installing collected packages: lxml, parsel, scrapy
  Running install for lxml ... done
Successfully installed lxml-3.6.0 parsel-1.0.2 scrapy-1.1.0

Recommended Posts

Install Scrapy on python3
Install python on WSL
Install Python on Pidora.
Install Python on Mac
Install Python 3 on Mac
Install Python3.4 on CentOS 6.6
Install python on windows
Install Python 2.7.3 on CentOS 5.4
Install Python 3.3 on Ubuntu 12.04
Install Python 3.4 on Mac
Install Python 3.6 on Docker
Install Scrapy on Raspbian (Jessie)
Install Python 3.8 on RHEL 8 (AppStream)
Install watchdog on Windows + Python 3.3
install python
Install pygame on python3.4 on mac
Install Python 3.8 on CentOS 7 (SCL)
Install pandas 0.14 on python3.4 [on Mac]
Install OpenCV on Ubuntu + python
Python --Install MySQLDB on EC2
Install Python 3.8 on CentOS 8 (AppStream)
[Python] [Chainer] [Windows] Install Chainer on Windows
Install Python Pillow on Amazon Linux
Install Python 3.8 on Ubuntu 18.04 (OS standard)
Install Python3 on Sakura server (FreeBSD)
Install Python 3.8 on Ubuntu 20.04 (OS standard)
Install python2.7 on windows 32bit environment
Install xgboost (python version) on Windows
Install Python on Windows + pip + virtualenv
Install scrapy in python anaconda environment
Install Pytorch on Blender 2.90 python on Windows
Install Python 3.7 Anaconda on MAC, but Python 2
Install python3 on Mac (El Capitan)
Install Python 3.9 on Ubuntu 20.04 (OS standard?)
Install Python development environment on Windows 10
Install Python on CentOS using Pyenv
Install confluent-kafka for Python on Ubuntu
Install Python 2.7 on Ubuntu 20.04 (OS standard?)
Steps to install python3 on mac
Install Python on CentOS using pyenv
Python on Windows
twitter on python3
Install Python locally
Install NAOqi python
python on mac
Install Python Control
Python on Windbg
CentOS8 --Install --Python3
[Kivy] How to install Kivy on Windows [Python]
Install python library on Lambda using [/ tmp]
Steps to install Python environment on Ubuntu
scipy stumbles with pip install on python 2.7.8
Install Python less than 3.5.3 on macOS Catalina
Install python on xserver to use pip
Install Python3 on mixhost server [Currently NG]
Install pyenv and Python 3.6.8 on Ubuntu 18.04 LTS
Install Python 3.8, Pip 3.8 on EC2 (Amazon Linux 2)
Install mecab on Marvericks
Install Tensorflow on Mac
Install Faiss on CentOS 7
Install pip on Mavericks