[PYTHON] Scraping with selenium ~ 2 ~

Continuing from the last time, this time I would like to use Selenium to perform scraping.

Execution environment

Referenced articles

I tried to summarize the cssSelector required for Selenium locator Get specific elements in Python Selenium Basics of Python and Selenium [Python] find_elements_by_css_selector ・ ・ ・ Get elements from css selector [Python3] Scraping via browser (dynamic pages, etc.) [Selenium] Organize HTML tags / elements / attributes / CSS properties Three settings to make for stable operation of Selenium (also supports Headless mode)

How to get CSS selector

When using selenium, I use find_element_by_css_selector ("selector") to get the desired css selector from the source code, but first I have to read the HTML hierarchy myself. I was wondering.

After reading various articles, I was wondering why these people were able to get the css selector properly, but it seems that they can copy the source code.

① Right-click on the web page where you want to automatically click on that location
(2) Press the verification button to verify, and check that the location of the source code you want is marked.
③ Right-click on the source code and press copy. There is a copy selector, so press it.
④ If you paste it in the editor, the desired css selector will be displayed.

It is possible to get the css selector in the flow like.

Difference between elements

ʻElements contains a list, but ʻelements contains a single number

How to use clear

clear can clear the value entered in advance in the search box. Therefore, if you want to search by entering something in the search box, you can delete the pre-filled values and search only what you want to search.

id and class

ʻId is ** only one can be used per web page **, but class` can be used many times per page. Therefore, it is easier to access the information you want to get by searching by id.


I managed to create a process that automatically likes on Instagram and automatically follows. When I made it, the mechanism was surprisingly interesting, so I enjoyed making it.

Recommended Posts

Scraping with selenium
Scraping with selenium ~ 2 ~
Scraping with Selenium
Successful scraping with Selenium
Scraping with Selenium [Python]
Scraping with selenium in Python
Scraping with Selenium + Python Part 1
Scraping with Selenium in Python
Scraping with Selenium + Python Part 2
I-town page scraping with selenium
Scraping with Python
Scraping with Python
Scraping with Selenium in Python (Basic)
Scraping with Python, Selenium and Chromedriver
Beginning with Selenium
Practice web scraping with Python and Selenium
Scraping with Python (preparation)
Try scraping with Python.
Scraping with Python + PhantomJS
Scraping with scrapy shell
ScreenShot with Selenium (Python)
Python web scraping selenium
Scraping with Python + PyQuery
Scraping with Beautiful Soup
Scraping RSS with Python
Serverless scraping using selenium with [AWS Lambda] -Part 1-
I tried scraping with Python
Automatically download images with scraping
Web scraping with python + JupyterLab
Scraping with chromedriver in python
Festive scraping with Python, scrapy
Save images with web scraping
Python: Working with Firefox with selenium
I was addicted to scraping with Selenium (+ Python) in 2020
Easy web scraping with Scrapy
Scraping with Tor in Python
Web scraping using Selenium (Python)
Scraping weather forecast with python
Memories of fighting with Selenium
scraping the Nikkei 225 with playwright-python
Try Selenium Grid with Docker
[Python + Selenium] Tips for scraping
I tried scraping with python
Web scraping beginner with python
Scraping 1
Table scraping with Beautiful Soup
Try scraping with Python + Beautiful Soup
Scraping multiple pages with Beautiful Soup
Scraping with Node, Ruby and Python
Web scraping with Python ① (Scraping prior knowledge)
Web scraping with BeautifulSoup4 (layered page)
Scraping Alexa's web rank with pyQuery
Summary of scraping relations (selenium, pyautogui)
Web scraping with Python First step
I tried web scraping with python.
Scraping with Python and Beautiful Soup
Scraping pages with pagination with Beautiful Soup
Scraping with Beautiful Soup in 10 minutes
Make testing with Selenium more accessible
Let's do image scraping with Python