[PYTHON] Successful scraping with Selenium

Scraping with Selenium

I wrote it in order to achieve the requirement that all 55 pages and 24 product images per page must be acquired. Selenium is really convenient. If you know the product name, product URL, and product image SRC on a text basis, it's OK for the time being, so it's very concise.

seleniumer.py


import os, re
import time
from selenium import webdriver

DRIVER_PATH = os.path.join(os.path.dirname(__file__), 'chromedriver')
browser = webdriver.Chrome(DRIVER_PATH)
url = 'https://www.XXXXXXXX'
browser.get(url)

for i in range(56):
    try:
        img_list = []
        urls_list = []
        name_list = []

        for i in browser.find_elements_by_xpath('//*[@id="find-results"]/div/div/div/a[1]/img[1]'):
            imgs = i.get_attribute('src')
            img_list.append(imgs)

        for a in browser.find_elements_by_xpath('//*[@id="find-results"]/div/div/div/a[1]'):
            urls = a.get_attribute('href')
            urls_list.append(urls)

        for t in browser.find_elements_by_xpath('//*[@id="find-results"]/div/div/div/a/div/span[1]/span'):
            name = t.text
            name_list.append(name)

        for img_src, urls_href, name_title in zip(img_list, urls_list, name_list):
            print (name_title, urls_href, img_src, "\n+++++++++++++++++++++++++++++++++++++++++++++++++++")

        link_elem = browser.find_element_by_class_name('control-page-next-button')
        link_elem.click()

        time.sleep(3)
    except:
        print ('not found!')

browser.close()

If you want to find out the xpath used in browser.find_elements_by_xpath, you can paste it from COPY in Chrome. I was impressed with the extremely useful functions.

It seems that the find_elements () method has to be turned around in for.

Recommended Posts

Successful scraping with Selenium
Scraping with selenium
Scraping with selenium ~ 2 ~
Scraping with Selenium
Scraping with Selenium [Python]
Scraping with selenium in Python
Scraping with Selenium + Python Part 1
Scraping with Selenium in Python
Scraping with Selenium + Python Part 2
I-town page scraping with selenium
Scraping with Selenium in Python (Basic)
Scraping with Python, Selenium and Chromedriver
Scraping with Python
Scraping with Python
Beginning with Selenium
Practice web scraping with Python and Selenium
Scraping with Python (preparation)
Try scraping with Python.
Scraping with Python + PhantomJS
Serverless scraping using selenium with [AWS Lambda] -Part 1-
Scraping with scrapy shell
ScreenShot with Selenium (Python)
Python web scraping selenium
Scraping with Python + PyQuery
Scraping with Beautiful Soup
Scraping RSS with Python
I was addicted to scraping with Selenium (+ Python) in 2020
I tried scraping with Python
Automatically download images with scraping
Web scraping with python + JupyterLab
Scraping with chromedriver in python
Festive scraping with Python, scrapy
Save images with web scraping
Python: Working with Firefox with selenium
Easy web scraping with Scrapy
Scraping with Tor in Python
Web scraping using Selenium (Python)
Scraping weather forecast with python
Memories of fighting with Selenium
scraping the Nikkei 225 with playwright-python
Try Selenium Grid with Docker
[Python + Selenium] Tips for scraping
I tried scraping with python
Web scraping beginner with python
Successful update_with_media with twitter API
Table scraping with Beautiful Soup
[Python, Selenium, PhantomJS] A story when scraping a website with lazy load
Try scraping with Python + Beautiful Soup
Scraping multiple pages with Beautiful Soup
Scraping with Node, Ruby and Python
Web scraping with Python ① (Scraping prior knowledge)
Web scraping with BeautifulSoup4 (layered page)
Scraping Alexa's web rank with pyQuery
Summary of scraping relations (selenium, pyautogui)
Web scraping with Python First step
I tried web scraping with python.
Scraping with Python and Beautiful Soup
selenium
Scraping pages with pagination with Beautiful Soup
Scraping with Beautiful Soup in 10 minutes
Make testing with Selenium more accessible