[PYTHON] Click the Selenium links in order to get the elements of individual pages

Click the links on the list page one by one to get the elements of the individual pages

Last time, I used Selenium to get all the elements of the list page and page forward. This time

    1. Click the link on the list page
  1. Open individual page and get element
    1. Go back to the list and click the following link
  2. Open individual page and get element

I did.

test.py


import os, re
import time
from selenium import webdriver

DRIVER_PATH = os.path.join(os.path.dirname(__file__), 'chromedriver')
browser = webdriver.Chrome(DRIVER_PATH)
url = 'https://wwwXXXX'
browser.get(url)
time.sleep(5)
for i in range(3):
    try:
        name_list = []

        path_front = '//*[@id="main"]/ul/li['
        count = 1
        path_end = ']/div[1]/a'

        for _ in range(5):
            path = path_front + str(count) + path_end
            for l in browser.find_elements_by_xpath(path):
                l.click()
                for t in browser.find_elements_by_xpath('//*[@id="main"]/div[1]/h1'):
                    name_list.append(t.text)
                    browser.back()
                    count += 1


        for name_title in zip(name_list):
            print (name_title, "\n+++++++++++++++++++++++++++++++++++++++++++++++++++")

        link_elem = browser.find_element_by_class_name('nextpostslink')
        link_elem.click()

        time.sleep(5)
    except:
        print ('not found!')

browser.close()

The processing is not beautiful at all, but in the end, I felt like I was adding the link number of li. That said, I've almost achieved my goal, so I'm done.

Recommended Posts

Click the Selenium links in order to get the elements of individual pages
How to get the number of digits in Python
Get the size (number of elements) of UnionFind in Python
[python] Get the rank of the values in List in ascending / descending order
How to get the vertex coordinates of a feature in ArcPy
Create a function to get the contents of the database in Go
I can't get the element in Selenium!
Get a capture of the entire web page in Selenium Python VBA
[Linux] Command to get a list of commands executed in the past
I want to sort a list in the order of other lists
Set the number of elements in a NumPy one-dimensional array to a power of 2 (0 padded)
Use the Java SDK of GoogleMapsAPI to get the result of reverse GeoCoding in Japanese.
Get the caller of a function in Python
Get only the subclass elements in a list
How to get dictionary type elements of Python 2.7
How to get the files in the [Python] folder
To get the path of the currently running python.exe
Use pygogo to get the log in json.
How to get a list of files in the same directory with python
I want to get custom data attributes of html as elements using Python Selenium
[Blender] How to get the selection order of vertices, edges and faces of an object
Get information equivalent to the Network tab of Chrome developer tools with Python + Selenium
How to check in Python if one of the elements of a list is in another list
[python] Get the list of classes defined in the module
Get to know the feelings of gradient boosting trees
Get the value selected in Selenium Python VBA pull-down
Try to get the contents of Word with Golang
About the uncluttered arrangement in the import order of flake8
[Python] Outputs all combinations of elements in the list
Script to get the expiration date of the SSL certificate
Get the URL of the HTTP redirect destination in Python
[Python] How to output the list values in order
To do the equivalent of Ruby's ObjectSpace._id2ref in Python
[Selenium] How to specify the relative path of chromedriver?
manage to get rid of heavy pyls in vim-lsp
I made a tool to get the answer links of OpenAI Gym all at once
Get the value of a specific key up to the specified index in the dictionary list in Python
[OCI] Python script to get the IP address of a compute instance in Cloud Shell
Seeking a unified way to wait and get for state changes in Selenium for Python elements
I tried to create a Python script to get the value of a cell in Microsoft Excel
How to find the optimal number of clusters in k-means
Try to get the function list of Python> os package
I tried to get the location information of Odakyu Bus
Test code to check for broken links in the page
How to get rid of server custom emoji in message.content
I want to get the operation information of yahoo route
How to manipulate the DOM in an iframe with Selenium
Sort the string array in order of length & Japanese syllabary
I want to judge the authenticity of the elements of numpy array
How to get the last (last) value in a list in Python
How to get all the keys and values in the dictionary
How to get a list of built-in exceptions in python
Get the number of occurrences for each element in the list
Get all IP addresses of instances in the autoscaling group
Keras I want to get the output of any layer !!
How to get an overview of your data in Pandas
[Shell] How to get the remote default branch in Git
Get the index of each element of the confusion matrix in Python
How to get a list of links from a page from wikipedia
How to get a quadratic array of squares in a spiral!
To get the name of the primitive etc. generated immediately before