[Python / Selenium] XPath
Here's some useful knowledge to keep in mind when scraping with Selenium.
XPath You will need it to get the Element. The image below is easy to understand.
Required for crawler creation! XPATH notation summary --Qiita
XPath Helper There is an extension that is very useful for identifying XPATHs, so I'll list it here. XPath Helper
You can get the XPath by holding down the Shift key and moving the cursor.
Chrome Extension! Installing and Using XPath Helper \ | WATLAB -Python, Signal Processing, AI-
Omit
XPath can also be omitted as shown below.
driver.find_elements(By.XPATH,'//div[@class="products"]')
Webdriver method
If you look at the article below, I think there is no problem. Selenium webdriver Summary of frequently used operation methods --Qiita
The methods I often use are summarized below.
driver.get
#Specify the URL to use
driver.get('url')
driver.back
#Return to the previous page
driver.back()
driver.quit It is possible to close the window.
driver.quit()
driver.execute_script I also use this quite a lot.
driver.execute_script('javascript:smarty.product.pager(2);')
Working with Selenium in Python and executing JavaScript embedded in html --Qiita
driver.find_elements As introduced above, it can be used as follows.
driver.find_elements(By.XPATH,'//div[@class="p-products"]')
Recommended Posts