python web scraping-get elements in bulk
Web scraping with Python-Introduction- [First step to improve work efficiency] 9-10 Study memo https://www.udemy.com/course/python-scraping-beginner/ Page to be scraped (with PW restrictions) https://scraping-for-beginner.herokuapp.com/mypage
Requirement: Get all the text information on the right side of the table below
If you press F12 and check with the development tools, you can see that all the text on the right is marked up with td.
1. Get the td element from the browser.
elems_td = browser.find_elements_by_tag_name('td')
2. Take elements one by one from td and put them in value
Put the information of elems_td that got all the elements of td into elem_td.
values = []
#elems_Take out the elements one by one from td and elem_Put in td
for elem_td in elems_td:
3. Put the element (text format) of elem_td in value.
#elem_Element of td(text format)In value
value = elem_td.text
4. Add all value elements to the list of values
#Add all value elements to the list of values
values.append(value)
All code
#Get td element from browser
elems_td = browser.find_elements_by_tag_name('td')
values = []
#elems_Take out the elements one by one from td and elem_Put in td
for elem_td in elems_td:
#elem_Element of td(text format)In value
value = elem_td.text
#Add all value elements to the list of values
values.append(value)
#Hit values
values
result
['Kohei Imanishi', 'Kikagaku Co., Ltd.', 'July 15, 1994', 'Chiba', 'basketball\n reading\n Gadget collection']
Recommended Posts