[Personal note] Web page scraping with python3
Precautions when scraping
Right-click on the page source code to see the page source instead
Use the one displayed by the developer tool
Extract text
<dt>price<span class="tax">(tax included)</span></dt>
To extract the text of the span tag embedded in the dt tag like
source = '<dt>price<span class="tax">(tax included)</span></dt>'
soup = BeautifulSoup(source, "html.parser")
soup.text
And .text can be extracted by specifying
Remove whitespace
<dt>
price
<span class="tax">(tax included)</span>
</dt>
When there is a white space in the tag such as
def remove_whitespace(str):
return ''.join(str.split())
source = '<dt>price<span class="tax">(tax included)</span></dt>'
soup = BeautifulSoup(source, "html.parser")
remove_whitespace(soup.text)
Can be taken out
Since the space in the center cannot be deleted with strip (), the space character is used as the delimiter with split ().
Join with .join
Find at Beautiful Soup
If you want to find a particular class
In one case
soup.find(class_='hoge')
When searching all
soup.find_all(class_='hoge')
If you want to find a specific id
In one case
soup.find(id='hoge')
When searching all
soup.find_all(id='hoge')
If you want to find a specific tag
In one case
soup.find('hoge')
When searching all
soup.find_all('hoge')
They can also have multiple conditions at the same time
soup.find('hoge',class_='fuga)
Recommended Posts