Here, I will explain about web scraping using Python.
BeautifulSoup
Suppose you want to crawl and scrape a web page that displays the following HTML file.
<ul class="list-group">
<li class="list-group-item"><a href="">Element 1</a></li>
<li class="list-group-item"><a href="">Element 2</a></li>
<li class="list-group-item"><a href="">Element 3</a></li>
</ul>
The Python script looks like this:
import requests
from bs4 import BeautifulSoup
url =URL to get HTML
response = requests.get(url)
response.encoding = response.apparent_encoding
bs = BeautifulSoup(response.text, 'html.parser')
ul = bs.select('ul.list-group')
for li in ul[0].select('li.list-group-item'):
a_tags = li.select('a')
a_tag = a_tags[0]
item_name = a_tag.text.strip()
Scrapy
Suppose you want to crawl and scrape an HTML file similar to the one using Beautiful Soup above.
<ul class="list-group">
<li class="list-group-item"><a href="">Element 1</a></li>
<li class="list-group-item"><a href="">Element 2</a></li>
<li class="list-group-item"><a href="">Element 3</a></li>
</ul>
import scrapy
class SampleSpider(scrapy.Spider):
name = 'sample'
allowd_domains = [domain]
start_urls = [
Target URL
]
def parse_list(self, response):
ul = response.css('ul.list-group')[0]
for li in ul.css('li.list-group-item'):
item_url = li.css('a::attr(href)').extract_first()
yield scrapy.Request(item_url, callback=parse_detail)
def parse_detail(self, response):
item_name = response.css('h1.item-name::text').extract_first()
return item_name
Here, I explained how to scrape the web using Beautiful Soup and Scrapy.
What is the programming language Python? Can it be used for AI and machine learning?
Recommended Posts