Getting Started with Python Web Scraping Practice

Introduction to Python Web Scraping Practice Rewritten in Python 3 with reference to

Preparation

Python version check

$ python3 -V
Python 3.8.6

Nikkei Stock Average acquisition

Package installation

$ pip3 install beautifulsoup4

Scraping

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "http://www.nikkei.com/markets/kabu/"

html = urlopen(url).read()
soup = BeautifulSoup(html, "html.parser")

nikkei_heikin = soup.find("span", class_="mkc-stock_prices").string
print(nikkei_heikin)

Package installation

$ pip3 install beautifulsoup4
$ pip3 install apscheduler
$ pip3 install requests
import csv
import datetime

import requests
from apscheduler.schedulers.blocking import BlockingScheduler
from bs4 import BeautifulSoup

sched = BlockingScheduler()

#Run every hour
# @sched.scheduled_job('interval', hours=1)


#Run at 0 minutes every hour
@sched.scheduled_job("cron", minute=0, hour="*/1")
def scheduled_job():

    #Access the Nikkei Stock Average page of the Nihon Keizai Shimbun and get the HTML
    r = requests.get("http://www.nikkei.com/markets/kabu/")
    r.raise_for_status()

    #Get the Nikkei Stock Average using Beautiful Soup
    soup = BeautifulSoup(r.text, "html.parser")
    nikkei_heikin = soup.select_one(
        "#CONTENTS_MARROW > div.mk-top_stock_average.cmn-clearfix > div.cmn-clearfix > div.mkc-guidepost > div.mkc-prices > span.mkc-stock_prices"
    ).get_text(strip=True)

    #Convert the current time to a string
    now = datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S")

    print(f"{now} {nikkei_heikin}"

    #Add the date and time and the Nikkei Stock Average to CSV
    with open("nikkei_heikin.csv", "a") as fw:
        writer = csv.writer(fw, dialect="excel", lineterminator="\n")
        writer.writerow([now, nikkei_heikin])


sched.start()

Recommended Posts

Getting Started with Python Web Scraping Practice
Getting Started with Python Web Scraping Practice
Getting Started with Python Web Applications
1.1 Getting Started with Python
Getting Started with Python
Getting Started with Python
Practice web scraping with Python and Selenium
Web scraping with python + JupyterLab
Getting Started with Python Django (1)
Getting Started with Python Django (4)
Getting Started with Python Django (3)
Getting Started with Python Django (6)
Web scraping beginner with python
Python3 | Getting Started with numpy
Getting Started with Python Django (5)
Getting Started with Python responder v2
Web scraping with Python ① (Scraping prior knowledge)
Web scraping with Python First step
I tried web scraping with python.
Getting Started with Python for PHPer-Classes
Getting Started with Python Basics of Python
Getting Started with Python Genetic Algorithms
Getting started with Python 3.8 on Windows
Getting Started with Python for PHPer-Functions
Scraping with Python
Scraping with Python
Getting Started with python3 # 1 Learn Basic Knowledge
WEB scraping with Python (for personal notes)
Getting Started with Flask with Azure Web Apps
[Personal note] Web page scraping with python3
Web scraping with Python ② (Actually scraping stock sites)
Horse Racing Site Web Scraping with Python
Getting Started with Python for PHPer-Super Basics
Getting started with Dynamo from Python boto
Easy web scraping with Python and Ruby
[For beginners] Try web scraping with Python
[FastAPI] Getting started with FastAPI, an ASGI web framework made by Python
Django 1.11 started with Python3.6
Scraping with Python (preparation)
Try scraping with Python.
Getting started with Android!
Scraping with Python + PhantomJS
Getting started with apache2
Getting Started with Golang 1
Getting Started with Django 1
Getting Started with Optimization
Getting Started with Golang 3
Getting Started with Numpy
Getting started with Spark
Scraping with Selenium [Python]
Python web scraping selenium
Scraping with Python + PyQuery
Getting Started with Pydantic
Getting Started with Golang 4
Getting Started with Jython
Scraping RSS with Python
Getting Started with Django 2
Getting started with Python with 100 knocks on language processing
AWS-Perform web scraping regularly with Lambda + Python + Cron
[Translation] Getting Started with Rust for Python Programmers
Introduction to Tornado (1): Python web framework started with Tornado