[PYTHON] web scraping

code

import requests

url = (Set URL)
r = requests.get(url)

Example

#print(r.text)
<!DOCTYPE html>
<html class="client-nojs" lang="ja" dir="ltr">
<head>
<meta charset="UTF-8"/>
<title>Japan- Wikipedia</title>
<script>document.documentElement.className="client-js";RLCONF={"wgBreakFrames":!1,"wgSeparatorTransformTable":["",""],"wgDigitTransformTable":["",""],"wgDefaultDateFormat":"ja","wgMonthNames":["","January","February","March","April","May","June","July",
"August","September","October","November","December"],"wgRequestId":"f8d48c18-541d-4c28-b21d-14bc30c6cdc4","wgCSPNonce":!1,"wgCanonicalNamespace":"","wgCanonicalSpecialPageName":!1,"wgNamespaceNumber":0,"wgPageName":"Japan","wgTitle":"Japan","wgCurRevisionId":
79582646,"wgRevisionId":79582646,"wgArticleId":1864744,"wgIsArticle":!0,"wgIsRedirect":!1,"wgAction":"view","wgUserName":null,"wgUserGroups":["*"],"wgCategories":["Page with reference error","Webarchive template archiveis link","Webarch
ive template wayback link","Articles with broken external links/January 2010-April","Articles with broken external links/February 2020",
"Articles with broken external links/March 2018","Articles with source template call error","Pages using ISBN magic links","Page with 3 columns specified in Div col","Articles with descriptions that require a source/May 2016","Items with a section that is about to be written
","Articles with muddy words(When)/2011","Page with 3 columns specified in Reflist","Page with graph","Japan","Island country","Existing sovereign country","G8 member countries"],"wgPageContentLanguage":"ja","wgPageContentModel":"wikitext","wgRelevantPageName":"Japan","wgRe
levantArticleId":1864744,"wgIsProbablyEditable":!1,"wgRelevantPageIsProbablyEditable":!1,"wgRestrictionEdit":["autoconfirmed"],"wgRestrictionMove":["autoconfirmed"],"wgMediaViewerOnClick":!0,"wgMediaViewerEnabledByDefault":!0,"wgPopupsRef
erencePreviews":!1,"wgPopupsConflictsWithNavPopupGadget":!1,"wgVisualEditor":{"pageLanguageCode":"ja",
"pageLanguageDir":"ltr","pageVariantFallbacks":"ja"},"wgMFDisplayWikibaseDescriptions":{"search":!0,"nearby":!0,"watchlist":!0,"tagline":!0},"wgWMESchemaEditAttemptStepOversample":!1,"wgULSCurrentAutonym":"Japanese","wgNoticeProject":"wikip
edia","wgCentralAuthMobileDomain":!1,"wgEditSubmitButtonLabelPublish":!0,"wgULSPosition":"interlanguage","wgWikibaseItemId":"Q17"};RLSTATE={"ext.globalCssJs.user.styles":"ready","site.styles":"ready","noscript":"ready","user.styles":"read
y","ext.globalCssJs.user":"ready","user":"ready","user.options":"loading","ext.cite.styles":"ready","mediawiki.page.gallery.styles":"ready","ext.tmh.thumbnail.styles":"ready","ext.graph.styles":"ready","skins.vector.styles.legacy":"ready"
,"jquery.tablesorter.styles":"ready","mediawiki.toc.styles":"ready","ext.visualEditor.desktopArticleTarget.noscript":"ready","ext.uls.interlanguage":"ready","ext.wikimediaBadges":"ready","wikibase.client.init":"ready"};RLPAGEMODULES=["ext
.cite.ux-enhancements",
...

Recommended Posts

web scraping
web scraping (prototype)
Scraping 1
Web scraping with python + JupyterLab
Web scraping notes in python3
Trade-offs in web scraping & crawling
Easy web scraping with Scrapy
Image collection by web scraping
Web scraping using AWS lambda
Algorithm-based web scraping library Scrapely
One-liner web scraping by tse
Various scraping
Start scraping
Web scraping with Python ① (Scraping prior knowledge)
Web scraping with BeautifulSoup4 (layered page)
Scraping Alexa's web rank with pyQuery
Web scraping with Python First step
I tried web scraping with python.
[Scraping] Python scraping
Web scraping for weather warning notifications.
Scraping sample
Beginners use Python for web scraping (4) ―― 1
10 questions to check before web scraping
WEB scraping with Python (for personal notes)
[Beginner] Python web scraping using Google Colaboratory
Getting Started with Python Web Scraping Practice
[Personal note] Web page scraping with python3
Web scraping with Python ② (Actually scraping stock sites)
Horse Racing Site Web Scraping with Python
Getting Started with Python Web Scraping Practice
Practice web scraping with Python and Selenium
Easy web scraping with Python and Ruby
Web scraping with BeautifulSoup4 (serial number page)
[For beginners] Try web scraping with Python
Python scraping notes
Scraping with selenium
I tried web scraping to analyze the lyrics.
Python Scraping get_ranker_categories
AWS-Perform web scraping regularly with Lambda + Python + Cron
Try web scraping now and get lottery 6 data
Scraping with selenium ~ 2 ~
Scraping with Python
About Twitter scraping
I tried web scraping using python and selenium
Scraping with Python
I tried scraping
Scraping with Selenium
Pharmaceutical company researchers summarized web scraping using Python
Scraping 100 Fortnite images
Python: Scraping Part 1
Get boat race match information by web scraping
WEB camera capture
Let's do web scraping with Python (weather forecast)
Let's do web scraping with Python (stock price)
Scraping using Python
Python: Scraping Part 2
Beginners can use Python for web scraping (1) Improved version
Data analysis for improving POG 1 ~ Web scraping with Python ~
Quick web scraping with Python (while supporting JavaScript loading)
Python beginners get stuck with their first web scraping
Beginners use Python for web scraping (4) --2 Scraping on Cloud Shell