[PYTHON] Cheat sheet when scraping with Google Colaboratory (Colab)

table of contents

-[How to use Beautiful Soup](How to use # beautiful-soup) -[How to use Selenium](How to use #selenium) -[How to use Pandas](How to use #pandas) -[How to handle spreadsheets](#How to handle spreadsheets) -Regular expression look-ahead, after-Yomi is described in another article.

How to use Beautiful Soup

How to eliminate garbled characters

When using requests, you would normally write it as follows,

from bs4 import BeautifulSoup
import requests

res = requests.get(url)
soup = BeautifulSoup(res.content, 'html.parser')

There are sites that are garbled with this, so if you do the following, the garbled characters can be eliminated considerably.

from bs4 import BeautifulSoup
import requests

res = requests.get(url)
soup = BeautifulSoup(res.content, "lxml", from_encoding='utf-8')

find code list

Select code list

List of selector specification methods

Change the attribute element according to the element you want to search. ʻId, class, href, name, summary, etc. Insert >if you want to get only child elements (one level down), and putspace` if you want to get offspring elements (all down one level).

How to use Selenium

Preparations for using Selenium

When using with Colab, Selenium download and UI specifications are not possible, so That setting is required.

#Download the libraries needed to use Selenium
!apt-get update
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
!pip install selenium

from selenium import webdriver

#Settings for using the driver without a UI
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',options=options)
driver.implicitly_wait(10)

When using Selenium and Beautiful Soup

As a use case, when the element cannot be acquired by just Beautiful Soup If you want to load the page with seleniumu and then extract the necessary information with Beautiful Soup.

driver.get(url)
html = driver.page_source.encode('utf-8')
soup = BeautifulSoup(html, 'html.parser')

Selenium basic code

When you want to select a dropdown

from selenium.webdriver.support.ui import Select

element = driver.find_element_by_xpath("xpath")
Select(element).select_by_index(indexnum) #Select by index
Select(element).select_by_value("value") #value of value
Select(element).select_by_visible_text("text") #Display text

List of Xpath specification methods

Refer to here for how to get other nodes

When changing tabs

Used when a new tab is created without page transition when clicked

handle_array = driver.window_handles
driver.switch_to.window(handle_array[1])

Wait until a specific element is displayed

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

#Wait until all elements on the page are loaded (timeout judgment in 15 seconds)
WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located)

#Wait until the element on the page with the specified ID is loaded (timeout judgment in 15 seconds)
WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.ID, 'ID name')))

#CLASS name Wait until the element on the specified page is loaded (timeout judgment in 15 seconds)
WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CLASS_NAME, 'CLASS name')))

#Wait until the element on the page specified by the CLASS name in Xpath is loaded (timeout judgment in 15 seconds)
WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, 'xpath')))

What to do when you can't click

target = driver.find_element_by_xpath('xpath')
driver.execute_script("arguments[0].click();", target)

How to use Pandas

How to create a data frame and add data

import pandas as pd
columns = ['Item 1', 'Item 2', 'Item 3', 'Item 4', 'Item 5']
df = pd.DataFrame(columns=columns)

#Data acquisition process

se = pd.Series([data1, data2, data3, data4, data5], columns)
df = df.append(se, columns)

When downloading Pandas data

from google.colab import files

filename = 'filename.csv'
df.to_csv(filename, encoding = 'utf-8-sig') 
files.download(filename)

When saving Pandas data to My Drive

from google.colab import drive

filename = filename.csv'
path = '/content/drive/My Drive/' + filename

with open(path, 'w', encoding = 'utf-8-sig') as f:
  df.to_csv(f)

How to work with spreadsheets

Preparations for working with spreadsheets

#Download the library needed to work with spreadsheets
!pip install gspread

from google.colab import auth
from oauth2client.client import GoogleCredentials
import gspread

#Authentication process
auth.authenticate_user()
gc = gspread.authorize(GoogleCredentials.get_application_default())

Frequently used code

ss_id = 'Spreadsheet ID'
sht_name = 'Sheet name'

workbook = gc.open_by_key(ss_id)
worksheet = workbook.worksheet(sht_name)

#When acquiring data
worksheet.acell('B1').value
worksheet.cell(2, 1).value

#When updating
worksheet.update_cell(row, column, 'Update contents')

gspread code list

Workbook operation

Seat operation

Cell manipulation

[Reference site] BeautifulSoup4 cheat sheet (selector, etc.) Python3 memo --Beautiful Soup4 stuff Basics of CSS selectors for web scraping Summary of frequently used operation methods of Selenium webdriver What is XPath? Learn the basic knowledge of XPath! Indispensable for web scraping! Summary of Xpath Summary of how to use the gspread library! Work with spreadsheets in Python