[PYTHON] How to set browser location in Headless Chrome

When scraping, some sites may ask for your browser's (geo) location. This article is a personal note about the solution when using Selenium + Headless Chrome + Python.

environment

EC2 (Amazon Linux 2 AMI)

reference

Setting sensors (location) in headless Chrome

tool

・ Selenium ・ Chrome driver (85.0.4183.87) ・ Python3 (3.6.2)

Why you need location information

An error occurs when I try to scrape a certain site using Headless Chrome. When I checked the screenshot, the following pop-up was displayed and I could not access the specified page.

__ * "Processing is not possible because the location information could not be obtained. Please allow the location information." * __

As a result of various investigations, it seems that the cause was that the location information of the browser could not be confirmed.

How to set location information in Headless Chrome

So I searched for a way to set location information in Headless Chrome, but the information is surprisingly small. .. ..

In the first place, I was wondering if there was a way to set location information in the Chrome browser, so when I looked it up, it seems that it can be set from the developer tools.

So, when I thought, "Isn't it possible to solve the problem if the developer tools can be operated even with headless chrome?", Some people asked the same question on Stack overflow, and a solution was presented. Apparently it uses a method called execute_cdp_cmd ().

execute_cdp_cmd() It seems that you can set Google Developper tools with the method of selenium.webdriver.

The code implementation example is below

location_setting.py

from selenium import webdriver

def start_chrome_driver():
	options = webdriver.ChromeOptions()
	options.add_argument("--headless")
	options.add_argument("--disable-gpu")
	# executable_Please change the path according to each environment
	driver = webdriver.Chrome(executable_path="/usr/local/bin/chromedriver", chrome_options=options)
	return driver

#Launch Headless Chrome
driver = start_chrome_driver()

#Permission to set location information (location information cannot be set without permission)
driver.execute_cdp_cmd(
    "Browser.grantPermissions",
    {
        "origin": "https://hromssp.obc.jp/",
        "permissions": ["geolocation"]
    },
)

#Latitude, longitude, latitude / longitude error(Unit: m)To set
driver.execute_cdp_cmd(
    "Emulation.setGeolocationOverride",
    {
        "latitude": 35.689487,
        "longitude": 139.691706,
        "accuracy": 100,
    },
)

• By the way, the latitude and longitude set above are the locations of the Tokyo Metropolitan Government Building.

result

Now you can set location information (impersonation?) In Headless Chrome, and you can access the page you want to scrape!