[PYTHON] Tips for using Selenium and Headless Chrome in a CUI environment

Use Selenium and Headless Chrome in a CUI environment

environment

how to use

As a prerequisite, Chrome is already installed.

Specify options to use Headless Chrome

example.py


options = webdriver.ChromeOptions()
options.binary_location = '/usr/bin/google-chrome'
options.add_argument('--headless')
options.add_argument('--disable-gpu')
options.add_argument('--no-sandbox')
options.add_argument('--window-size=1200x600')

Start Service to start faster than normal start

example.py


#Service startup
service = Service(executable_path='/usr/local/bin/chromedriver')
service.start()
#Connect to Chrome
driver = webdriver.Remote(service.service_url, desired_capabilities=options.to_capabilities())

All you have to do now is run Slenium as usual.

If you get an error when using Selenium methods

The following error may occur in the CUI environment

error.log


selenium.common.exceptions.WebDriverException: Message: unknown error: an X display is required for keycode conversions, consider using Xvfb
  (Session info: headless chrome=59.0.3071.86)
  (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.8.0-49-generic x86_64)

It happened when I ran the sendKeys () method. This can be avoided by entering characters etc. with JavaScript instead of sendKeys (). If you get other errors, you can probably avoid them by running JavaScript.

Example:

example.py


# driver.get_element_by_css_selector('hoge').sendKeys('fuga')Raises an exception
def _set_value_for_element(selector: str, value: str):
        return 'document.querySelector("{selector}").setAttribute("value", "{value}")'.format(selector=selector, value=value)

driver.get(url)
# sendKeys()I get an exception when I run JS
driver.execute_script(_set_value_for_element(user_name_selector, user_name))
driver.execute_script(_set_value_for_element(user_password_selector, user_password))
driver.find_element_by_css_selector(login_button_selector).click()

Recommended Posts

Tips for using Selenium and Headless Chrome in a CUI environment
Tips for using ElasticSearch in a good way
I put Selenium and headless chrome in AWS lambda. (Notes under Win10 environment, etc.)
I tried using Selenium with Headless chrome
Screenshots of Megalodon in selenium and Chrome.
I tried using Headless Chrome from Selenium
Reading and creating a mark sheet using Python OpenCV (Tips for reading well)
Selenium + WebDriver (Chrome) + Python | Building environment for scraping
Tips for using Realsense SR300 on MacBook in 2020
A complete guidebook to using pyenv, pip and python in an offline environment
Create a data collection bot in Python using Selenium
Building a Docker working environment for R and Python
Selenium-Screenshot is useful for screenshots of web pages in Python3, Selenium and Google Chrome
Building a training environment for penetration testing using Naumachia
[Selenium] Open the link in a new tab and move it [Python / Chrome Driver]
Command line collection for using virtual environment in Anaconda
Memo for building a machine learning environment using Python
Problems and solutions for using the remote debugging environment for Linux apps in Visual Studio 2017
Use tensorflow in an environment without root
Build an interactive environment for machine learning in Python
Create an environment for test automation with AirtestIDE (Tips)
Tips for using Selenium and Headless Chrome in a CUI environment
install tensorflow in anaconda + python3.5 environment
An implementation of ArcFace for TensorFlow
Python3 TensorFlow for Mac environment construction
Created an environment for Anaconda & Jupyter
Seeking a unified way to wait and get for state changes in Selenium for Python elements
Create a personality diagnostic tool in a cloud environment using Google Colaboratory and IBM Watson Personality Insights
Building a Python environment on a Mac and using Jupyter lab
A story that stumbled when using pip in a proxy environment
Tips for coding short and easy to read in Python
Collaborate in a remote environment
[Python + Selenium] Tips for scraping
Create a Python environment for professionals in VS Code on Windows
Build a Chainer environment using CUDA and cuDNN on a p2 instance
Building a Docker working environment for R and Python 2: Japanese support
Build and test a CI environment for multiple versions of Python
Build a Selenium environment on Amazon Linux 2 in the shortest time
How about Anaconda for building a machine learning environment in Python?
Build a local development environment for Lambda + Python using Serverless Framework
Tips for implementing a slightly difficult Model or Training in Keras
Try using virtualenv, which can build a virtual environment for Python
Build and try an OpenCV & Python environment in minutes using Docker
Create a CGH for branching a laser in Python (laser and SLM required)
[DynamoDB] [Docker] Build a development environment for DynamoDB and Django with docker-compose
Build a development environment using Jupyter and Flask with Python in Docker (supports both VS Code / code-server)
Build a go environment using Docker
Workaround for Chrome headless mode error
[TouchDesigner] Tips for for statements using python
Tips for replacing and debugging functions
How to make a model for object detection using YOLO in 3 hours
A collection of tips for speeding up learning and reasoning with PyTorch
How to scrape pages that are “Access Denied” in Selenium + Headless Chrome
Try using PHATE, a dimensionality reduction and visualization method for biological data
Quickly build a python environment for deep learning and data science (Windows)
I compared using Dash and Streamlit in Docker environment using B league data
I tried using NVDashboard (for those who use GPU in jupyter environment)
A useful note when using Python for the first time in a while
Build a lightweight server in Python and listen for Scratch 2 HTTP extensions