[PYTHON] selenium trick (proxy / headless)

background

A memo of the knowledge gained when working with selenium

↓ Introduction method ■ Installing Selenium WebDriver-Trying it out

By the way, with python it is possible to start with explosive speed like this

$ pip install selenium

#Or
$ conda install selenium

base

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()

Boot options

Can be started in headless mode by adding --headless

options.add_argument('--headless')
driver = webdriver.Chrome(options=options)

Works outside the sandbox by adding --no-sandbox

options.add_argument('--no-sandbox')
driver = webdriver.Chrome(options=options)

Proxy settings

option.add_argument('--proxy-server=http://%s' % PROXY)
option.add_argument('--proxy-auth=%s' % PROXY_AUTH)
driver = webdriver.Chrome(options=options)

Set to open in a new window instead of opening in a new tab of an existing window

options.add_argument('--new-window')
driver = webdriver.Chrome(options=options)

Execute js fired with onclick attribute

There seems to be a time when the button system is automated with selenium. If you want to execute js, you can execute it directly as shown below without clicking a button.

browser.execute_script('javascript:hogehoge();')

Switch iframe

iframe = driver.find_element_by_id('wpnt-notes-iframe2')
#Switch to the acquired inline frame
driver.switch_to.frame(iframe)

Difference between quit and exit

「driver.close()」 Only the active tab is closed.

「driver.quit()」 Close all tabs and quit the browser.

docker-selenium

If you want to put selenium in a clean state, it requires a little troublesome procedure. I wondered if it could be used quickly like docker, and it existed.

I saw it using docker-selenium

It seems to use VNC to run chrome inside it and run it there. It seems to be very convenient to share files easily if you have knowledge of docker.

By preparing a container for VNC and a container for browser testing, you can quickly prepare a test environment that is separate from the development environment like this.

version: '3.7'

services:
  chrome-server:
    build:
      context: ./chrome
      dockerfile: Dockerfile
    container_name: 'chrome-server'
    hostname: 'chrome-server'
    ports:
      - 9000:5900
      - 4444:4444
    environment:
      - 'TZ=Asia/Tokyo'
      - 'HUB_PORT_4444_TCP_ADDR=hub'
      - 'HUB_PORT_4444_TCP_PORT=4444'
    volumes:
      - ./chrome/downloads:/root/Downloads
  py-server:
    build:
      context: ./py-server
      dockerfile: Dockerfile
     container_name: 'py-server'
     hostname: 'py-server'
     environment:
       - 'TZ=Asia/Tokyo'
     tty: true
     depends_on:
       - chrome-server
     volumes:
       - ./py-server/src:/app/src
       - ./py-server/images:/app/images
     env_file:
       - app.env

It is very convenient that the browser test environment is completed with docker.

reference

■ chrome boot options ■ [Python] Manipulate iframe contents with Selenium ■ Let's master Chrome DevTools! Basics of debugging with browser development tools essential for web development