Scraping is done using Selenium in Python.
MacOSX10.11.1
Xvfb and more needed Reference: CentOS6.4 + Selenium environment creation
$ sudo pip install selenium
You need to install the library separately.
$ sudo pip install pyvirtualdisplay
http://stackoverflow.com/questions/26070834/how-to-fix-selenium-webdriverexception-the-browser-appears-to-have-exited-befor
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
#When using pyvirtualdisplay
from pyvirtualdisplay import Display
display = Display(visible=0, size=(1024, 768))
display.start()
driver = webdriver.Firefox()
Later, when running the script on Linux (CentOS) FireFox is more convenient (easier), so I use FireFox.
url = "http://google.com"
driver.get(url)
driver.close()
#When using pyvirtualdisplay
display.stop()
Recommended Posts