Installation der erforderlichen Pakete
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"
$ sudo aptitude install phantomjs xvfb
$ pip install selenium pyvirtualdisplay
from selenium import webdriver
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
# <Display cmd_param=['Xvfb', '-br', '-nolisten', 'tcp', '-screen', ' - snip -
driver = webdriver.PhantomJS()
driver.get("http://www.example.com)
type(driver.page_source)
# <class 'str'>
driver.page_source
# '<!DOCTYPE html><html itemscope="" itemtype="http://schema.org/Web - snip -
from bs4 import BeautifulSoup
soup = BeautifulSoup(driver.page_sourve)
i = [ {"href": x["href"], "text": x.string, "class": x._class } for x in soup.find_all("a") ]
print(i)
# [{'class': None, 'text': 'MENU', 'href': 'javascript:;'}, {'class': None, 'text': 'oberste Seite', 'href': '/'}, {'class': None, 'text': 'Plattform', 'href': '/pf/'}, - snip -
Selbst jetzt (September 2016) gibt es die folgenden Probleme. Wenn Sie also Phantomjs unter Ubuntu 16.04 verwenden möchten, sollten Sie es wie gewohnt anstelle des Pakets installieren. https://bugs.launchpad.net/ubuntu/+source/phantomjs/+bug/1578444
Recommended Posts