[Introduction to Python3 Day 20] Chapter 9 Unraveling the Web (9.1-9.4)

9.1 Web client

--The web is a client server system. Client → Server ** Request **. From server to client, open a TCP / IP connection, send URL and other information via HTTP, and receive ** response **. --The response has status, response data and format. --The most well-known web client is a web browser. You can send HTTP requests in a variety of ways.

--Each HTTP connection is independent and independent of other connections. (Stateless) → Cookies are used as a way to manage state instead of HTTP. --The server puts client-specific information in a cookie and sends it to the client. → When the client sends the cookie back to the server, the server uniquely identifies the client from the cookie.

9.1.1 Python standard web library

Web clients and server modules weren't very well organized in Python 2. → In Python3, it is summarized in a package.

--http manages client server HTTP details. --client does client-side processing. --server helps develop web servers with Python. --Cookies and cookie jars manage cookies that store necessary information across multiple site visits.

--urllib runs on http. --request processes the client's request. --response processes the server response. --parse separates the URL into parts.


>>> import urllib.request as ur
>>> url="https://raw.githubusercontent.com/koki0702/introducing-python/master/dummy_api/fortune_cookie_random1.txt"
>>> conn=ur.urlopen(url)
>>> print(conn)
<http.client.HTTPResponse object at 0x10733c8d0>
>>> data=conn.read()
>>> print(data)
b'You will be surprised by a loud noise.\\r\\n\\n[codehappy] http://iheartquotes.com/fortune/show/20447\n'
#Display of HTTP status code.
#A 200 indicates that all the code worked.
>>> print(conn.status)
200

#Data format display
#Content in HTTP response header-Specified by Type.
>>> print(conn.getheader("Content-Type"))
text/plain; charset=utf-8
>>> for key,value in conn.getheaders():
...     print(key,value)
... 
Content-Type text/plain; charset=utf-8
Content-Security-Policy default-src 'none'; style-src 'unsafe-inline'; sandbox
Strict-Transport-Security max-age=31536000
X-Content-Type-Options nosniff
X-Frame-Options deny
X-XSS-Protection 1; mode=block
ETag W/"9f5651c47de1d25d4c531d22c98b96ea61d10d7e4f5c6eb6cbeecd9e01cdfbf8"
Cache-Control max-age=300
X-Geo-Block-List 
Via 1.1 varnish-v4
X-GitHub-Request-Id D326:4317:329188:36E387:5E2EBEE7
Content-Length 99
Accept-Ranges bytes
Date Mon, 27 Jan 2020 10:43:52 GMT
Via 1.1 varnish
Connection close
X-Served-By cache-tyo19932-TYO
X-Cache MISS
X-Cache-Hits 0
X-Timer S1580121832.776951,VS0,VE269
Vary Authorization,Accept-Encoding, Accept-Encoding
Access-Control-Allow-Origin *
X-Fastly-Request-ID 034e4e0799a3de2ed0ae382d63bcd716a0574002
Expires Mon, 27 Jan 2020 10:48:52 GMT
Source-Age 0
>>> 

9.1.2 Beyond the standard library

--It is more convenient to use the requests of the third party module.


>>> import requests
>>> url="https://raw.githubusercontent.com/koki0702/introducing-python/master/dummy_api/fortune_cookie_random1.txt"
>>> resp=requests.get(url)
>>> resp
<Response [200]>
>>> print(resp.text)
I know that there are people who do not love their fellow man, and I people like that!
    -- Tom Lehrer, Satirist and Professor
    [codehappy] http://iheartquotes.com/fortune/show/21465


9.2 Web server

9.2.1 The simplest web server with Python


$ python -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
127.0.0.1 - - [27/Jan/2020 20:09:04] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Jan/2020 20:09:04] code 404, message File not found
127.0.0.1 - - [27/Jan/2020 20:09:04] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [27/Jan/2020 20:12:55] "GET /wheel HTTP/1.1" 200 -
q
C
^C      

python -m http.server 9999
Serving HTTP on 0.0.0.0 port 9999 (http://0.0.0.0:9999/) ...
^C


9.2 Web server

--A web framework does more than a simple web server because it provides the functionality to create a website. The main functions are shown below. --Routing: Parses the URL and calls the server function. --Template: Stream server-side data into an HTML page.

python



#Other values can be specified for the port number.
python -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
#127.0.0.1 :Client address
#The first- :Remote user name
#Second- :Login user name
#"GET / HTTP/1.1":Commands sent to the web server
#GET is an HTTP method,/Is the requested resource (root directory), HTTP/1.1 is the HTTP version.
#200:HTTP status code
127.0.0.1 - - [27/Jan/2020 20:09:04] "GET / HTTP/1.1" 200 -

9.2.2 WSGI A leap forward since the definition of WSGI, a universal API between Pytho web applications and web servers.

9.2.3 Framework

The web framework provides some or all of the following features:

--Routing

9.2.4 Bottle

--Bottle is made entirely of Python files, so it's very easy to try and deploy.

bottle installation



pip install bottle
Collecting bottle
  Downloading bottle-0.12.18-py3-none-any.whl (89 kB)
     |████████████████████████████████| 89 kB 692 kB/s 
Installing collected packages: bottle
Successfully installed bottle-0.12.18

bottle1.py



from bottle import route, run

#To associate the URL with the function immediately following it@Use the route decorator.
#in this case,/(home page)Is processed by the home function.
@route("/")
def home():
    return "It is not fancy, but it is my home page"

#run()The function runs a Python test web server built into bottle.
run(host="localhost",port=9999)

Execution statement



python3 bottle1.py 
Bottle v0.12.18 server starting up (using WSGIRefServer())...
Listening on http://localhost:9999/
Hit Ctrl-C to quit.

Execution result



#http://localhost:Access 9999.
It is not fancy, but it is my home page

bottle2.py



#Make bottle return the contents of this file when the home page is requested.
from bottle import route, run, static_file

@route("/")
#Root indicates"."Is the current directory.
def home():
    return static_file("index.html",root=".")

run(host="localhost",port=9999)

Execution statement


python bottle2.py
Bottle v0.12.18 server starting up (using WSGIRefServer())...
Listening on http://localhost:9999/
Hit Ctrl-C to quit.

Execution result



#http://localhost:9999/Access to.
My new and improved home page!!!

bottle3.py



from bottle import route, run, static_file

@route("/")
def home():
    return static_file("index.html",root=".")

#Pass arguments to the function via the URL.
#echo()The argument thing of is in the URL/echo/Substitute the string argument after.
@route("/echo/<thing>")
def echo(thing):
    return "Say hello to my little friend: %s" % thing

# debug=True
# reloader=True
run(host="localhost",port=9999)

Execution statement



python bottle3.py
Bottle v0.12.18 server starting up (using WSGIRefServer())...
Listening on http://localhost:9999/
Hit Ctrl-C to quit.

127.0.0.1 - - [27/Jan/2020 20:59:21] "GET /echo/Mothra HTTP/1.1" 200 37

Execution result



#http://localhost:9999/echo/Access Mothra.
Say hello to my little friend: Mothra

--If debug = True is specified, a debug page will be created when an HTTP error is returned. --If reloader = True, the page reloads in the browser when you make changes to your Python code.

9.2.5 Flask

--It's smarter than Bottle.

Flask installation


pip install flask
...
Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 Werkzeug-0.16.1 click-7.0 flask-1.1.1 itsdangerous-1.1.0

--The default home directory for Flask static files is static, and URLs for files in that directory also start with / static. Therefore, the home directory is set to "." So that the URL / is mapped to the index.html file. --Set debug = True in the run () function to enable automatic reloading. bottle uses separate arguments for debugging and reloading. Also, when an exception occurs, Flask returns a specially formatted page with detailed information about where the problem occurred.

flask1.py



from flask import Flask

app=Flask(__name__, static_folder=".",static_url_path="")

@app.route("/")
def home():
    return app.send_static_file("index.html")

@app.route("/echo/<thing>")
def echo(thing):
    return thing

# reloader=True
app.run(port=9999, debug=True)

Execution result


#http://127.0.0.1:9999/Access to.
My new and improved home page!!!


#http://127.0.0.1:9999/echo/Enter Godzilla in your browser.
Godzilla

--Flask has a template called jinja2.

flask2.html



<html lang="en" dir="ltr">
  <head>
    <meta charset="utf-8">
    <title>Flask2 Example</title>
  </head>
  <body>
    Say hello to my little friend:{{ thing }}
  </body>
</html>>

flask2.py



from flask import Flask, render_template

app=Flask(__name__)

#thing=thing read from URL<thing>Is assigned to a variable called thing, and this is"flask2.html"Passed to.
@app.route("/echo/<thing>")
def echo(thing):
    return render_template("flask2.html",thing=thing)


app.run(port=9999,debug=True)

Execution result


#http://127.0.0.1:9999/echo/Enter Gamera in your browser

< Say hello to my little friend:Gamera >

9.2.5.1 Passing arguments in the form of part of the URL path

flask3.html


<html>
<head>
<title>Flask3 Example</title>
</head>
<body>
Say hello to my little friend: {{ thing }}.
Alas, in just destroyed {{ place }}!
</body>
</html>

flask3a.py



from flask import Flask, render_template

app=Flask(__name__)

@app.route("/echo/<thing>/<place>")
def echo(thing, place):
    return render_template("flask3.html",thing=thing, place=place)


app.run(port=9999, debug=True)

Execution result



#http://127.0.0.1:9999/echo/Rodan/Visit McKeesport.

Say hello to my little friend: Rodan. Alas, in just destroyed McKeesport!

flask3b.py



from flask import Flask, render_template, request

app=Flask(__name__)

#Pass it as a GET argument.
@app.route("/echo/")
def echo():
    thing=request.args.get("thing")
    place=request.args.get("place")
    return render_template("flask3.html",thing=thing, place=place)

app.run(port=9999, debug=True)

Execution result



#http://127.0.0.1:9999/echo/?thing=Gorgo&place=Visit Wilmerding.

Say hello to my little friend: Gorgo. Alas, in just destroyed Wilmerding!

flask3c.py



from flask import Flask, render_template, request

app=Flask(__name__)

@app.route("/echo/")
#Of the dictionary**Use to convert the key into a dictionary and falask3.Can be passed to html.
def echo():
    kwargs={}
    kwargs["thing"]=request.args.get("thing")
    kwargs["place"]=request.args.get("place")
    return render_template("flask3.html",**kwargs)

app.run(port=9999, debug=True)

9.2.6 Web servers other than Python

--The following two web servers are used in the production system. --Apache with mod_wsgi module --nginx with uWSGI application server

9.2.7 Other frameworks

There are many.

9.3 Web Services and Automation

The web is used as a powerful way to connect applications and data in a variety of formats other than HTML.

9.3.1 Web API and REST

--Web API can provide web page data --RESTful services use HTTP verbs in a fixed way. --HEAD: Get information about resources. --GET: Get resource data from the server. --POST: Update server data. --PUT: This verb creates a new resource. --DELETE: Delete

9.3.2 JSON

--Suitable for exchanging data between web clients and servers.

9.3.3 Crawling and scraping

For some study. "Scrapy Tutorial" https://doc-ja-scrapy.readthedocs.io/ja/latest/intro/tutorial.html

9.3.5 HTML scraping with Beautiful Soup

--Beautiful Soup is useful if you have already retrieved the HTML and data from your website.

9.4 Review assignment

Create a skeleton-only website with a 9-1 Flask debug and cyclable development web server. The server should be started using the host name localhost and port 5000.

flask9-1.py



from flask import Flask

app=Flask(__name__)

app.run(port=5000, debug=True)

9-2 Let's add a home () function that processes requests for the home page. Set it up to return the string It is alive.

flask9-2.py



from flask import Flask

app=Flask(__name__)

@app.route("/")
def home():
    return "It is alive"

app.run(port=5000, debug=True)

9-3 Let's create a jinja2 template file named home.html with the following contents.

home.html



<html>
<head>
  <title>It is alive!</title>
<body>
  I am of course referring to {{thing}},which is {{height}} feet tall and {{color}}.
</body>
</html>

9-4 Let's rewrite the home () function of the server to use the home.html template.

Pass three GET arguments, thing, height, and color, to the home () function.

flask9-4.py



from flask import Flask, render_template, request

app=Flask(__name__)

@app.route("/")
def home():
    thing=request.args.get("thing")
    height=request.args.get("height")
    color=request.args.get("color")
    return render_template("home.html",thing=thing, height=height, color=color)

app.run(port=5000, debug=True)

Impressions

I reviewed the Web system quickly. Even if you use this area, it seems to be a long way off.

References

"Introduction to Python3 by Bill Lubanovic (published by O'Reilly Japan)"

Recommended Posts

[Introduction to Python3 Day 20] Chapter 9 Unraveling the Web (9.1-9.4)
[Introduction to Python3 Day 13] Chapter 7 Strings (7.1-7.1.1.1)
[Introduction to Python3 Day 14] Chapter 7 Strings (7.1.1.1 to 7.1.1.4)
[Introduction to Python3 Day 15] Chapter 7 Strings (7.1.2-7.1.2.2)
[Introduction to Python3 Day 21] Chapter 10 System (10.1 to 10.5)
[Introduction to Python3, Day 17] Chapter 8 Data Destinations (8.1-8.2.5)
[Introduction to Python3, Day 17] Chapter 8 Data Destinations (8.3-8.3.6.1)
[Introduction to Python3 Day 19] Chapter 8 Data Destinations (8.4-8.5)
[Introduction to Python3 Day 18] Chapter 8 Data Destinations (8.3.6.2 to 8.3.6.3)
[Introduction to Python3 Day 12] Chapter 6 Objects and Classes (6.3-6.15)
[Introduction to Python3 Day 22] Chapter 11 Concurrency and Networking (11.1 to 11.3)
[Introduction to Python3 Day 11] Chapter 6 Objects and Classes (6.1-6.2)
[Introduction to Python3 Day 23] Chapter 12 Become a Paisonista (12.1 to 12.6)
[Introduction to Python3 Day 8] Chapter 4 Py Skin: Code Structure (4.1-4.13)
[Introduction to Python3 Day 1] Programming and Python
[Introduction to Python3 Day 3] Chapter 2 Py components: Numbers, strings, variables (2.2-2.3.6)
[Introduction to Python3 Day 2] Chapter 2 Py Components: Numbers, Strings, Variables (2.1)
[Introduction to Python3 Day 4] Chapter 2 Py Components: Numbers, Strings, Variables (2.3.7-2.4)
Introduction to Effectiveness Verification Chapter 1 in Python
[Introduction to Python3 Day 7] Chapter 3 Py Tools: Lists, Tuples, Dictionaries, Sets (3.3-3.8)
[Introduction to Python3 Day 5] Chapter 3 Py Tools: Lists, Tuples, Dictionaries, Sets (3.1-3.2.6)
[Introduction to Python3 Day 10] Chapter 5 Py's Cosmetic Box: Modules, Packages, Programs (5.4-5.7)
[Introduction to Python3 Day 9] Chapter 5 Py's Cosmetic Box: Modules, Packages, Programs (5.1-5.4)
[Introduction to Python3 Day 6] Chapter 3 Py tool lists, tuples, dictionaries, sets (3.2.7-3.2.19)
Introduction to Python Let's prepare the development environment
Introduction to OpenCV (python)-(2)
Introduction to Python with Atom (on the way)
[Introduction to Algorithm] Find the shortest path [Python3]
Introduction to Effectiveness Verification Chapter 2 Written in Python
Day 66 [Introduction to Kaggle] The easiest Titanic forecast
[Introduction to Python] How to iterate with the range function?
[Chapter 5] Introduction to Python with 100 knocks of language processing
[Chapter 3] Introduction to Python with 100 knocks of language processing
[Chapter 2] Introduction to Python with 100 knocks of language processing
[Introduction to Udemy Python3 + Application] 27. How to use the dictionary
[Introduction to Udemy Python3 + Application] 30. How to use the set
[Introduction to Python] How to stop the loop using break?
[Introduction to Python] Basic usage of the library matplotlib
[Technical book] Introduction to data analysis using Python -1 Chapter Introduction-
[Chapter 4] Introduction to Python with 100 knocks of language processing
Introduction to Python Django (2) Win
Introduction to serial communication [Python]
[Introduction to Python] <list> [edit: 2020/02/22]
Introduction to Python (Python version APG4b)
An introduction to Python Programming
Introduction to Python For, While
[Python] Introduction to scraping | Program to open web pages (selenium webdriver)
[Part.2] Crawling with Python! Click the web page to move!
Save images on the web to Drive with Python (Colab)
[Introduction to Python] How to get data with the listdir function
[Introduction to Udemy Python 3 + Application] 58. Lambda
[Introduction to Udemy Python 3 + Application] 31. Comments
Leave the troublesome processing to Python
Introduction to Python Numerical Library NumPy
Practice! !! Introduction to Python (Type Hints)
[Introduction to Python] <numpy ndarray> [edit: 2020/02/22]
[Introduction to Udemy Python 3 + Application] 57. Decorator
Introduction to Python Hands On Part 1
[Introduction to Python] How to parse JSON
[Introduction to Udemy Python 3 + Application] 56. Closure
In the python command python points to python3.8