[PYTHON] I tried benchmarking a web application framework

I will try it somehow. The benchmark tool and the web application are running on the same host, and only the results are posted, so it may not be quite good, but for reference only ....

environment

OS: Mac OS X Yosemite(10.10.1) CPU: Intel Core i7 (3GHz) Memory: 16GB (8GB DIMM(DDR3/1600MHz) * 2) Storage: SSD

Execution / measurement method

Use apache bench.

$ ab -n 10000 -c 100 -rk "http://127.0.0.1:8888/"

The application used for the measurement only returns the string "Benchmarking ...". Click here for the source code. https://github.com/TakesxiSximada/benchmarks

result

rails + unicorn (workers=8) 【Ruby】

version

ruby: 2.1.6 rails: 4.2.1

result

First time

Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   12.624 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      4100000 bytes
HTML transferred:       150000 bytes
Requests per second:    792.13 [#/sec](mean)
Time per request:       126.242 [ms](mean)
Time per request:       1.262 [ms](mean, across all concurrent requests)
Transfer rate:          317.16 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   5.4      0     540
Processing:     7  125  58.5    113     649
Waiting:        7  125  58.2    113     648
Total:         10  126  58.7    113     649

Percentage of the requests served within a certain time (ms)
  50%    113
  66%    119
  75%    124
  80%    127
  90%    140
  95%    156
  98%    328
  99%    541
 100%    649 (longest request)

Second time

Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   14.515 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      4100000 bytes
HTML transferred:       150000 bytes
Requests per second:    688.95 [#/sec](mean)
Time per request:       145.149 [ms](mean)
Time per request:       1.451 [ms](mean, across all concurrent requests)
Transfer rate:          275.85 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0      21
Processing:     8  144  38.2    132     373
Waiting:        8  144  38.1    132     372
Total:         10  144  38.1    132     373

Third time

Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   12.725 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      4100000 bytes
HTML transferred:       150000 bytes
Requests per second:    785.83 [#/sec](mean)
Time per request:       127.253 [ms](mean)
Time per request:       1.273 [ms](mean, across all concurrent requests)
Transfer rate:          314.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       5
Processing:     7  126  33.4    120     514
Waiting:        7  126  33.1    120     514
Total:         11  127  33.3    120     514

revel 【Go】

version

Go: 1.4 revel: 0.12.0

result

First time

Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        2030 bytes

Concurrency Level:      100
Time taken for tests:   5.768 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      23010000 bytes
HTML transferred:       20300000 bytes
Requests per second:    1733.65 [#/sec](mean)
Time per request:       57.682 [ms](mean)
Time per request:       0.577 [ms](mean, across all concurrent requests)
Transfer rate:          3895.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     4   58 222.0     33    2275
Waiting:        4   58 222.0     33    2275
Total:          4   58 222.2     33    2276

Second time

Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        2030 bytes

Concurrency Level:      100
Time taken for tests:   4.821 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      23010000 bytes
HTML transferred:       20300000 bytes
Requests per second:    2074.45 [#/sec](mean)
Time per request:       48.205 [ms](mean)
Time per request:       0.482 [ms](mean, across all concurrent requests)
Transfer rate:          4661.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     6   48 147.7     32    1544
Waiting:        6   48 147.7     32    1544
Total:          6   48 147.9     32    1545

Third time

Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        2030 bytes

Concurrency Level:      100
Time taken for tests:   5.442 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      23010000 bytes
HTML transferred:       20300000 bytes
Requests per second:    1837.60 [#/sec](mean)
Time per request:       54.419 [ms](mean)
Time per request:       0.544 [ms](mean, across all concurrent requests)
Transfer rate:          4129.22 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.8      0      13
Processing:     4   54 195.3     32    2013
Waiting:        4   54 195.3     32    2013
Total:          4   54 196.0     32    2020

Tornado 【Python】

version

Python: 3.4.2 Tornado: 4.1.dev1

result

First time

Server Software:        TornadoServer/4.1.dev1
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   5.214 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      2330000 bytes
HTML transferred:       110000 bytes
Requests per second:    1917.78 [#/sec](mean)
Time per request:       52.144 [ms](mean)
Time per request:       0.521 [ms](mean, across all concurrent requests)
Transfer rate:          436.37 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.1      0      23
Processing:     1   37  23.6     33     315
Waiting:        1   37  23.6     33     315
Total:          1   37  23.7     33     315

Second time

Server Software:        TornadoServer/4.1.dev1
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   4.846 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      2330000 bytes
HTML transferred:       110000 bytes
Requests per second:    2063.60 [#/sec](mean)
Time per request:       48.459 [ms](mean)
Time per request:       0.485 [ms](mean, across all concurrent requests)
Transfer rate:          469.55 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     1   48   5.6     47     166
Waiting:        1   48   5.6     47     166
Total:          4   48   5.6     47     169

Third time

Server Software:        TornadoServer/4.1.dev1
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   5.551 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      2330000 bytes
HTML transferred:       110000 bytes
Requests per second:    1801.52 [#/sec](mean)
Time per request:       55.509 [ms](mean)
Time per request:       0.555 [ms](mean, across all concurrent requests)
Transfer rate:          409.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     2   55  19.6     48     217
Waiting:        2   55  19.6     48     217
Total:          4   55  19.6     48     217

Pyramid + Gunicorn (workers=8) 【Python】

version

Python: 3.4.2 Pyramid: 1.5.7 Gunicorn: 19.3.0

result

First time

Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   9.700 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1750000 bytes
HTML transferred:       150000 bytes
Requests per second:    1030.92 [#/sec](mean)
Time per request:       97.000 [ms](mean)
Time per request:       0.970 [ms](mean, across all concurrent requests)
Transfer rate:          176.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       6
Processing:    32   96  13.8     92     183
Waiting:       32   95  13.5     92     182
Total:         35   97  13.7     92     183

Second time

Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   24.306 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1750000 bytes
HTML transferred:       150000 bytes
Requests per second:    411.42 [#/sec](mean)
Time per request:       243.063 [ms](mean)
Time per request:       2.431 [ms](mean, across all concurrent requests)
Transfer rate:          70.31 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  134 1334.6      0   13421
Processing:     2  108  51.1     96     403
Waiting:        2  107  50.5     95     403
Total:          2  243 1329.2     97   13503

Third time

Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   9.781 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1750000 bytes
HTML transferred:       150000 bytes
Requests per second:    1022.40 [#/sec](mean)
Time per request:       97.809 [ms](mean)
Time per request:       0.978 [ms](mean, across all concurrent requests)
Transfer rate:          174.73 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.8      0      35
Processing:     5   96  21.3     91     254
Waiting:        5   95  21.1     90     254
Total:          8   96  21.1     91     255

Django + Gunicorn (workers=8) 【Python】

version

Python: 3.4.2 Django: 1.8.2 Gunicorn: 19.3.0

result

First time

Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   14.071 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1840000 bytes
HTML transferred:       150000 bytes
Requests per second:    710.69 [#/sec](mean)
Time per request:       140.708 [ms](mean)
Time per request:       1.407 [ms](mean, across all concurrent requests)
Transfer rate:          127.70 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       4
Processing:    58  140  16.5    138     221
Waiting:       58  139  16.4    136     219
Total:         62  140  16.5    138     221

Second time

Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   17.237 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1840000 bytes
HTML transferred:       150000 bytes
Requests per second:    580.14 [#/sec](mean)
Time per request:       172.373 [ms](mean)
Time per request:       1.724 [ms](mean, across all concurrent requests)
Transfer rate:          104.24 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   38 376.3      0    3890
Processing:     2  134  30.2    134     305
Waiting:        1  132  30.2    133     304
Total:          2  172 370.8    134    3905

Third time

Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   24.894 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1840000 bytes
HTML transferred:       150000 bytes
Requests per second:    401.70 [#/sec](mean)
Time per request:       248.942 [ms](mean)
Time per request:       2.489 [ms](mean, across all concurrent requests)
Transfer rate:          72.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   42 416.2      0    4449
Processing:     1  206  82.0    201     683
Waiting:        1  204  81.4    199     675
Total:          2  248 416.8    204    4565

Bottle + Gunicorn (workers=8) 【Python】

version

Python: 3.4.2 Bottle: 0.12.8 Gunicorn: 19.3.0

result

First time

Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   5.422 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1800000 bytes
HTML transferred:       150000 bytes
Requests per second:    1844.34 [#/sec](mean)
Time per request:       54.220 [ms](mean)
Time per request:       0.542 [ms](mean, across all concurrent requests)
Transfer rate:          324.20 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0      11
Processing:     2   54  46.9     51    1089
Waiting:        1   53  46.9     51    1089
Total:          4   54  47.1     51    1091

Second time

Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   5.615 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1800000 bytes
HTML transferred:       150000 bytes
Requests per second:    1780.82 [#/sec](mean)
Time per request:       56.154 [ms](mean)
Time per request:       0.562 [ms](mean, across all concurrent requests)
Transfer rate:          313.03 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     2   56  52.5     52    1246
Waiting:        1   55  52.5     52    1246
Total:          4   56  52.6     52    1248

Third time

Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   6.429 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1800000 bytes
HTML transferred:       150000 bytes
Requests per second:    1555.50 [#/sec](mean)
Time per request:       64.288 [ms](mean)
Time per request:       0.643 [ms](mean, across all concurrent requests)
Transfer rate:          273.43 [Kbytes/sec] received

CherryPy 【Python】

CherryPy cannot be measured because an error will occur if -c is 100. For the time being, it was carried out with -c 20. However, some requests are still in error.

version

Python: 3.4.2 CherryPy: 3.7.0

result

First time

Server Software:        CherryPy/3.7.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      20
Time taken for tests:   19.241 seconds
Complete requests:      10000
Failed requests:        28
   (Connect: 3, Receive: 11, Length: 14, Exceptions: 0)
Keep-Alive requests:    9984
Total transferred:      1777152 bytes
HTML transferred:       149760 bytes
Requests per second:    519.72 [#/sec](mean)
Time per request:       38.483 [ms](mean)
Time per request:       1.924 [ms](mean, across all concurrent requests)
Transfer rate:          90.20 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   4.6      0     125
Processing:     0   20 399.2      8   19115
Waiting:        0   14 330.9      7   19114
Total:          1   20 400.4      8   19239

Second time

Server Software:        CherryPy/3.7.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      20
Time taken for tests:   19.076 seconds
Complete requests:      10000
Failed requests:        19
   (Connect: 1, Receive: 5, Length: 13, Exceptions: 0)
Keep-Alive requests:    9986
Total transferred:      1777671 bytes
HTML transferred:       149790 bytes
Requests per second:    524.23 [#/sec](mean)
Time per request:       38.151 [ms](mean)
Time per request:       1.908 [ms](mean, across all concurrent requests)
Transfer rate:          91.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   2.5      0     126
Processing:     0   18 299.8      8   10006
Waiting:        0    8   2.7      7      39
Total:          1   18 299.8      8   10006

Third time

Server Software:        CherryPy/3.7.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      20
Time taken for tests:   18.978 seconds
Complete requests:      10000
Failed requests:        24
   (Connect: 0, Receive: 7, Length: 17, Exceptions: 0)
Keep-Alive requests:    9983
Total transferred:      1777137 bytes
HTML transferred:       149745 bytes
Requests per second:    526.91 [#/sec](mean)
Time per request:       37.957 [ms](mean)
Time per request:       1.898 [ms](mean, across all concurrent requests)
Transfer rate:          91.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   3.9      0     129
Processing:     0   19 316.0      8   10007
Waiting:        0    8   2.6      7      26
Total:          1   19 316.0      8   10007

Extra edition

Nginx [laughs]

No, it's really fast ... or rather, it's natural.

version

Nginx: 1.6.2

result

First time

Server Software:        nginx/1.6.2
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        16 bytes

Concurrency Level:      20
Time taken for tests:   0.384 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    9911
Total transferred:      2509555 bytes
HTML transferred:       160000 bytes
Requests per second:    26072.36 [#/sec](mean)
Time per request:       0.767 [ms](mean)
Time per request:       0.038 [ms](mean, across all concurrent requests)
Transfer rate:          6389.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     0    1   0.6      1       5
Waiting:        0    1   0.6      1       5
Total:          0    1   0.6      1       5

Second time

Server Software:        nginx/1.6.2
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        16 bytes

Concurrency Level:      20
Time taken for tests:   0.346 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    9910
Total transferred:      2509550 bytes
HTML transferred:       160000 bytes
Requests per second:    28939.79 [#/sec](mean)
Time per request:       0.691 [ms](mean)
Time per request:       0.035 [ms](mean, across all concurrent requests)
Transfer rate:          7092.37 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     0    1   0.5      1       6
Waiting:        0    1   0.5      1       6
Total:          0    1   0.5      1       6

Third time

Server Software:        nginx/1.6.2
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        16 bytes

Concurrency Level:      20
Time taken for tests:   0.346 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    9909
Total transferred:      2509545 bytes
HTML transferred:       160000 bytes
Requests per second:    28860.28 [#/sec](mean)
Time per request:       0.693 [ms](mean)
Time per request:       0.035 [ms](mean, across all concurrent requests)
Transfer rate:          7072.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     0    1   0.5      1       4
Waiting:        0    1   0.5      1       4
Total:          0    1   0.5      1       4

Recommended Posts

I tried benchmarking a web application framework
I tried to make a Web API
I made a WEB application with Django
I tried web scraping with python.
Build a web application with Django
Creating a web application using Flask ①
Creating a web application using Flask ③
Creating a web application using Flask ④
I tried collecting one-liner web servers
I tried "Streamlit" which turns the Python code into a web application as it is
I tried to make a 2channel post notification application with Python
I tried to make a todo application using bottle with python
I made a scaffolding tool for the Python web framework Bottle
I tried to create a linebot (implementation)
I tried to create a linebot (preparation)
Try using the web application framework Flask
Display matplotlib diagrams in a web application
I tried drawing a line using turtle
I tried a functional language with Python
[Python] A quick web application with Bottle!
Run a Python web application with Docker
Let's make a web framework with Python! (1)
Let's make a web framework with Python! (2)
I tried 3D detection of a car
I tried the Python Tornado Testing Framework
I made a web application that converts photos into Van Gogh's style
I made a web application in Python that converts Markdown to HTML
I tried to discriminate a 6-digit number with a number discrimination application made with python
[Streamlit] I hate JavaScript, so I make a web application only with Python
Start a web server using Bottle and Flask (I also tried using Apache)
I want to make a web application using React and Python flask
Implement a simple application with Python full scratch without using a web framework.
I tried to build a super-resolution method / ESPCN
I tried web scraping to analyze the lyrics.
What I was addicted to when creating a web application in a windows environment
I tried web scraping using python and selenium
Go beginner tried to create a cloud native web application using Datastore / GAE
I made a GUI application with Python + PyQt5
I tried to build a super-resolution method / SRCNN ①
I tried scraping
I tried PyQ
I tried playing a typing game in Python
I tried AutoKeras
I tried to extend Python's Active Record framework
Steps to develop a web application in Python
I made a LINE Bot with Serverless Framework!
I tried using Pythonect, a dataflow programming language.
I tried reading a CSV file using Python
I want to create a web application that uses League of Legends data ①
I tried papermill
I tried to generate a random character string
I tried adding a Python3 module in C
I tried to build a super-resolution method / SRCNN ③
I tried to build a super-resolution method / SRCNN ②
I tried running alembic, a Python migration tool
I tried django-slack
I tried spleeter
I tried using a database (sqlite3) with kivy
I tried cgo
I tried to make a ○ ✕ game using TensorFlow
I installed Chainer, a framework for deep learning