[PYTHON] I tried to describe the traffic in real time with WebSocket

This is the article on the 11th day of NetOpsCoding AdventCalender.

This time, I will implement a traffic graph that is almost always confirmed when performing network operations, using a technology called * WebSocket *. The demo and source code are pasted at the end.

** This implementation method is implemented by referring to the tool used by @yuyarin for monitoring in Cedec2015. ** **

1.First of all

Cacti, Zabbix, MRTG, etc. are famous as tools for monitoring traffic. In most cases, the SNMP acquisition interval is about 5 minutes. Basically, I am satisfied with the confirmation, but in actual work such as opening a new line / adjusting traffic, I often want to see the traffic at * the moment *. Depending on the device, real-time traffic can be monitored based on the CLI, but with the CLI, it is difficult to notice changes in the status of multiple locations, and is there a delay in anomaly detection?

In order to solve such a problem, I will create a graph that is * visually easy to understand and can capture short-term traffic changes in real time.

  1. To draw in real time on the web = There are two main methods for dynamically drawing a graph on a Web screen.

WebSocket is a relatively new technology. You can freely decide the timing of data transmission / reception and the method of communication. Once you have posted a session, you can easily transfer data in both directions.

Graph drawing can be realized with either, but anyway, I will try using WebSocket, which seems to be applicable to other tool implementations! !! Anyway, let's use it immediately!

  1. Execution environment =

To display on the Web, a platform such as nginx or apache is required. This demo server uses nginx. In addition, 8080 is used for WS communication this time.

The WebSocket library uses tornado. There are Gevent-WebSocket, ws4py, etc., but tornado came to me the most.

Also, although HighCharts is used as a graph drawing tool, there is no problem with other graph tools that can be added dynamically. Also, HightCharts is charged except for personal use, so those who are concerned should use other tools. ccchart is recommended because it has abundant samples.

  1. Implementation on the WebSocket-Server side =

Let's implement the server side. The server-side implementation of WebSocket is not difficult if only the point of "asynchronous processing" is suppressed. When it listens at a specific URI and the client is connected, it becomes possible to freely exchange data with each other, so it becomes possible to send the result of acquiring SNMP. SNMP is acquired by command execution by subprocess, and the result is extracted by regular expression. Also, this time, the OID that looks at the Counter of the NIC of the own server is specified.

After implementing the following code, just execute it with the python command and the WebSocket server construction is completed!

ws_traffic_server.py


#! /usr/bin/env python
# -*- coding: utf-8 -*-
import tornado.ioloop
import tornado.web
import tornado.websocket
from tornado.options import define, options ,parse_command_line
from datetime import datetime
import shlex, subprocess , time ,re , json ,threading

SLEEP_TIME = 20 #Interval to get SNMP

COMMUNITY = 'dev'
IP = '127.0.0.1'
IFMIB_OID = '1.3.6.1.2.1.31.1.1.1'

snmp_command_in = 'snmpget -v 2c -c %s %s %s.6.2'%(COMMUNITY,IP,IFMIB_OID)
snmp_command_out = 'snmpget -v 2c -c %s %s %s.10.2'%(COMMUNITY,IP,IFMIB_OID)

#Specify the port that WebSocket listens on
define("port", default = 8080,type = int)

class SendWebSocket(tornado.websocket.WebSocketHandler):

    #Event called when a connection is secured
    def open(self):
        print 'Session Opened. IP:' + self.request.remote_ip

    #Event when a disconnection event occurs, such as when the browser is closed
    def on_close(self):
        print "Session closed"

    #Event called when a message is sent from the client
    def on_message(self, message):
        if message == 'hello':
            pre_counter_in = int(self.exe_snmp(snmp_command_in))
            pre_counter_out = int(self.exe_snmp(snmp_command_out))

            #Time on WebSocket.timer cannot be used. Reproduce the delay with the following variable CallBack
            #SLEE_Start communication processing with SNMP in the latter half after TIME seconds
            tornado.ioloop.IOLoop.instance().call_later(SLEEP_TIME,self.snmp_second_half,{'in_counter':pre_counter_in,'out_counter':pre_counter_out})

    #SLEEP_Get SNMP after TIME seconds
    def snmp_second_half(self,pre_counters):
        result = {}
        pos_counter_in = int(self.exe_snmp(snmp_command_in))
        pos_counter_out = int(self.exe_snmp(snmp_command_out))

        #Traffic calculation from the counter difference from the specified number of seconds
        traffic_in = (pos_counter_in - pre_counters['in_counter'])  / SLEEP_TIME
        traffic_out = (pos_counter_out - pre_counters['out_counter'])  / SLEEP_TIME
        #Send JSON-converted traffic data to a web client
        try:
            result.update({'traffic_in' : traffic_in , 'traffic_out' : traffic_out})
            result.update({'timestamp' : time.mktime(datetime.now().timetuple())})
            self.write_message(json.dumps(result))
        except:
            print "Client is already disconnectted."

    #Returns only the value of the SNMP execution result
    #e.g. [IF-MIB::ifHighSpeed.21 = Gauge32: 1000] -> [1000]
    def exe_snmp(self,snmp_command):
        split_command = shlex.split(snmp_command)
        exec_output = subprocess.check_output(split_command)
        r = re.compile("(.*)(: )(.*)")
        snmp_result = r.match(exec_output).group(3)
        return snmp_result

    #Only accepts communication from hosts that are specified as not True
    def check_origin(self, origin):
        return True

#Waits for a connection request to WS with the specified URI
app = tornado.web.Application([
    (r"/ws/ifmon/", SendWebSocket),
])

if __name__ == "__main__":
    parse_command_line()
    app.listen(options.port)
    mainloop = tornado.ioloop.IOLoop.instance()
    mainloop.start() #Start WebSocket Server

What is easy to get hooked on here is the process of acquiring the traffic of network devices from SNMP and calculating it. Traffic calculation by SNMP (Current IFcounter-IFcounter after n seconds) / n Since it cannot be calculated without waiting for several tens of seconds, the program must also wait for n seconds. However, time.sleep () in the standard library cannot be used for asynchronous communication such as WebSocket. Since the entire process sleeps, it will affect other viewers as well.

This was solved by a function in tornado called ʻIOLoop.call_later ()` that can call back after a delay.

I feel that the program is being created by force, so I would appreciate it if you could comment if you should write it correctly.

  1. Client-side implementation = The processing on the client side is performed by Javascript. The client side needs to know four event handlers in order to handle the traffic data sent from the server.
Event Occurrence timing
onopen Event called when the connection with the server is completed
onmessage Event that occurs when data is received(Main place)
onclose Event called when the connection with the server is lost
onerror Event called when an error occurs

When the connection to the server is completed and the ʻonopenevent occurs, a signalhellois sent to the server to start SNMP. When you sendhello, the server side reacts and sends the traffic data. When the reception of data from the server is detected, the ʻonmessage event is generated, so the graph is drawn in this event. If you catch an action that disconnects WebSocket such as closing the browser, the ʻonclose` event will occur and the communication will end there.

The graph settings are long, so I will omit them. The relevant part of WebSocket processing is implemented as follows.

ws_client.js


var WS_URL = 'ws://www5445uo.sakura.ne.jp:8080/ws/ifmon/'

var init_ws_Traffic = function(){
  var traffic_chart =Traffic_Chart('traffic_chart', {}); //Highhart graph constructor
  var ws = new WebSocket(WS_URL);
  console.log('----- WebSocket Open -----')

  ws.onopen = function() {//WS connection establishment
    ws.send('hello');
  };

  ws.onerror = function(event){
    $("p#error").text('Failed join to the server');
    console.log('Connection faild....')
  };

  ws.onmessage = function(event) { //Traffic data processing
    var data = JSON.parse(event.data);
    var timestamp = data['timestamp']*1000 + 60 * 9 * 60 * 1000;
    var value = data['traffic_in']
    document.getElementById('traffic_in').innerHTML = convert_bps(value); //Display as text
    traffic_chart.series[0].addPoint([timestamp, value], true, true); //Inbound Traffic Graph Add is done here
    value = data['traffic_out']
    document.getElementById('traffic_out').innerHTML = convert_bps(value);
    traffic_chart.series[1].addPoint([timestamp, value], true, true);//Outbound Traffic Graph Add is done here
    ws.send('hello'); //Next traffic request
  };
  ws.onclose = function () {
    ws.send('close');
    ws.close();
  };
  window.onbeforeunload = function () {
    ws.send('close');
    ws.close()
  };
}

The above two are necessary for WebSocket communication. If you check what you actually made on the Web screen, it will look like this. スクリーンショット 2015-12-11 1.19.25.png It's a graph ...!

Now you can get a traffic graph every 10 seconds for work! If you are making something similar and have a better method, please let me know.

  1. Bonus (demo and source code) = I don't think it's good without the feeling of being updated in real time, so I uploaded what I made to the demo server. It may be for a limited time, but please take a look at what it looks like. If you wait for about 20 seconds, something will change.

** There is no live demo because the demo server has disappeared **

Also, the code used in the demo is listed below. https://github.com/Mabuchin/wstraffic

  1. Finally = We have introduced how to implement a traffic graph for use during work! I think that it will be more convenient if you modify it according to your purpose, such as making it possible to depict the traffic of multiple devices in one graph with a little rewriting. If you have a chance to use it, please try it!

Recommended Posts

I tried to describe the traffic in real time with WebSocket
I tried to illustrate the time and time in C language
I tried to process the image in "sketch style" with OpenCV
I tried to process the image in "pencil style" with OpenCV
How to write offline real time I tried to solve the problem of F02 with Python
I tried to save the data with discord
I tried to integrate with Keras in TFv1.1
How to write offline real time I tried to solve E11 with python
I tried to classify guitar chords in real time using machine learning
How to write offline real time I tried to solve E12 with python
I tried to learn the sin function with chainer
I tried to graph the packages installed in Python
I tried to touch the CSV file with Python
I tried to solve the soma cube with python
I tried to predict the horses that will be in the top 3 with LightGBM
I tried to implement time series prediction with GBDT
I tried to solve the problem with Python Vol.1
I tried to find the entropy of the image with python
I tried to simulate how the infection spreads with Python
I tried to analyze the whole novel "Weathering with You" ☔️
I tried to find the average of the sequence with TensorFlow
I tried to notify the train delay information with LINE Notify
I tried to summarize the code often used in Pandas
I tried to display the time and today's weather w
I tried to summarize the commands often used in business
I tried to implement the mail sending function in Python
I can't log in to the admin page with Django3
I tried to predict Boston real estate prices with PyCaret
I tried to divide the file into folders with Python
I tried to create an article in Wiki.js with SQLAlchemy
I tried to move the ball
I tried to estimate the interval.
(Python: OpenCV) I tried to output a value indicating the distance between regions while binarizing the video in real time.
I tried to automatically post to ChatWork at the time of deployment with fabric and ChatWork Api
I also tried to imitate the function monad and State monad with a generator in Python
I wrote a doctest in "I tried to simulate the probability of a bingo game with Python"
I tried to solve the ant book beginner's edition with python
I tried to automate the watering of the planter with Raspberry Pi
I tried to display the video playback time (OpenCV: Python version)
I tried to get started with Bitcoin Systre on the weekend
I tried to expand the size of the logical volume with LVM
For the time being, I want to convert files with ffmpeg !!
I tried to improve the efficiency of daily work with Python
I tried to log in to twitter automatically with selenium (RPA, scraping)
I tried to make a function to judge whether the major stock exchanges in the world are daylight saving time with python
I tried to implement Autoencoder with TensorFlow
I tried to summarize the umask command
I tried to implement permutation in Python
I tried tensorflow for the first time
I tried to visualize AutoEncoder with TensorFlow
I tried to recognize the wake word
I tried to get started with Hy
I tried to implement PLSA in Python 2
I tried to create serverless batch processing for the first time with DynamoDB and Step Functions
I tried to summarize the graphical modeling.
I tried to implement ADALINE in Python
I tried to estimate the pi stochastically
I tried to touch the COTOHA API
I tried to implement PPO in Python
I tried to implement CVAE with PyTorch
I tried playing with the image with Pillow