[PYTHON] I tried to describe the traffic in real time with WebSocket

This is the article on the 11th day of NetOpsCoding AdventCalender.

This time, I will implement a traffic graph that is almost always confirmed when performing network operations, using a technology called * WebSocket *. The demo and source code are pasted at the end.

** This implementation method is implemented by referring to the tool used by @yuyarin for monitoring in Cedec2015. ** **

1.First of all

Cacti, Zabbix, MRTG, etc. are famous as tools for monitoring traffic. In most cases, the SNMP acquisition interval is about 5 minutes. Basically, I am satisfied with the confirmation, but in actual work such as opening a new line / adjusting traffic, I often want to see the traffic at * the moment *. Depending on the device, real-time traffic can be monitored based on the CLI, but with the CLI, it is difficult to notice changes in the status of multiple locations, and is there a delay in anomaly detection?

In order to solve such a problem, I will create a graph that is * visually easy to understand and can capture short-term traffic changes in real time.

2. To draw in real time on the web = There are two main methods for dynamically drawing a graph on a Web screen.

• How to get updated data from the browser side regularly (polling by Ajax)
• How to capture and reflect the data thrown by the server (two-way communication by WebSocket)

WebSocket is a relatively new technology. You can freely decide the timing of data transmission / reception and the method of communication. Once you have posted a session, you can easily transfer data in both directions.

Graph drawing can be realized with either, but anyway, I will try using WebSocket, which seems to be applicable to other tool implementations! !! Anyway, let's use it immediately!

3. Execution environment =

• Python 2.7.6
  • tornado 4.3
• nginx 1.8.0
• jquery 1.8.2
• HighCharts

To display on the Web, a platform such as nginx or apache is required. This demo server uses nginx. In addition, 8080 is used for WS communication this time.

The WebSocket library uses tornado. There are Gevent-WebSocket, ws4py, etc., but tornado came to me the most.

Also, although HighCharts is used as a graph drawing tool, there is no problem with other graph tools that can be added dynamically. Also, HightCharts is charged except for personal use, so those who are concerned should use other tools. ccchart is recommended because it has abundant samples.

5. Implementation on the WebSocket-Server side =

Let's implement the server side. The server-side implementation of WebSocket is not difficult if only the point of "asynchronous processing" is suppressed. When it listens at a specific URI and the client is connected, it becomes possible to freely exchange data with each other, so it becomes possible to send the result of acquiring SNMP. SNMP is acquired by command execution by subprocess, and the result is extracted by regular expression. Also, this time, the OID that looks at the Counter of the NIC of the own server is specified.

After implementing the following code, just execute it with the python command and the WebSocket server construction is completed!

ws_traffic_server.py

#! /usr/bin/env python
# -*- coding: utf-8 -*-
import tornado.ioloop
import tornado.web
import tornado.websocket
from tornado.options import define, options ,parse_command_line
from datetime import datetime
import shlex, subprocess , time ,re , json ,threading

SLEEP_TIME = 20 #Interval to get SNMP

COMMUNITY = 'dev'
IP = '127.0.0.1'
IFMIB_OID = '1.3.6.1.2.1.31.1.1.1'

snmp_command_in = 'snmpget -v 2c -c %s %s %s.6.2'%(COMMUNITY,IP,IFMIB_OID)
snmp_command_out = 'snmpget -v 2c -c %s %s %s.10.2'%(COMMUNITY,IP,IFMIB_OID)

#Specify the port that WebSocket listens on
define("port", default = 8080,type = int)

class SendWebSocket(tornado.websocket.WebSocketHandler):

    #Event called when a connection is secured
    def open(self):
        print 'Session Opened. IP:' + self.request.remote_ip

    #Event when a disconnection event occurs, such as when the browser is closed
    def on_close(self):
        print "Session closed"

    #Event called when a message is sent from the client
    def on_message(self, message):
        if message == 'hello':
            pre_counter_in = int(self.exe_snmp(snmp_command_in))
            pre_counter_out = int(self.exe_snmp(snmp_command_out))

            #Time on WebSocket.timer cannot be used. Reproduce the delay with the following variable CallBack
            #SLEE_Start communication processing with SNMP in the latter half after TIME seconds
            tornado.ioloop.IOLoop.instance().call_later(SLEEP_TIME,self.snmp_second_half,{'in_counter':pre_counter_in,'out_counter':pre_counter_out})

    #SLEEP_Get SNMP after TIME seconds
    def snmp_second_half(self,pre_counters):
        result = {}
        pos_counter_in = int(self.exe_snmp(snmp_command_in))
        pos_counter_out = int(self.exe_snmp(snmp_command_out))

        #Traffic calculation from the counter difference from the specified number of seconds
        traffic_in = (pos_counter_in - pre_counters['in_counter'])  / SLEEP_TIME
        traffic_out = (pos_counter_out - pre_counters['out_counter'])  / SLEEP_TIME
        #Send JSON-converted traffic data to a web client
        try:
            result.update({'traffic_in' : traffic_in , 'traffic_out' : traffic_out})
            result.update({'timestamp' : time.mktime(datetime.now().timetuple())})
            self.write_message(json.dumps(result))
        except:
            print "Client is already disconnectted."

    #Returns only the value of the SNMP execution result
    #e.g. [IF-MIB::ifHighSpeed.21 = Gauge32: 1000] -> [1000]
    def exe_snmp(self,snmp_command):
        split_command = shlex.split(snmp_command)
        exec_output = subprocess.check_output(split_command)
        r = re.compile("(.*)(: )(.*)")
        snmp_result = r.match(exec_output).group(3)
        return snmp_result

    #Only accepts communication from hosts that are specified as not True
    def check_origin(self, origin):
        return True

#Waits for a connection request to WS with the specified URI
app = tornado.web.Application([
    (r"/ws/ifmon/", SendWebSocket),
])

if __name__ == "__main__":
    parse_command_line()
    app.listen(options.port)
    mainloop = tornado.ioloop.IOLoop.instance()
    mainloop.start() #Start WebSocket Server

What is easy to get hooked on here is the process of acquiring the traffic of network devices from SNMP and calculating it. Traffic calculation by SNMP (Current IFcounter-IFcounter after n seconds) / n Since it cannot be calculated without waiting for several tens of seconds, the program must also wait for n seconds. However, time.sleep () in the standard library cannot be used for asynchronous communication such as WebSocket. Since the entire process sleeps, it will affect other viewers as well.

This was solved by a function in tornado called ʻIOLoop.call_later ()` that can call back after a delay.

I feel that the program is being created by force, so I would appreciate it if you could comment if you should write it correctly.

6. Client-side implementation = The processing on the client side is performed by Javascript. The client side needs to know four event handlers in order to handle the traffic data sent from the server.

When the connection to the server is completed and the ʻonopenevent occurs, a signalhellois sent to the server to start SNMP. When you sendhello, the server side reacts and sends the traffic data. When the reception of data from the server is detected, the ʻonmessage event is generated, so the graph is drawn in this event. If you catch an action that disconnects WebSocket such as closing the browser, the ʻonclose` event will occur and the communication will end there.

The graph settings are long, so I will omit them. The relevant part of WebSocket processing is implemented as follows.

ws_client.js

var WS_URL = 'ws://www5445uo.sakura.ne.jp:8080/ws/ifmon/'

var init_ws_Traffic = function(){
  var traffic_chart =Traffic_Chart('traffic_chart', {}); //Highhart graph constructor
  var ws = new WebSocket(WS_URL);
  console.log('----- WebSocket Open -----')

  ws.onopen = function() {//WS connection establishment
    ws.send('hello');
  };

  ws.onerror = function(event){
    $("p#error").text('Failed join to the server');
    console.log('Connection faild....')
  };

  ws.onmessage = function(event) { //Traffic data processing
    var data = JSON.parse(event.data);
    var timestamp = data['timestamp']*1000 + 60 * 9 * 60 * 1000;
    var value = data['traffic_in']
    document.getElementById('traffic_in').innerHTML = convert_bps(value); //Display as text
    traffic_chart.series[0].addPoint([timestamp, value], true, true); //Inbound Traffic Graph Add is done here
    value = data['traffic_out']
    document.getElementById('traffic_out').innerHTML = convert_bps(value);
    traffic_chart.series[1].addPoint([timestamp, value], true, true);//Outbound Traffic Graph Add is done here
    ws.send('hello'); //Next traffic request
  };
  ws.onclose = function () {
    ws.send('close');
    ws.close();
  };
  window.onbeforeunload = function () {
    ws.send('close');
    ws.close()
  };
}

The above two are necessary for WebSocket communication. If you check what you actually made on the Web screen, it will look like this. It's a graph ...!

Now you can get a traffic graph every 10 seconds for work! If you are making something similar and have a better method, please let me know.

7. Bonus (demo and source code) = I don't think it's good without the feeling of being updated in real time, so I uploaded what I made to the demo server. It may be for a limited time, but please take a look at what it looks like. If you wait for about 20 seconds, something will change.

** There is no live demo because the demo server has disappeared **

Also, the code used in the demo is listed below. https://github.com/Mabuchin/wstraffic

8. Finally = We have introduced how to implement a traffic graph for use during work! I think that it will be more convenient if you modify it according to your purpose, such as making it possible to depict the traffic of multiple devices in one graph with a little rewriting. If you have a chance to use it, please try it!