Device monitoring with On-box Python in IOS-XE

Introduction

When monitoring Cisco network equipment, you may want to see the results of the show command. This time, I would like to get the result of the show command from the Guest Shell of IOS-XE using python and send it to Elasticsearch for visualization.

About Guest Shell

Guest Shell is a Linux container environment hosted on IOx and is implemented in products such as the Catalyst 9000 series and ISR 4000 series. Guest Shell provides a python cli library that allows you to execute IOSd commands from a python script that runs in Guest Shell. For more information on IOx, please refer to here.

The image diagram when communicating is as follows.

[Programmability Configuration Guide, Cisco IOS XE Fuji 16.9.x](https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/prog/configuration/169/b_169_programmability_cg/guest_shell. from html)

environment

Build the environment with the following configuration. (The construction method is omitted) The monitoring target is CSR1. Although it is simplified, we also prepared a VM to put a communication load on CSR1.

Elasticsearch preparation

Elasticsearch is a full-text search engine and is also used as a database management system called a document store (or document-oriented). I will write the terms briefly. (I understand until I get tired of it)

We will acquire the following information as monitoring items from CSR1.

--Time (date) --IOSd CPU usage (%) --kernel CPU usage (%) --IOSd memory usage (%) --Kernel memory usage (%) --GigabitEthernet1 transmission / reception amount (bps) --Interface / line protocol status list (text) --Routing table (text)

Define an Index Template to store the above data. Let's also set the Index when storing Document from CSR1 to router-csr.

curl -XPUT 'http://elasticsearch.example.com:9200/_template/router' -H 'Content-Type: application/json' -d @router_template.json

router_template.json

{
  "template": "router-*",
  "mappings":{
    "monitoring": {
      "dynamic": "strict",
      "properties": {
        "@timestamp":{
          "type": "date",
          "format": "date_time_no_millis"
        },
        "cpu_utilization_iosd":{
          "type": "integer"
        },
        "cpu_utilization_kernel":{
          "type": "integer"
        },
        "memory_utilization_iosd":{
          "type": "integer"
        },
        "memory_utilization_kernel":{
          "type": "integer"
        },
        "GigabitEthernet1_input":{
          "type": "integer"
        },
        "GigabitEthernet1_output":{
          "type": "integer"
        },
        "interface_status":{
          "type": "keyword"
        },
        "routing_table":{
          "type": "keyword"
        }
      }
    }
  }
}

CSR 1000V settings

For the transmission / reception amount of GigabitEthernet1, we will use the average for 30 seconds. It is possible to calculate from the difference in the number of packets sent and received, but I think it is easier to adjust load-interval and get it from show interface.

CSR1#conf t
CSR1(config)#interface gigabitethernet 1
CSR1(config-if)#load-interval 30

Next, set to enable Guest Shell and set NAT to communicate from Guest Shell at the same time.

CSR1#conf t
CSR1(config)#iox

CSR1(config)#ip http server

CSR1(config)# interface GigabitEthernet3
CSR1(config-if)# ip nat outside
CSR1(config-if)# exit

CSR1(config)# interface VirtualPortGroup0
CSR1(config-if)# ip address 192.168.35.1 255.255.255.0
CSR1(config-if)# ip nat inside
CSR1(config-if)# exit

CSR1(config)# ip nat inside source list GS_NAT_ACL interface GigabitEthernet3 overload
CSR1(config)# ip access-list standard GS_NAT_ACL
CSR1(config-std-nacl)# permit 192.168.35.0 0.0.0.255
CSR1(config-std-nacl)# exit

CSR1(config)#app-hosting appid guestshell
CSR1(config-app-hosting)# app-vnic gateway0 virtualportgroup 0 guest-interface 0
CSR1(config-app-hosting-gateway0)# guest-ipaddress 192.168.35.2 netmask 255.255.255.0
CSR1(config-app-hosting-gateway0)# app-default-gateway 192.168.35.1 guest-interface 0
CSR1(config-app-hosting)# name-server0 192.0.2.1
CSR1(config-app-hosting)# end

CSR1#guestshell enable

Guest Shell is now enabled. Running guestshell from privileged mode will launch the container bash and allow you to log in. Let's start Guset Shell and execute IOSd commands.

CSR1#guestshell
[guestshell@guestshell ~]$ dohost "show app-hosting list"

App id                           State
------------------------------------------------------
guestshell                       RUNNING

[guestshell@guestshell ~]$ python
Python 2.7.5 (default, Jun 17 2014, 18:11:42) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cli
>>> cli.cli("show app-hosting list")
'\nApp id                           State\n------------------------------------------------------\nguestshell                       RUNNING\n'
>>> exit()
[guestshell@guestshell ~]$

Use EEM to execute Guest Shell commands from IOS-XE. You can run a python script on a regular basis by triggering ʻevent timer watchdog`.

CSR1#conf t
CSR1(config)#event manager applet elasticsearch
CSR1(config-applet)#event timer watchdog time 30
CSR1(config-applet)#action 100 cli command "enable"
CSR1(config-applet)#action 200 cli command "guestshell run python /home/guestshell/monitoring.py"

Execution script

Extract the target character string from the show command corresponding to each monitoring item and use it as the value of the dictionary type object, and use the same key as Mapping. It is necessary to convert the format for time and calculate the value for memory usage.

--Time (date)-> show clock --IOSd CPU usage (%)-> show processes cpu | i CPU --kernel CPU usage (%)-> show processes cpu platform | i CPU --IOSd memory usage (%)-> show processes memory | i Processor --kernel memory usage (%)-> show processes memory platform | i System memory --GigabitEthernet1 send / receive (bps)-> show interface GigabitEthernet1 --Interface / line protocol status list (text)-> show ip interface brief --Routing table (text)-> show ip route

Concatenate all dictionary objects and send them as JSON data to Elasticsearch.

The code will be posted at the link below. https://gist.github.com/ecodrive-18/f808df5a69b9a47e97288789cfa53c9d

Install the necessary libraries before executing.

CSR1#guestshell 
[guestshell@guestshell ~]$ sudo pip install -r requirements.txt 

requirements.txt

elasticsearch>=6.0.0,<7.0.0
python-dateutil==2.8.1

Confirmation and visualization of execution results

Try to query Elasticsearch to see if the script execution result can be stored.

curl -XGET 'http://elasticsearch.example.com:9200/router-csr/monitoring/_search?size=1' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2765  100  2765    0     0   128k      0 --:--:-- --:--:-- --:--:--  128k
{
  "took": 13,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 232,
    "max_score": 1,
    "hits": [
      {
        "_index": "router-csr",
        "_type": "monitoring",
        "_id": "hHV2Xm4BlqMaqiTz5GDW",
        "_score": 1,
        "_source": {
          "GigabitEthernet1_output": "30000",
          "routing_table": "\nCodes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP\n       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area \n       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2\n       E1 - OSPF external type 1, E2 - OSPF external type 2\n       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2\n       ia - IS-IS inter area, * - candidate default, U - per-user static route\n       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP\n       a - application route\n       + - replicated route, % - next hop override, p - overrides from PfR\nGateway of last resort is 172.16.0.254 to network 0.0.0.0\nS*    0.0.0.0/0 [254/0] via 172.16.0.254\n      10.0.0.0/32 is subnetted, 2 subnets\nO        10.0.0.254 [110/2] via 192.168.255.2, 19:32:27, GigabitEthernet2\nO        10.100.0.254 [110/2] via 192.168.255.2, 19:32:27, GigabitEthernet2\n      172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks\nC        172.16.0.0/24 is directly connected, GigabitEthernet3\nL        172.16.0.207/32 is directly connected, GigabitEthernet3\nS        172.16.0.254/32 [254/0] via 172.16.0.254, GigabitEthernet3\n      192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks\nC        192.168.0.0/24 is directly connected, GigabitEthernet1\nL        192.168.0.254/32 is directly connected, GigabitEthernet1\nO     192.168.1.0/24 [110/2] via 192.168.255.2, 19:34:13, GigabitEthernet2\n      192.168.35.0/24 is variably subnetted, 2 subnets, 2 masks\nC        192.168.35.0/24 is directly connected, VirtualPortGroup0\nL        192.168.35.1/32 is directly connected, VirtualPortGroup0\n      192.168.255.0/24 is variably subnetted, 2 subnets, 2 masks\nC        192.168.255.0/30 is directly connected, GigabitEthernet2\nL        192.168.255.1/32 is directly connected, GigabitEthernet2\n",
          "cpu_utilization_iosd": "0",
          "cpu_utilization_kernel": "22",
          "memory_utilization_kernel": "61",
          "@timestamp": "2019-11-12T16:14:30+09:00",
          "interface_status": "\nInterface              IP-Address      OK? Method Status                Protocol\nGigabitEthernet1       192.168.0.254   YES manual up                    up      \nGigabitEthernet2       192.168.255.1   YES manual up                    up      \nGigabitEthernet3       172.16.0.207    YES DHCP   up                    up      \nVirtualPortGroup0      192.168.35.1    YES manual up                    up      \n",
          "memory_utilization_iosd": "14",
          "GigabitEthernet1_input": "787000"
        }
      }
    ]
  }
}

There was information sent from CSR1 in hits.hits._source. It is stored properly.

Visualize with Kibana

Let's visualize the information accumulated in Elasticsearch with Kibana. I couldn't find the function to display the text stored in Visualize as it is ... It seems that Markdown of Canvas has that function, so I tried using it. If you display the text as it is in the code block, line breaks will be made, so it's nice. Of course, you can also create graphs.

Summary

I tried to visualize by sending device information from the Guest Shell of IOS-XE. If this function is implemented in many devices, it may be possible to monitor devices using a monitoring agent instead of SNMP. By the way, when performing pull type monitoring, Python must be a resident process with Guest Shell getting IOSd VTY, so maxrun must be set to 0 when setting EEM's ʻevent`. ..