[PYTHON] Merge JSON format data with Ansible

Introduction

In Previous article, compare the difference between the JSON data defined in advance and the JSON data after setting, and check if the setting is reflected as expected. did. This time I would like to merge two JSON data using Ansible. Specifically, try merging existing and additional settings before configuring Cisco IOS-XE interface settings in RESTCONF.

1. Prepared environment

We have used CSR1000v 16.9.3 from the Cisco DevNet Sandbox IOS XE on CSR Recommended Code (https://devnetsandbox.cisco.com/RM/Diagram/Index/d6ec9e7c-9cf1-488b-8f1d-876cd9f52cb4). It was. I installed and used Ansible 2.9.2 on the venv virtual environment of Python 3.6.8.

2. Filter plugin

JSONMerge has been imported as an Ansible filter plugin. This is a Python-based module that merges head (additional JSON data) with base (original JSON data) and outputs the result. You can specify the merge method in Merge Strategies. If not specified, it will be ʻoverwriteand the old value will be overwritten. Actually, I think that there are many cases where you want to add an element to the array data[〇, △]. In that case, you can select ʻappend (simply add) or ʻarrayMergeById` (merge if the specific key value in the element is the same, add a new one). An example will be introduced later, but you can specify what data to process and how to process it in the same way as JSON Schema.

2-1. Advance preparation

Install JSON Merge.

$ pip install jsonmerge

2-2. Python code (excerpt)

Similar to this article, the following files are stored in the filter_plugins directory. (Actually, other custom filters are also written in the same file.) What we are doing is the following two points.

--Judgment whether JSON Merge is installed by HAS_JSONMERGE --Returns a merge of two JSON data according to JSON Schema

custom_filters1.py


from __future__ import absolute_import, division, print_function
__metaclass__ = type

from ansible.module_utils.six import PY3, string_types
from ansible.errors import AnsibleError, AnsibleFilterError

try:
    from jsonmerge import Merger
    HAS_JSONMERGE = True
except ImportError:
    HAS_JSONMERGE = False


class FilterModule(object):

    def jsonmerge(self, output1, output2, schema):
        if not HAS_JSONMERGE:
            raise AnsibleFilterError("JSONMerge not found. Run 'pip install jsonmerge'")
        merger = Merger(schema)
        merge_result = merger.merge(output1, output2)
        return merge_result

    def filters(self):
        return {
            'jsonmerge': self.jsonmerge
        }

3. Inventory file

Same as Previous article I used something.

  1. Playbook The contents of each task are as follows.

  2. Get the existing interface settings with the restconf_get module (1 of the GigabieEthernet1 to 3 has already been set)

  3. Output the result of task 1

  4. In the playbook variable content_data, define the Description of GigabitEthernet2, de-shutdown (no shutdown), and IP address settings. Merge the result of task 1 with content_data with the filter plugin jsonmerge. At this time, in the task variable schema, specify Merge Strategy of the interface setting (ʻinterface:) in ʻarrayMergeById. Specify that the interface name is the ID (ʻidRef: name`) and the same ID merges the settings.

playbook_restconf_int_merge1.yml


---

- hosts: cisco
  gather_facts: no

  vars:
    content_data:
      ietf-interfaces:interfaces:
        interface:
          - description: "For TEST"
            enabled: true
            ietf-ip:ipv4:
              address:
                - ip: 192.168.1.1
                  netmask: 255.255.255.0
            name: GigabitEthernet2

  tasks:
    - name: get interface info   # (1)
      restconf_get:
        path: /data/ietf-interfaces:interfaces
      register: result_before

    - name: display output data   # (2)
      debug:
        msg: "{{ result_before.response }}"

    - name: merge existing and setup interface settings   # (3)
      debug:
        msg: "{{ result_before.response | jsonmerge(content_data, schema) }}"
      vars:
        schema:
          properties:
            ietf-interfaces:interfaces:
              properties:
                interface:
                  mergeStrategy: arrayMergeById
                  mergeOptions:
                    idRef: name

5. Output result

If you look at the result of task 3, you can see that they are merged without any problem.

$ ansible-playbook -i inventory_restconf1.ini playbook_restconf_int_merge1.yml

PLAY [cisco] **********************************************************************************************************

TASK [get interface info] *********************************************************************************************
ok: [csr1000v-1]

TASK [display output data] ********************************************************************************************
ok: [csr1000v-1] => {
    "msg": {
        "ietf-interfaces:interfaces": {
            "interface": [
                {
                    "description": "MANAGEMENT INTERFACE - DON'T TOUCH ME",
                    "enabled": true,
                    "ietf-ip:ipv4": {
                        "address": [
                            {
                                "ip": "10.10.20.48",
                                "netmask": "255.255.255.0"
                            }
                        ]
                    },
                    "ietf-ip:ipv6": {},
                    "name": "GigabitEthernet1",
                    "type": "iana-if-type:ethernetCsmacd"
                },
                {
                    "enabled": false,
                    "ietf-ip:ipv4": {},
                    "ietf-ip:ipv6": {},
                    "name": "GigabitEthernet2",
                    "type": "iana-if-type:ethernetCsmacd"
                },
                {
                    "description": "Network Interface",
                    "enabled": false,
                    "ietf-ip:ipv4": {},
                    "ietf-ip:ipv6": {},
                    "name": "GigabitEthernet3",
                    "type": "iana-if-type:ethernetCsmacd"
                }
            ]
        }
    }
}

TASK [merge existing and setup interface settings] ********************************************************************
ok: [csr1000v-1] => {
    "msg": {
        "ietf-interfaces:interfaces": {
            "interface": [
                {
                    "description": "MANAGEMENT INTERFACE - DON'T TOUCH ME",
                    "enabled": true,
                    "ietf-ip:ipv4": {
                        "address": [
                            {
                                "ip": "10.10.20.48",
                                "netmask": "255.255.255.0"
                            }
                        ]
                    },
                    "ietf-ip:ipv6": {},
                    "name": "GigabitEthernet1",
                    "type": "iana-if-type:ethernetCsmacd"
                },
                {
                    "description": "For TEST",
                    "enabled": true,
                    "ietf-ip:ipv4": {
                        "address": [
                            {
                                "ip": "192.168.1.1",
                                "netmask": "255.255.255.0"
                            }
                        ]
                    },
                    "ietf-ip:ipv6": {},
                    "name": "GigabitEthernet2",
                    "type": "iana-if-type:ethernetCsmacd"
                },
                {
                    "description": "Network Interface",
                    "enabled": false,
                    "ietf-ip:ipv4": {},
                    "ietf-ip:ipv6": {},
                    "name": "GigabitEthernet3",
                    "type": "iana-if-type:ethernetCsmacd"
                }
            ]
        }
    }
}

PLAY RECAP ************************************************************************************************************
csr1000v-1                 : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0  

(Reference) Interface setting change

For your reference, paste the result of actually setting the interface with the ʻuri` module.

(Reference 1) Playbook for changing settings

playbook_restconf_int_merge2.yml


---

- hosts: cisco
  gather_facts: no

  vars:
    content_data:
      ietf-interfaces:interfaces:
        interface:
          - description: "For TEST"
            enabled: true
            ietf-ip:ipv4:
              address:
                - ip: 192.168.1.1
                  netmask: 255.255.255.0
            name: GigabitEthernet2

  tasks:
    - name: merge interface settings
      uri:
        url: https://{{ansible_host}}:{{ansible_httpapi_port}}/restconf/data/ietf-interfaces:interfaces
        method: PATCH
        headers:
          Content-Type: application/yang-data+json
          Accept: application/yang-data+json
        body_format: json
        body: "{{ content_data | to_json }}"
        status_code:
          - 200
          - 204
        url_username: "{{ ansible_user }}"
        url_password: "{{ ansible_password }}"
        force_basic_auth: yes
        validate_certs: no

    - name: get interface info
      restconf_get:
        path: /data/ietf-interfaces:interfaces
      register: result

    - name: display output data
      debug:
        msg: "{{ result.response }}"

(Reference 2) Output result of setting change

$ ansible-playbook -i inventory_restconf1.ini playbook_restconf_int_merge2.yml

PLAY [cisco] **********************************************************************************************************

TASK [merge interface settings] ***************************************************************************************
ok: [csr1000v-1]

TASK [get interface info] *********************************************************************************************
ok: [csr1000v-1]

TASK [display output data] ********************************************************************************************
ok: [csr1000v-1] => {
    "msg": {
        "ietf-interfaces:interfaces": {
            "interface": [
                {
                    "description": "MANAGEMENT INTERFACE - DON'T TOUCH ME",
                    "enabled": true,
                    "ietf-ip:ipv4": {
                        "address": [
                            {
                                "ip": "10.10.20.48",
                                "netmask": "255.255.255.0"
                            }
                        ]
                    },
                    "ietf-ip:ipv6": {},
                    "name": "GigabitEthernet1",
                    "type": "iana-if-type:ethernetCsmacd"
                },
                {
                    "description": "For TEST",
                    "enabled": true,
                    "ietf-ip:ipv4": {
                        "address": [
                            {
                                "ip": "192.168.1.1",
                                "netmask": "255.255.255.0"
                            }
                        ]
                    },
                    "ietf-ip:ipv6": {},
                    "name": "GigabitEthernet2",
                    "type": "iana-if-type:ethernetCsmacd"
                },
                {
                    "description": "Network Interface",
                    "enabled": false,
                    "ietf-ip:ipv4": {},
                    "ietf-ip:ipv6": {},
                    "name": "GigabitEthernet3",
                    "type": "iana-if-type:ethernetCsmacd"
                }
            ]
        }
    }
}

PLAY RECAP ************************************************************************************************************
csr1000v-1                 : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0 

Finally

In this case, the merge result was as expected. However, if I wanted to add more than one nested array data instead of overwriting it, the JSON schema conflicted and didn't work. There may be some way to do it, but I've reached the limit of my energy and I'd like to try again: sweat:

Recommended Posts

Merge JSON format data with Ansible
Format json with Vim (with python)
Read json data with python
Export DB data in json format
[Introduction to Python] How to handle JSON format data
Convert json format data to txt (using yolo)
Flow memo when getting json data with urllib
Data analysis with python 2
Use ansible with cygwin
[Python] Use JSON with Python
Merge datasets with pandas
Visualize data with Streamlit
Reading data with TensorFlow
json parse with gdb
Merge array with PyYAML
Output log in JSON format with Python standard logging
Data visualization with pandas
Read json file with Python, format it, and output json
Shuffle data with pandas
Data Augmentation with openCV
Normarize data with Scipy
Try writing JSON format data to object storage Cloudian/S3
SQL format with sqlparse
LOAD DATA with PyMysql
[Python] Use JSON format data as a dictionary type object
Sample data created with python
Embed audio data with Jupyter
Graph Excel data with matplotlib (1)
Load nested json with pandas
POST json with Python3 script
Data handling 3 (development) About data format
Extract Twitter data with CSV
Get Youtube data with python
Write data in HDF format
Binarize photo data with OpenCV
Graph Excel data with matplotlib (2)
Save tweet data with Django
Format C source with pycparser
Handle JSON files with Matlab
String format with Python% operator
Data processing tips with Pandas
Interpolate 2D data with scipy.interpolate.griddata
Get mackerel data and spit out json for ansible dynamic inventory