How to manage the configuration of the existing environment after implementing Ansible)

As mentioned above, by implementing Ansible, the construction work itself has become easier. In addition, for business departments that prepare infrastructure for "new", After receiving a GitHub pull request from the requester, the infrastructure staff confirms and approves the changed part, the setting is reflected in the masseter branch, and the construction work is completed. Was

Illustrate a simpler GitHub Flow operation that incorporates pull requests / reviews (1/2) http://www.atmarkit.co.jp/ait/articles/1401/21/news042.html

However, it was not so easy for business departments that have an existing environment.

If I was asked to change the MW settings of a server, I had to prepare various Ansible configuration files including the Vars file from scratch, which was a very troublesome situation.

I came up with the idea of creating an automatic inventory file creation tool.

Because of the above issues, I came up with a tool that automatically generates Ansible configuration files from the server in the existing environment.

I made a shabby image diagram. The flow of the tool is as follows

1. Specify the SV and MW to be executed on the local PC
2. push to git
3. From the jenkins GUI, specify the file mentioned in ② and execute the job.
4. The jenkins execution server clones the git master repository and obtains the specified target server and MW configuration file by sftp.
5. After acquisition, the conversion script to Ansible file is executed, and Ansible configuration file is created under the execution directory (vars / host_vars / group_vars).
6. Various created configuration files will be uploaded to git

Demo (Try to automatically create vars file from configuration file using apache as an example)

Here, I would like to extract the Apache configuration file and automatically create the following Vars file. It is an image of actually writing a part of ⑤ Actually, it is divided into various modules so that it can meet various requirements, but since it is a demo, it is extracted for simple Apache parameters.

environment

・ Python3.4 -Apache 2.2.15

Vars file to create

After executing the tool, the information of the configuration file (/etc/httpd/conf/httpd.conf) is extracted and stored in the following items.

KeepAlive: ''
KeepAliveTimeout: ''
MaxKeepAliveRequests: '100'
PidFile: run/httpd.pid
ServerRoot: '"/etc/httpd"'
ServerTokens: OS
Timeout: '60'
load_module:[]

Preparation

Install apache and create main.py directly under the / etc / httpd directory Check if httpd.conf to be extracted of parameters is placed just in case

[root@localhost httpd]$ sudo yum install httpd
[root@localhost httpd]$ pwd
/etc/httpd
[root@localhost httpd]# ls conf/httpd.conf
conf/httpd.conf
[root@localhost httpd]$vi main.py

Create main.py

main.py

# coding: UTF-8
import yaml
from pathlib import Path
import re


#Describe the process of extracting parameters
class CommonFunc:
    def __init__(self):
        pass

    # file_Stores the contents of the target file specified by the name argument in an array line by line.
    def func_read_files(self, file_name):
        lines = []
        with file_name.open() as f:
            lines.extend(f.readlines())
        return lines

    #Store the line containing the character string specified by parameter in the array
    def func_search_parameter(self, lines, parameter):
        searched_line = []
        for line_tmp in lines:
            #Remove leading whitespace
            line = re.sub(r'^\s+', '', line_tmp)
            try:
                if line.startswith(parameter):
                    #Delete commented out line
                    tmp = re.sub(r"^\s*#.*", "", line)
                    searched_line.append(tmp.rstrip('\r\n'))
            except:
                searched_line = ""

        return searched_line

    #Searches the line containing the character string specified by parameter and stores it in the array. Output the difference from the template file using the set
    def func_set_parameter(self, lines, compared_lines, parameter):
        src_list = []
        dst_list = []
        for line in lines:
            if line.find(parameter) >= 0:
                src_list.append(line.rstrip('\r\n'))
        for compared_line in compared_lines:
            dst_list.append(compared_line.rstrip('\r\n'))
        src_set = set(src_list)
        return src_set.difference(dst_list)

    #Search for lines that contain the string specified by parameter. Separate with spaces and store the second in the array
    def func_split_line_unique(self, lines, parameter):
        splited_line = []
        for line_tmp in lines:
            #Remove leading whitespace
            line = re.sub(r'^\s+', '', line_tmp)
            if line.startswith(parameter):
                try:
                    splited_line = line.split()[1]
                except:
                    splited_line = ""
        return splited_line


#Create a dictionary and store the value in Value. The process of extracting the actual parameters calls the process of CommonFunc.
class MakeApacheDictionary(CommonFunc):

    def __init__(self):
        self.output_files = "test_vars.yml"
        self.httpd_conf_path = Path("/etc", "httpd", "conf", "httpd.conf")
        self.module_template_path = Path("/etc", "httpd", "load_module_template.txt")
        self.httpd_lines = self.func_read_files(self.httpd_conf_path)
        self.base_module_lines = self.func_search_parameter(self.httpd_lines, "LoadModule")
        self.base_module_template_lines = self.func_read_files(self.module_template_path)



    def apache_yaml_dict(self):
        self.yml_list = {
        #Unique in instance
            'ServerTokens': "",
            'ServerRoot': "",
            'PidFile': "",
            'Timeout': "",
            'KeepAlive': "",
            'MaxKeepAliveRequests': "",
            'KeepAliveTimeout': "",
        #Output the difference from the template file as a set
            'load_module': []}

        d = self.yml_list
        d['ServerTokens'] = self.func_split_line_unique(self.httpd_lines, "ServerTokens")
        d['ServerRoot'] = self.func_split_line_unique(self.httpd_lines, "ServerRoot")
        d['PidFile'] = self.func_split_line_unique(self.httpd_lines, "PidFile")
        d['Timeout'] = self.func_split_line_unique(self.httpd_lines, "Timeout")
        d['MaxKeepAliveRequests'] = self.func_split_line_unique(self.httpd_lines, "MaxKeepAliveRequests")
        d['load_module'].extend(self.func_set_parameter(self.base_module_lines, self.base_module_template_lines,"LoadModule"))


    def create_yaml(self):
        with open(self.output_files, 'w') as f:
            f.write(yaml.safe_dump(self.yml_list, default_flow_style=False))


#Instance generation
a = MakeApacheDictionary()
a.apache_yaml_dict()
a.create_yaml()

Description

--Class is divided into "CommonFunc" class that describes the process of actually extracting parameters, and MakeApacheDictionary that creates a dictionary and dumps it as a yaml file. --The latter class inherits the former so that the extraction process can be used. --Unique values in configuration files such as ServerRoot and TimeOut can be obtained by hooking with the startswith method and splitting as described in func_split_line_unique. --For items such as LoadModule and RewriteCond that have multiple settings for the same item, prepare standard settings separately as a template file and output the difference using a set.

Template example

func_set_parameter searches and lists the lines starting with LoadModule in httpd.conf, compares them with the contents described in the template file, and outputs the difference.

[root@localhost collect_config]# more load_module_template.txt
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
etc ...

Execution result

The created dictionary Passed to the create_yml function (where the yaml.safe_dump module is used) and dumped to the script execution directory

[root@localhost httpd]# python main.py
[root@localhost httpd]# more test_vars.yml
KeepAlive: 'Off'
KeepAliveTimeout: '15'
MaxKeepAliveRequests: '100'
PidFile: run/httpd.pid
ServerRoot: '"/etc/httpd"'
ServerTokens: OS
Timeout: '60'
load_module:
- LoadModule auth_basic_module modules/mod_auth_basic.so
- LoadModule authn_file_module modules/mod_authn_file.so
- LoadModule auth_digest_module modules/mod_auth_digest.so
[root@localhost httpd]#

At the end

Thank you for reading until the end. I hope it will be useful for those who are similarly troubled with configuration management of the existing environment.