[PYTHON] I want to receive the configuration file and check if the JSON file generated by jinja2 is a valid JSON

The JSON file that receives the configuration file and is generated by jinja2 is used when deploying. I want to check if this generated JSON file is a valid JSON.

Command used

An image like j2cli.

Use the following two files.

--Configuration file (.json) --template file (.json.j2)

For example, when the configuration file is as follows

{
  "name": "staging"
}

Use a template like this.

{
  "app-name": "{{name}}-app",
  "batch-name": "{{name}}-batch"
}

As a result, the following JSON is output. This is used as the actual configuration file.

{
  "app-name": "staging-app",
  "batch-name": "staging-batch"
}

In the case of dev-like settings, the output may be as follows.

{
  "app-name": "dev-app",
  "batch-name": "dev-batch"
}

problem

The generated JSON may be invalid. I hope it will finally be noticed when it is deployed in production. I want to check in advance.

things to do

The policy is as follows.

1. Emit template with jinja2
2. Json.loads the emitted JSON again
3. If an error occurs, it should be invalid JSON

It looks like this in python code.

import jinja2
import json


env = jinja2.Environment(loader=jinja2.FileSystemLoader(["."]), undefined=jinja2.StrictUndefined)
t = env.get_or_select_template(template_path)
json.loads(t.render(**config))  #Error if invalid

However, there are some caveats.

Note 1 If the expected value is not included in the configuration file

{"foo": "foo-{{name}}"}

If you emit with jinja2 without setting anything in such a template, the following will occur, but no error will occur.

{"foo": "foo-"}

This can be detected by changing the undefined setting of jinja2. If you add ʻundefined = jinja2.StrictUndefined, ʻUndefinedError will occur.

Note 2 If the template contains an excessive amount of "}"

It is bad if the template is flawed and contains too many "}". Although the generated JSON is valid as JSON. Not the expected output.

If you accidentally write a template like the one below.

{
  "app-name": "{{name}}}-app"
}

The following is valid JSON, though. Not the expected output.

{
  "app-name": "staging}-app"
}

For the time being, you should also check the value of dict.

A script that checks if the result of emit with jinja2 is valid JSON

Based on the above, I made a script that checks whether the result of emit with jinja2 is valid JSON. Use it like this. If the variable environ does not exist, an error will occur as shown below. An example of using this is an error when two settings named ʻenviron` do not exist in the configuration file.

$ python check.py --conf-dir conf/ --template-dir template/ || echo ng
template/back.json.j2(conf/master.json): UndefinedError 'environ' is undefined
template/back.json.j2(conf/production.json): UndefinedError 'environ' is undefined
template/back.json.j2(conf/rook.json): UndefinedError 'environ' is undefined
ng

The implementation is as follows.

import sys
import os.path
import argparse
import jinja2
import json


def validate_conf(d, path=None):
    path = path or []
    if hasattr(d, "items"):
        for k, v in d.items():
            path.append(k)
            validate_conf(v, path=path)
            path.pop()
    elif isinstance(d, (list, tuple)):
        for i, x in enumerate(d):
            path.append(i)
            validate_conf(x, path=path)
            path.pop()
    elif isinstance(d, str):
        v = d
        if "{" in v:
            raise ValueError("invalid value: '{{' is included. path={}".format(path))
        if "}" in v:
            raise ValueError("invalid value: '}}' is included. path={}".format(path))
    return d


def check(env, config, template_path):
    t = env.get_or_select_template(template_path)
    data = json.loads(t.render(**config))
    return validate_conf(data)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--conf-dir", required=True)
    parser.add_argument("--template-dir", required=True)
    args = parser.parse_args()

    env = jinja2.Environment(loader=jinja2.FileSystemLoader(["."]), undefined=jinja2.StrictUndefined)
    status = 0
    for root, _, conf_files in os.walk(args.conf_dir):
        for conf_file in conf_files:
            try:
                confpath = os.path.join(root, conf_file)
                with open(confpath) as rf:
                    conf = json.load(rf)
            except Exception as e:
                sys.stderr.write("{confpath}: {e.__class__.__name__} {e}\n".format(confpath=confpath, e=e))
                status = 1
                continue

            for root2, _, template_files in os.walk(args.template_dir):
                for template_file in template_files:
                    try:
                        filepath = os.path.join(root2, template_file)
                        check(env, conf, filepath)
                    except Exception as e:
                        sys.stderr.write("{filepath}({confpath}): {e.__class__.__name__} {e}\n".format(
                            confpath=confpath, filepath=filepath, e=e)
                        )
                        status = 1
    sys.exit(status)


if __name__ == "__main__":
    main()