Since I often write scripts in Python for work, I will post the template code as a memorandum.

python3 series template

Overview

This template does the following:

Parsing command line arguments (using click)
Load configuration class
Log output (using logging)
Load library
Library unit tests

File placement

app_home/
       ├ bin/
       │   └  my_batch.py   #← Script to execute
       ├ conf/
       │   └  my_batch_conf.py #← Setting class
       ├ lib/
       │   ├  __init__.py    #← Required to load the module
       │   └  my_lib.py      #← Library
       ├ tests/        
       │   └  test_my_lib.py #← Unit test code
       ├ log/                #← Log output destination
       └ Pipfile             #← List the libraries to use

Contents of the main body my_batch.py

import sys
import os
import click
import logging

#The parent directory is the application home(${app_home})Set to
app_home = os.path.abspath(os.path.join( os.path.dirname(os.path.abspath(__file__)) , ".." ))
# ${app_home}To the library load path
sys.path.append(os.path.join(app_home))

#Load your own library
from lib.my_lib import MyLib

#Load configuration class
from conf.my_batch_conf import MyBatchConf

#Command line argument handling. must_arg is a required option, optional_arg is optional
@click.command()
@click.option('--must_arg','-m',required=True)
@click.option('--optional_arg','-o',default="DefaultValue")
def cmd(must_arg,optional_arg):
    #Program name without extension from own name(${prog_name})To
    prog_name = os.path.splitext(os.path.basename(__file__))[0]

    #Logger settings

    #format
    log_format = logging.Formatter("%(asctime)s [%(levelname)8s] %(message)s")
    #level
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)
    #Handler to standard output
    stdout_handler = logging.StreamHandler(sys.stdout)
    stdout_handler.setFormatter(log_format)
    logger.addHandler(stdout_handler)
    #Handler to log file
    file_handler = logging.FileHandler(os.path.join(app_home,"log", prog_name + ".log"), "a+")
    file_handler.setFormatter(log_format)
    logger.addHandler(file_handler)

    #Start processing
    try:
        #Log output
        logger.info("start")

        #Use command line arguments
        logger.error(f"must_arg = {must_arg}")
        logger.error(f"optional_arg = {optional_arg}")

        #Library call
        mylib = MyLib()
        logger.info(mylib.get_name())

        #Use of set values
        logger.info(MyBatchConf.key1)
        logger.info(MyBatchConf.key2)

        #Even if an exception occurs ...
        raise Exception("My Exception")

    except Exception as e:
        #Catch and log exceptions
        logger.exception(e)
        sys.exit(1)

if __name__ == '__main__':
    cmd()

Contents of configuration class conf / my_batch_conf.py

class MyBatchConf(object):
    key1 = "key1_value"
    key2 = True

*) I used to use configParser before, but the config class doesn't need to be parsed, and the IDE complements it, so I don't use it anymore.

Contents of library my_lib.py

class MyLib(object):
    def get_name(self):
        return "my_lib"

Contents of the library unit test code test_my_lib.py

import sys,os
import unittest

# ../Put lib in the load path
app_home = os.path.abspath(os.path.join( os.path.dirname(os.path.abspath(__file__)) , ".." ))
sys.path.append(os.path.join(app_home,"lib"))

# ../Loading the library under test
from my_lib import MyLib

class TestMyLib(unittest.TestCase):

    def test_get_name(self):
        ml = MyLib()
        self.assertEqual("my_lib", ml.get_name())

if __name__ == '__main__':
    unittest.main()

Run

Execute with no arguments

$ python bin/my_batch.py

Execution result (manual appears by click function)

Usage: my_batch.py [OPTIONS]
Try "my_batch.py --help" for help.

Error: Missing option "--must_arg" / "-m".

Execute with arguments

$ python bin/my_batch.py -m SpecifiedValue

Execution result

2019-06-28 16:42:53,335 [    INFO] start
2019-06-28 16:42:53,335 [   ERROR] must_arg = SpecifiedValue
2019-06-28 16:42:53,335 [   ERROR] optional_arg = DefaultValue
2019-06-28 16:42:53,335 [    INFO] my_lib
2019-06-28 16:42:53,336 [    INFO] key1_value
2019-06-28 16:42:53,336 [    INFO] True
2019-06-28 16:42:53,336 [   ERROR] My Exception
Traceback (most recent call last):
  File "bin/my_batch.py", line 62, in cmd
    raise Exception("My Exception")
Exception: My Exception

The same content is output to the log (log / my_batch.log)

2019-06-28 16:42:53,335 [    INFO] start
2019-06-28 16:42:53,335 [   ERROR] must_arg = SpecifiedValue
2019-06-28 16:42:53,335 [   ERROR] optional_arg = DefaultValue
2019-06-28 16:42:53,335 [    INFO] my_lib
2019-06-28 16:42:53,336 [    INFO] key1_value
2019-06-28 16:42:53,336 [    INFO] True
2019-06-28 16:42:53,336 [   ERROR] My Exception
Traceback (most recent call last):
  File "bin/my_batch.py", line 62, in cmd
    raise Exception("My Exception")
Exception: My Exception

test

Run unit tests

bash-3.2$ python tests/test_my_lib.py

Execution result

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

tests All test code test_ * .py below is executed at once

$ python -m unittest  discover tests "test_*.py"

Finally

This code is also available on github → https://github.com/fetaro/python-batch-template-for-v3

python2 system template

Overview

This template does the following:

Optional perspective
Read configuration file
Log output
Load library
Library unit tests

Difference from python3 series

Works with pyhon2 series
Use optparse instead of click to parse command line arguments
Character coding declaration #-*-coding: utf-8-*- is required at the beginning of the file (because the source code has Japanese)
You must use .format () to format the string instead of the f expression
ConfigParser package name is uppercase

File placement

app_home/
       ├ bin/
       │   └  my_batch.py   #← Script to execute
       ├ conf/
       │   └  my_batch.conf #← Configuration file
       ├ lib/
       │   ├  __init__.py   #← Required to load the module
       │   └  my_lib.py     #← Library
       ├ tests/        
       │   └  test_my_lib.py#← Unit test code
       └ log/               #← Log output destination

Contents of my_batch.py

# -*- coding: utf-8 -*-
import sys
import os
from optparse     import OptionParser
from ConfigParser import ConfigParser
import logging

#The parent directory is the application home(${app_home})Set to
app_home = os.path.abspath(os.path.join( os.path.dirname(os.path.abspath(__file__)) , ".." ))

# ${app_home}/Add lib to library load path
sys.path.append(os.path.join(app_home,"lib"))

#Load your own library
from my_lib import MyLib

if __name__ == "__main__" :
    #Program name without extension from own name(${prog_name})To
    prog_name = os.path.splitext(os.path.basename(__file__))[0]

    #Optional perspective
    usage = "usage: %prog (Argument-1) [options]"
    parser = OptionParser(usage=usage)
    parser.add_option("-d", "--debug",dest="debug", action="store_true", help="debug", default=False)

    #Store options and arguments separately
    (options, args) = parser.parse_args()

    #Argument check
    if len(args) != 1:
        sys.stderr.write("argument error. use -h or --help option\n")
        sys.exit(1)

    #Read the config file
    config = ConfigParser()
    conf_path = os.path.join(app_home,"conf", prog_name + ".conf")
    config.read(conf_path)

    #Logger settings

    #format
    log_format = logging.Formatter("%(asctime)s [%(levelname)8s] %(message)s") 
    #level
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)
    #Handler to standard output
    stdout_handler = logging.StreamHandler(sys.stdout)
    stdout_handler.setFormatter(log_format)
    logger.addHandler(stdout_handler)
    #Handler to log file
    file_handler = logging.FileHandler(os.path.join(app_home,"log", prog_name + ".log"), "a+")
    file_handler.setFormatter(log_format)
    logger.addHandler(file_handler)

    
    #Start processing
    try:
        #Log output
        logger.info("start")
        logger.error("arg1 = {0}".format(args[0]))

        #Get options
        logger.info(options.debug)

        #Library call
        mylib = MyLib()
        logger.info(mylib.get_name())

        #Read setting value
        logger.info(config.get("section1","key1"))
        logger.info(config.getboolean("section2","key2"))

        #Even if an exception occurs ...
        raise Exception("My Exception")
        
    except Exception as e:
        #Catch and log exceptions
        logger.exception(e)
        sys.exit(1)

Contents of my_batch.conf

[section1]
key1  = key1_value

[section2]
key2 = true

Contents of my_lib.py

class MyLib(object):
    def get_name(self):
        return "my_lib"

Contents of unit test code test_my_lib.py

# -*- coding: utf-8 -*-
import sys,os
import unittest

# ../Put lib in the load path
app_home = os.path.abspath(os.path.join( os.path.dirname(os.path.abspath(__file__)) , ".." ))
sys.path.append(os.path.join(app_home,"lib"))

# ../Loading the library under test
from my_lib import MyLib

class TestMyLib(unittest.TestCase):

    def test_get_name(self):
        ml = MyLib()
        self.assertEqual("my_lib", ml.get_name())

if __name__ == '__main__':
    unittest.main()

Run

bash-3.2$ python bin/my_batch.py argument1
2016-08-16 23:25:03,492 [    INFO] start
2016-08-16 23:25:03,492 [   ERROR] arg1 = argument1
2016-08-16 23:25:03,492 [    INFO] False
2016-08-16 23:25:03,492 [    INFO] my_lib
2016-08-16 23:25:03,492 [    INFO] key1_value
2016-08-16 23:25:03,492 [    INFO] True
2016-08-16 23:25:03,492 [   ERROR] My Exception
Traceback (most recent call last):
  File "bin/my_batch.py", line 73, in <module>
    raise Exception("My Exception")
Exception: My Exception

Contents of the log (log / my_batch.log)

2016-08-16 23:25:03,492 [    INFO] start
2016-08-16 23:25:03,492 [   ERROR] arg1 = argument1
2016-08-16 23:25:03,492 [    INFO] False
2016-08-16 23:25:03,492 [    INFO] my_lib
2016-08-16 23:25:03,492 [    INFO] key1_value
2016-08-16 23:25:03,492 [    INFO] True
2016-08-16 23:25:03,492 [   ERROR] My Exception
Traceback (most recent call last):
  File "bin/my_batch.py", line 73, in <module>
    raise Exception("My Exception")
Exception: My Exception

Unit test execution results

bash-3.2$ python tests/test_my_lib.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

tests When testing all the test code test_ *. Py below at once

bash-3.2$ python -m unittest  discover tests "test_*.py"

The contents of this file are also published on github. Please do as you like → https://github.com/fetaro/python-batch-template

Template for writing batch scripts in python

python3 series template

Overview

File placement

Contents

Run

test

Finally

python2 system template

Overview

File placement

Contents

Run