[LINUX] Bash (Shell Script), I hate it anymore: What I researched to replace Bash with Python 3.x

2020/04/10 postscript

[To move from Bash to Ruby and Python3 I compared more processing with Bash <-> Ruby <-> Python3 (introduced on another site). ](Https://debimate.jp/2020/04/05/bashshell-script%e3%81%8b%e3%82%89ruby%e3%82%84python%e3%81%ab%e4%b9%97% e3% 82% 8a% e6% 8f% 9b% e3% 81% 88% ef% bc% 81% e9% a0% bb% e7% b9% 81% e3% 81% ab% e4% bd% bf% e3% 81% 86% e5% 87% a6% e7% 90% 86% e3% 82% 92% e5% 90% 84% e8% a8% 80 /)


This article describes what I did to replace the Bash Script with Python 3.x. The motivation for thinking about replacement was that I thought that "Bash Script over 300 Steps will strangle me."

I understand the strengths of Bash. Except for embedded environments, it is guaranteed to work with major distributions and If you write with POSIX compatibility in mind, you will have fewer corrections when porting. Familiar to many developers above all, it means a lot of people (members) who can modify the Script.     But as a practical matter, I don't write Bash given the benefits mentioned above. Because I create it without thinking about it, I will see pain at a later date (example: the flow below).

  1. Create a Script for fun
  2. Personal, but just in case, deploy Script to matter members
  3. Receive complaints from members (doesn't work in virtual environment, is not robust, has no sense, etc ...)
  4. As a result of the correction, ** Script is bloated **
  5. Come up with improvements (additional features) in the middle of the night and bloat the Script the next morning
  6. When a problem occurs in Script, it is said that "Mr. A should fix it (because it is large)"
  7. After a long time of correction work, I get regret that ** "Why did you write this scale in Bash?" **

Based on this reality and the advice of Google teacher (quoted below) We came to the conclusion that "Python 3.x from 2017".

Original: Source "Shell Style Guide" If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date.

Translation If you're writing a script that's> 100 lines or longer, you should probably write it in Python instead. Note that the script grows. In another language to avoid time-consuming fixes later Rewrite the script as soon as possible.

Verification environment

-Debian 8.6 (64bit) ・ Bash (GNU bash 4.3.30) ・ Python (Version 3.5.1, Anaconda 4.0.0)

table of contents

  1. "Execution of external command" and "Connecting external command and pipe"
  2. How to use here documents
  3. How to get user input
  4. How to change the standard output text color
  5. How to check the authority
  6. Check options
  7. How to debug

"Execution of external command" and "Connecting external command and pipe"

In the case of bash, you can execute it by writing the command and option as it is in Script, For Python, execute via the subprocess module.

An example of using an external command (resize command) (Bash, Python) is as follows.


resize -s 30 50       #30 characters per line, 50 lines


#!/usr/bin/env python3
 -*- coding: utf-8 -*-                                             
import subprocess

#The command you want to pass to the subprocess+List options
#State including space(run(["resize -s 30 50"]))Error if described in

An example of connecting an external command and a pipe is as follows. Compared to Bash's "|" notation, there is more content to describe.


#!/usr/bin/env python3                            
import subprocess

#Connect pipes for standard output and standard error by specifying the second and third arguments
result = subprocess.Popen(["resize","-s","30","50"], \
                           stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = result.communicate() #Acquisition of standard output and standard error

print("Standard output:%s" % out.decode('utf-8'))   #If decode processing is not performed, it will be treated as a byte.
print("Standard error:%s" % err.decode('utf-8'))

An example of writing the output result of an external command to a text file is as follows. In addition, the execution result of "python sample.py" is also shown.


#!/usr/bin/env python3
import subprocess

#Opened log file(command.log)Flow until standard output is written to
# "communicate()"of[0]部分は、複数of返り値of中から標準出力を受け取るためof記述
log_file = open("command.log", "w")
result = subprocess.Popen(["resize","-s","30","50"], stdout=subprocess.PIPE)

#If you want to add"a"Open the file by specifying.
log_file = open("command.log", "a")
result = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE)


16 in total
-rw-r--r--1 nao nao 44 December 24 19:15 command.log
-rw-r--r--1 nao nao 543 December 24 15:58 python_bash.txt
-rwxr-xr-x 1 nao nao 673 December 24 19:15 sample.py
-rwxr-xr-x 1 nao nao 77 December 24 15:58 sample.sh

How to use here documents

You will have the opportunity to use here-documents when creating sample files and templates. Below is an example of using Bash and Python here-documents.



#Output the contents of here document.Output to txt.
cat << EOS > output.txt

The scope of here documents is"<< EOS"The line following the line that passed the command(①)From
Then alone"EOS"One line above the line where(②)Until.
The EOS part can be a free character string, but
The same string must be used at the beginning and end of the here document.

$VAR   #When using variables in here-documents.
\$VAR  #When not output as a variable.



#!/usr/bin/env python3

#At the end of the here document[1:-1]If there is no, blank lines are included before and after the string.
heredoc = """
In the Python here document,
Enclose the string in three double quotes.
If you want to expand variables in a here document,
I will describe it as such.
And the variable in the string is the function format()Will be expanded with.
"""[1:-1].format(**locals())  # **Dictionary variables by specification{'VAR':"check"}Receive at.

output = open("output.txt", "w")

How to get user input

The following are three examples of user input acquisition.

Number of characters to enter Echo back Enter(Decision)Need for
Example 1 No limit Yes necessary
Example 2 No limit None necessary
Example 3 One letter None Unnecessary


echo -n "Please enter characters:"  #option"-n"Suppresses line breaks
echo "${USER_INPUT}"

echo -n "Input reception without echo back:"
stty -echo        #Echo back OFF
stty echo         #Echo back ON
echo "${USER_INPUT}"

echo -n "One-letter input reception. No Enter input required:"
read -s -n 1 USER_INPUT  #Bash only
echo "${USER_INPUT}"


#!/usr/bin/env python3
import os
import getch # "pip install getch"It is necessary to install with.

print("Please enter characters:", end="")
user_input = input()

#Besides the following methods, getpass of getpass module()If you use
#Input can be accepted without echo back.
#However, by default on the terminal"Password:"It will be displayed.
print("Input reception without echo back:", end="")
os.system("stty -echo")
user_input = input()
os.system("stty echo")

print("One-letter input reception. No Enter input required:", end="")
user_input = getch.getch()

How to change the standard output text color

Only the text colors for error message (red) and warning message (yellow) are shown below. On the Bash side, it seems that a function that can pass a string with a pipe instead of an argument is easier to use, Since it is a sample, it is in the form of passing arguments.


function error_message (){
    echo -n -e "\033[31m\c"  #Escape sequence to make characters red
    echo "$1"
    echo -n -e "\033[m\c"   #Restore text color
function warning_message (){
    echo -n -e "\033[33m\c"  #Escape sequence to make letters yellow
    echo "$1"
    echo -n -e "\033[m\c"   #Restore text color

error_message "error"
warning_message "warning"


#!/usr/bin/env python3
def error_message(message):
    print("\033[31m%s\033[0m" % message)

def warning_message(message):
    print("\033[33m%s\033[0m" % message)


How to check permissions

The following is an example of when administrator privileges are required when executing Script.



#Check the execution UID and UID,"0"(root)If so, you have administrator privileges.
# ":-"Part${EUID}If there is no value in${UID}Means to substitute the value of
if [ ${EUID:-${UID}} = 0 ]; then
    echo "You have administrator privileges."
    echo "You must have administrator privileges to run this script."


#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os

#Confirm the existence of administrator authority by acquiring UID and EUID.
if os.getuid() == 0 and os.geteuid() == 0:
    print("You have administrator privileges.")
    print("You must have administrator privileges to run this script.")

How to check options

For Bash, use getopts to interpret the options. At this time, if Script is executed using an option that is not defined, An error statement will be displayed automatically. The sample script and execution result are shown below.



#getopts passes an optional character as the first argument,
#That option(Example:"d")Immediately after the option character if takes an argument":"Write(Example:"d:")。
#In the example below"d"When"f"が引数を必要Whenするオプションです。
while getopts d:f:h OPT
    case $OPT in
        d)  DIR_NAME=$OPTARG   #Optional arguments are stored in the variable OPTARG.
        f)  FILE_NAME=$OPTARG
        h)  echo "The usage of the script is xxx."
            exit 0
        *) echo "The usage of the script is xxx."  #When an option that is not specified comes
            exit 1

echo "Directory name:${DIR_NAME}"
echo "file name:${FILE_NAME}"

Execution result.

$ bash sample.sh -f file_name -d directory_name
Directory name: directory_name
File name: file_name

$ bash sample.sh -f 
sample.sh:Options require arguments-- f
The usage of the script is xxx.

$ bash sample.sh -k
sample.sh:Illegal option-- k
The usage of the script is xxx.

For Python3, the argparse module provides a powerful mechanism. You don't need to write a function to output help, and unlike Bash, you can use the long option. The sample script and execution result are shown below.


#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import argparse

#Function add_argument()Add options with.
#Two types of notation at once(Example:"-d"、"--dir")Can be registered.
#type is the type of the optional argument, dest is the optional argument(value)Write the name of the variable that stores.
#For help, the implications of this option(Text to be displayed during help)Will be described.
parser = argparse.ArgumentParser()
parser.add_argument("-d", "--dir", type=str, dest="dir", 
                    help="Write the directory name used in Script")
parser.add_argument("-f","--file", type=str, dest="file",
                    help="Write the file name used in Script")
args = parser.parse_args()

print("Directory name:%s" % args.dir)
print("file name:%s" % args.file)

Execution result.

$ python sample.py -d directory_name -f file_name
Directory name: directory_name
File name: file_name

$ python sample.py -h
usage: sample.py [-h] [-d DIR] [-f FILE]

optional arguments:
  -h, --help            show this help message and exit
  -d DIR, --dir Write the directory name used in the DIR Script
  -f FILE, --file Write the file name used in FILE Script

$ python sample.py -s
usage: sample.py [-h] [-d DIR] [-f FILE]
sample.py: error: unrecognized arguments: -s

How to debug

For Bash, debugging is done in a very primitive way. In general, you're probably practicing script debugging with the following options:

option Description
-u If there is an undefined variable, the process ends.
-v Display the contents of Script in the order of execution.
The variable name is displayed as it is.
-x Display the contents of Script in the order of execution.
Variables can be expanded to display debug information

As a usage, give the above option to Shebang (example: "#! / Bin / bash -x"), Insert "set -x" where you want to start debugging Script and "set + x" where you want to end it. The -x option displays a null command (:), so you can insert comments that are only visible during debugging.

The sample script and execution result are shown below.


#!/bin/bash -x                                                                  
START="Start script"
END="End of script"

echo ${START} #Variables are displayed in an expanded state
set +x        #Canceling options
              :This null command is not displayed
echo "This echo command outputs only the character string part"

set -x      #Option activation
            :This null command is displayed
echo "${END}" 

Execution result.

$ ./sample.sh 
+ START=Start script
+ END=End of script
+start echo script
Start script
+ set +x
This echo command outputs only the character string part
+ :This null command is displayed
+end echo script
End of script

For Python3, debugging is possible with the debugger pdb. To use it, insert "import pdb; pdb.set_trace ()" where you want to debug and The debugger starts when the Script reaches the above insertion position.

Since I am not using pdb myself, I will write an article that I referred to this time (I will add it in the future). ・ "Pdb — Python Debugger" ・ "Python Debugging Tips (@TakesxiSximada)" ・ "Introduction to Efficient Debugging in Python"


It's a very basic content, but because there were "items that took longer to investigate than expected" I will leave it as a memorandum.

In terms of replacing Bash with Python ・ "What is the equivalent of Bash trap (signal detection)?" -"Which Python function corresponds to basic commands such as cd / pwd / mkdir in the first place?" ・ "Is it easier to process files with sed or awk, or should I use Python?" And there is no end to what to look for. At a later date, this content will also be published in Qiita.

Postscript: A story that hurt in Python

Python 3.x instead of Bash. It's been half a year since I came up with that idea. In the last six months, I've had a painful eye with Python only once. The Python 3 I created didn't work.

Migrate Python scripts from the development environment (Debian 8.8) to another verification environment (CentOS 7.x) When I tried to run it on CentOS, the script didn't work. The reason is that CentOS 7.x uses 2.7 by default. Moreover, because it is a verification environment, it is not possible to install a new package (Python3). I've rewritten Python to work from 3.x to 2.7.

From this experience, I gained the knowledge that "after checking the execution environment, write the script". It's important, Runtime.

Recommended Posts

Bash (Shell Script), I hate it anymore: What I researched to replace Bash with Python 3.x
What I did to welcome the Python2 EOL with confidence
I want to use a wildcard that I want to shell with Python remove
What I was addicted to with json.dumps in Python base64 encoding
What to do with PYTHON release?
I want to debug with Python
What skills do I need to program with the FBX SDK Python?
I want to do it with Python lambda Django, but I will stop
I want to tweet on Twitter with Python, but I'm addicted to it
Use Python from Java with Jython. I was also addicted to it.
[Zaif] I tried to make it easy to trade virtual currencies with Python
Wrangle x Python book I tried it [2]
I wanted to solve ABC160 with Python
I want to analyze logs with Python
I want to play with aws with python
What I did with a Python array
Wrangle x Python book I tried it [1]
What I was addicted to Python autorun
I wanted to solve ABC172 with Python
What I did to save Python memory
When I tried to create a virtual environment with Python, it didn't work
I was able to mock AWS-Batch with python, moto, so I will leave it
[Python] I introduced Word2Vec and played with it.
I want to use MATLAB feval with python
I wanted to solve NOMURA Contest 2020 with Python
[Python] What I did to do Unit Test
I want to make a game with Python
I wanted to install Python 3.4.3 with Homebrew + pyenv
I tried to get CloudWatch data with Python
I tried to output LLVM IR with Python
[Improved version] Script to monitor CPU with Python
I want to use Temporary Directory with Python2
I tried to automate sushi making with python
#Unresolved I want to compile gobject-introspection with Python3
I want to solve APG4b with Python (Chapter 2)
What I did when updating from Python 2.6 to 2.7
I want to write to a file with Python
Python script to get note information with REAPER
I tried to find out how to streamline the work flow with Excel x Python ②
I tried to find out how to streamline the work flow with Excel x Python ④
[Introduction to system trading] I drew a Stochastic Oscillator with python and played with it ♬
I want to replace the variables in the python template file and mass-produce it in another file.
I tried to find out how to streamline the work flow with Excel x Python ①
What I did when I was angry to put it in with the enable-shared option
python note: What does it mean to set a seed with random number generation?
I tried to find out how to streamline the work flow with Excel x Python ③
I tried to build an environment for machine learning with Python (Mac OS X)
How to add help to HDA (with Python script bonus)
I want to handle optimization with python and cplex
I tried to implement Minesweeper on terminal with python
I tried to get started with blender python script_Part 01
I tried to touch the CSV file with Python
I tried to draw a route map with Python
I tried to solve the soma cube with python
How to use python interactive mode with git bash
I tried to get started with blender python script_Part 02
I was addicted to scraping with Selenium (+ Python) in 2020
I tried to implement an artificial perceptron with python
I want to inherit to the back with python dataclass
I want to work with a robot in python.
I tried to automatically generate a password with Python3