Output timing is incorrect when standard (error) output is converted to a file in Python

Introduction

Is it just my environment?

In a Python script, when the standard (error) output is converted to a file or executed from another Python script, the output timing is incorrect. It comes out at once when the process ends, not when print or sys.stdout.write is executed.

Write to a file

Script with strange output timing

This is the executed Python script. Just give a number every second.

child1.py


from time import sleep

for i in range(10):
    print(str(i))
    sleep(1)

When this is executed with the following command, nothing is output and a log is output at once after 10 seconds have passed.

$  python3 ./child1.py > stdout.log & tail -f stdout.log

Modified version

Next is the modified version. Try running sys.stdout.flush after print.

child2.py


from time import sleep
import sys

for i in range(10):
    print(str(i))
    sys.stdout.flush()
    sleep(1)

Similarly, if you execute the following command, it will be output every second.

$  python3 ./child2.py > stdout.log & tail -f stdout.log

By the way,

$  python3 ./child1.py

And, there is no problem in outputting to the screen.

Call from another Python script

As the parent script, I used the one posted in here.

parent.py


import sys
import subprocess

def get_lines(cmd):
    '''
    :param cmd:str command to execute.
    :rtype: generator
    :return:Standard output(Line by line).
    '''
    proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    while True:
        line = proc.stdout.readline()
        if line:
            yield line.decode('utf-8')

        if not line and proc.poll() is not None:
            break

if __name__ == '__main__':
    for line in get_lines(cmd='python3 ./child1.py'): # ./child2.When I run py, it is output every second
        sys.stdout.write(line)

With this, even when outputting to the screen, child1.py is output at once after 10 seconds, and child2.py is output every second.

Conclusion

Is it such a thing? → Such a thing.

(Addition) As pointed out in the comment below, it was confirmed that if you add the -u option when executing the python3 command, it will be output at any time. I've become smarter again. \ # I'll try to write the "I couldn't do it" system.

Recommended Posts

Output timing is incorrect when standard (error) output is converted to a file in Python
Change the standard output destination to a file in Python
Write standard output to a file
Determine if standard output is piped when running a Python script
How to create a JSON file in Python
How to output "Ketsumaimo" as standard output in Python
Error when trying to install psycopg2 in Python
[Python] What to check when you get a Unicode Decode Error in Django
Parse a JSON string written to a file in Python
A memorandum to run a python script in a bat file
I want to randomly sample a file in Python
Things to note when initializing a list in Python
Use communicate () when receiving output in a Python subprocess
Output a binary dump in binary and revert to a binary file
When publishing a package to PyPI, "HTTP Error: 400 Client Error: File already exists." Is displayed.
How to import a file anywhere you like in Python
A standard way to develop and distribute packages in Python
[Python] How to output a pandas table to an excel file
Use Heroku in python to notify Slack when a specific word is muttered on Twitter
When a character string of a certain series is in the Key of the dictionary, the character string is converted to the Value of the dictionary.
Output to csv file with Python
Make standard output non-blocking in Python
Create a binary file in Python
When writing a program in Python
Output the key list included in S3 Bucket to a file
Convert to a string while outputting standard output with Python subprocess
I tried to convert a Python file to EXE (Recursion error supported)
[Introduction to Python] How to output a character string in a Print statement
[Python] Execution time when a function is entered in a dictionary value
Read the standard output of a subprocess line by line in Python
Timezone specification when converting a string to datetime type in python
[Python] When you want to use all variables in another file
"Value Error: Unable to configure handler'file_output_handler'" when starting a python program
Python script written in PyTorch is converted to exe with PyInstaller
What to do when the value type is ambiguous in Python?
Automatically convert to py file when ui file is updated in PySide
Write a script in Shell and Python to notify you in Slack when the process is finished
What to do if there is a decimal in python json .dumps
Create a shortcut to run a Python file in VScode on your terminal
Precautions when pickling a function in python
Convert psd file to png in Python
[Python] How to put any number of standard inputs in a list
How to get a stacktrace in python
I want to convert a table converted to PDF in Python back to CSV
Hash in Perl is a dictionary in Python
[Python] What to do when an error related to SSL authentication is returned
How to write a string when there are multiple lines in python
[GPS] Create a kml file in Python
I made a program to check the size of a file in Python
Error due to UnicodeDecodeError when reading CSV file with Python [For beginners]
How to use is and == in Python
What is the fastest way to create a reverse dictionary in python?
How to input a character string in Python and output it as it is or in the opposite direction.
What to do when a warning message is displayed in pip list
What to do if you get the error RuntimeError: Python is not installed as a framework when trying to use matplitlib and pylab in Python 3.3
I created a script to check if English is entered in the specified position of the JSON file in Python.
Various ways to read the last line of a csv file in Python
[Python] I want to know the variables in the function when an error occurs!
[python] What to do when an error occurs in send_keys of headless chrome
How to drop Google Docs in one folder in a .txt file with python
Use libsixel to output Sixel in Python and output a Matplotlib graph to the terminal.