Python template for log analysis at explosive speed

Summary

During work, I often analyze ** free-format logs output by development machines that were put in by someone other than myself. I write the log analysis code according to it every time, but as I have done log analysis many times, I have created templates and techniques that will improve, so I will develop it.

Template code

#!/usr/bin/env python3

import sys,re

for line in sys.stdin: #Read from standard output
    line = line.strip() #Remove spaces and line breaks at the beginning and end of lines
    print(line)

Only this!

The above code is written in Python3, but Python2 is no different except that the Shebang and print functions become print statements.

Use cases and techniques

Try to analyze and aggregate the rows using the template.

#!/usr/bin/env python3

import sys,re #Handles standard input and regular expressions

#Information that crosses lines is stored in the variable declared here.
hoge_count = 0

#Turn the loop line by line
for line in sys.stdin:
    line = line.strip()
    # print(line) #For debugging, comment out if not needed

    #If you're looking for a line that starts with a particular string, than a regular expression.startwith()Is convenient
    if line.startswith('hoge'):
      hoge_count += 1

    if line.startswith('fuga'):
      #If you want to look ahead to the next line.readline()Call
      next_line = sys.stdin.readline().strip()
      print("next to fuga =", next_line)

    #Spaces and comma delimiters.split()use
    if line.startswith('moge'):
       moge_cols = line.split(' ')
       print("moge line cols =", moge_cols)

    #Use regular expressions for complex matches
    m = re.match('(.*)_muga_(.*)', line)
    if m:
       print("muga line left: ", m.group(1), "right:", m.group(2))

#Output of aggregation result
print("hoge_count =", hoge_count)

Let's eat this input.

input.txt

hoge1
hoge2
fuga
next fuga
moge 1 2 3
left_muga_right

Run

$ python3 analyze.py < input.txt

You can get this input.

output

next to fuga = next fuga
moge line cols = ['moge', '1', '2', '3']
muga line left:  left right: right
hoge_count = 2

Similarly, equivalent code can be written in Python2.

Summary

Introduced code templates and techniques for log analysis in 5 minutes. Written in Python3, but it is also possible in Python2.