(Python: OpenCV) I tried to output a value indicating the distance between regions while binarizing the video in real time.

Introduction

Even though I work in the non-IT department of the manufacturing industry, I have come to work with an awareness of AI and IoT. I work closer to production engineering jobs that are close to the manufacturing site, but one of the challenges in systematization is that it is not balanced with the profits to the factory. There is something I want to do, but if I estimate it, the introduction cost will be high (labor costs are often high ...) and I may give up. Therefore, the following content is what I am thinking that it is good to make it from the viewpoint of skill improvement.

1. Create a program for image processing (image ver.)
2. ** This time: Create a program for image processing (video ver.) **
3. Build environment + store program on Raspberry Pi
4. Check if the captured image can be processed in real time
5. Search for good processing methods that will lead to factory improvement
6. Improve KPI values and achieve results

We aim to process the recorded video used at the manufacturing site in real time (binarization, calculation / display of a certain distance, etc.) and display the result. Surprisingly, it can help on-site workers to easily manufacture even simple processed video images.

Well, it's been a long time, but the outline of this time is as follows.

• Display the value on the video display screen
• Screen display Adjust the timing when the screen value appears

The previous article is here.

Binarize the image. Furthermore, the shortest distance between two regions can be calculated (Ver1.1). https://qiita.com/Fumio-eisan/items/10c54af7a925b403f59f

Simply display the video

First, perform the process of displaying the video. This time, we will process the video with the following capture for about 7 seconds.

• The actual video is stored on github listed at the bottom.

video.ipynb

#Simply display
import cv2
import sys

file_path = 'sample_.mov'
delay = 1
window_name = 'frame'

cap = cv2.VideoCapture(file_path)
text = 'text.wmv'

if not cap.isOpened():
    sys.exit()

while True:
    ret, frame = cap.read()
   # if not ret: #If you insert these two lines, the video will be played once.
   #     break
    if ret:
        frame = cv2.resize(frame, dsize=(600, 400))
        cv2.imshow(window_name, frame)
        if cv2.waitKey(delay) & 0xFF == ord('q'):
            break
    else:
        cap.set(cv2.CAP_PROP_POS_FRAMES, 0)

cv2.destroyWindow(window_name)

It is OK if you describe the path of the video you want to display in file_path. In the case of this program, the video will continue to play in an infinite loop. You can stop the video playback by pressing the q key.

Regarding the display of the video, we are shooting according to the following procedure.

1. Load the video with the cv2.VideoCapture () method
2. Play with the syntax in while
3. Continue to display every frame (30 frames in 1 second if 30fps)
4. (Depending on the process) Infinite loop or end when playback is completed once, etc.

It has become. I will write a note later, but since it is processed every frame in 3., you need to be careful if you want to change the telop (value to be displayed, etc.) that you want to display every second or every few seconds.

I want to add subtitles

video.ipynb

cv2.putText(frame, text,(100, 30), cv2.FONT_HERSHEY_SIMPLEX, 1.0, (0, 255, 0), thickness=2)

Subtitle output is possible with this display. The arguments are as follows. 1st argument: video frame, 2nd argument: text to be displayed, 3rd argument: x, y position, 4th argument: font, 5th argument: character size, 6th argument: BGR color information, 7th argument : Character thickness

I want to binarize

Now, let's binarize it and display it in black and white. The method is OK if you define the upper and lower limits of the color to be extracted like the image and define it with the cv2.inRange () method.

video.ipynb

#Binarization process
import cv2
import sys

camera_id = 0
delay = 1
window_name = 'frame'
file_path = 'sample_.mov'

cap = cv2.VideoCapture(file_path)

import numpy as np
bgrLower = np.array([0, 100, 100])    #Lower limit of colors to extract(BGR)
bgrUpper = np.array([250,250, 250])

if not cap.isOpened():
    sys.exit()

while True:
    ret, frame = cap.read()
    if not ret:
        break
    frame = cv2.resize(frame, dsize=(600, 400))

    img_mask = cv2.inRange(frame, bgrLower, bgrUpper) 
    contours, hierarchy = cv2.findContours(img_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    contours.sort(key=lambda x: cv2.contourArea(x), reverse=True)
    #target_contour = max(contours, key=lambda x: cv2.contourArea(x))
    #img_mask = cv2.line(img_mask, (250,300), (350,300), (120,120,120), 10) #The second argument is the start point, the third argument is the end point, the fourth argument is the color, and the fifth argument is the line thickness.
    #img_mask=cv2.drawContours(img_mask,contours[0:2],-1,(120,120,120),5)

    cv2.imshow(window_name, img_mask)
    #cv2.imshow(window_name,img_mask, [contours[0],contours[1]])

    if cv2.waitKey(delay) & 0xFF == ord('q'):
        break

cv2.destroyWindow(window_name)

I was able to successfully binarize.

I want to output the distance between the enclosed areas

Well, this is the main subject. Define between the enclosed areas as shown below and find the distance between them. And I would like to output that value as appropriate.

Find the distance between two points

video.ipynb

    contours, hierarchy = cv2.findContours(img_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)#Boundary drawing
    contours.sort(key=lambda x: cv2.contourArea(x), reverse=True)#Sort boundaries

Enclose the white area with the cv2.findContours () method. The return value is stored in contours. Then sort the contours by area.

Get the coordinates of two regions

video.ipynb

    x1=np.unravel_index(np.argmax(contours[0],axis=0), contours[0].shape)
    x2=np.unravel_index(np.argmax(contours[1],axis=0), contours[0].shape)
    
    img_mask = cv2.line(img_mask, tuple(x1[0][0]), tuple(x2[0][0]), (120,120,120), 3)

Gets the coordinates surrounding the area. Returns x1, x2 with argmax, which takes the maximum value of the coordinates in contours [0], [1](the value where either x or y is the maximum). In the case of argmax, it is flattened to one dimension (determined in one dimension regardless of x and y coordinates), so the unravel_index () method returns the index as coordinates.

Then, actually insert the coordinates with the cv2.line () method to connect the coordinates.

(Supplement) Understand the numerical values stored in contours

• contours [n]: Contains an array of coordinates that form the nth region
• contours [n] [m]: Stores the array of the mth coordinate in the nth area
• contours [n] [m] [0]: stores the mth array in the nth area
• contours [n] [m] [0]: The mth x coordinate in the nth area is displayed.
• contours [n] [m] [1]: The mth y coordinate in the nth area is displayed.

It's complicated like this.

I want to display the distance value on the screen

Now, let's display this calculated value on the screen. Normally, it can be displayed by the cv2.putText () method. However, if it is left as it is, the value for each frame will be calculated and displayed. This will make the value flickering and hard to see. As a countermeasure, you can update the value every certain number of frames and display it. The following is done so that the value is updated in 30 frames (≈ about every second this time) using the if syntax.

video.ipynb

    if idx % 30 == 0:
        text =str(math.floor(np.linalg.norm(x1[0][0]-x2[0][0])))
    cv2.putText(img_mask, text,(300, 100), cv2.FONT_HERSHEY_SIMPLEX, 1.0, (255, 255, 255), thickness=3)
    

Actual result

Here is the result of processing and outputting in this way.

Originally, I wish I could give a value to white or black for each of the two areas of white and black, but I couldn't adjust it well. In addition, a line was drawn between the areas of the adapter, which is the white part (this time it is one thing, but it is divided into two areas), and it was also possible to output a numerical value indicating the distance.

At the end

Well, this time I made a program to play while processing the video. The point was that it was not possible to perform heavy processing because it was processed frame by frame. Originally, I wanted to find the shortest distance between regions and subtract that distance, but I couldn't implement it that much. Also, since there were many articles in which OpenCV itself was summarized in c ++, I had a hard time searching. ..

The program is stored below. https://github.com/Fumio-eisan/movie_20200406