If you want to run OpenCV in Python and display the results of analysis and processing, it is easiest to use the cv2.imshow () method.
But if you want to do more than just display, OpenCV windows aren't enough.
One way to solve this problem is to use Pygame, a Python GUI framework [^ Pygame reason for choosing].
In this post, I will write about how to convert OpenCV images to images for Pygame.
[Reason for choosing ^ Pygame]: The reason for Pygame is that it can also be drawn on Raspbian Lite (super important). Most GUI frameworks (including OpenCV windowing) rely on the X Window System and cannot be viewed on Raspbian Lite without a built-in GUI. However, Pygame also has the ability to draw using SDL, so it also works with Raspbian Lite.
#After writing the OS, log in with ssh
#Update package list
sudo apt update
#Install OpenCV for Python 3
sudo apt install python3-opencv
#Install Pygame for Python 3
sudo apt install python3-pygame
#Check Python version
python3 --version
#Check the version of OpenCV
python3 -c 'import cv2; print(cv2.__version__)'
#Check the version of Pygame
python3 -c 'import pygame; print(pygame.version.ver)'
According to "Draw an image created with opencv with pygame. --BlankTar", it is as follows. It seems that it can be converted by the method.
opencv_image = opencv_image[:,:,::-1] #Since OpenCV is BGR and pygame is RGB, it is necessary to convert it.
shape = opencv_image.shape[1::-1] #OpenCV(height,width,Number of colors), Pygame(width, height)So this is also converted.
pygame_image = pygame.image.frombuffer(opencv_image.tostring(), shape, 'RGB')
Let's write the code to convert this way.
show-image.py
import time
import cv2
import pygame
def get_opencv_img_res(opencv_image):
height, width = opencv_image.shape[:2]
return width, height
def convert_opencv_img_to_pygame(opencv_image):
"""
Convert OpenCV images for Pygame.
see https://blanktar.jp/blog/2016/01/pygame-draw-opencv-image.html
"""
opencv_image = opencv_image[:,:,::-1] #Since OpenCV is BGR and pygame is RGB, it is necessary to convert it.
shape = opencv_image.shape[1::-1] #OpenCV(height,width,Number of colors), Pygame(width, height)So this is also converted.
pygame_image = pygame.image.frombuffer(opencv_image.tostring(), shape, 'RGB')
return pygame_image
def main():
#Load images with OpenCV
image_path = '/usr/share/info/gnupg-module-overview.png' #The path of the image file that was originally included in Raspbian Buster Lite
opencv_image = cv2.imread(image_path)
#Initialize Pygame
pygame.init()
width, height = get_opencv_img_res(opencv_image)
screen = pygame.display.set_mode((width, height))
#Convert OpenCV images for Pygame
pygame_image = convert_opencv_img_to_pygame(opencv_image)
#Draw image
screen.blit(pygame_image, (0, 0))
pygame.display.update() #Update screen
#Wait 5 seconds to finish
time.sleep(5)
pygame.quit()
if __name__ == '__main__':
main()
Start command
sudo python3 show-image.py
# Note:You need to add "sudo" and run it with root privileges in order to view it using the SDL library.
#"With desktop" version of Raspbian that is not Lite, and "ssh command"-For example, if you are connecting with the "X" option added,
#No "sudo" command is required as it can be viewed using the X Window System.
# see https://www.subthread.co.jp/blog/20181206/
When you start it, the image will be displayed for 5 seconds and it will end automatically. The conversion works correctly.
Now, let's consider the case of displaying the analyzed video in real time by the above method. Such applications are often used, such as when analyzing and displaying images read from a camera. In this case, the important thing is the conversion speed. Let's measure.
show-image.py
import time
import cv2
import pygame
def get_opencv_img_res(opencv_image):
height, width = opencv_image.shape[:2]
return width, height
def convert_opencv_img_to_pygame(opencv_image):
"""
Convert OpenCV images for Pygame.
see https://blanktar.jp/blog/2016/01/pygame-draw-opencv-image.html
"""
opencv_image = opencv_image[:,:,::-1] #Since OpenCV is BGR and pygame is RGB, it is necessary to convert it.
shape = opencv_image.shape[1::-1] #OpenCV(height,width,Number of colors), Pygame(width, height)So this is also converted.
pygame_image = pygame.image.frombuffer(opencv_image.tostring(), shape, 'RGB')
return pygame_image
def main():
#Load images with OpenCV
image_path = '/usr/share/info/gnupg-module-overview.png' #The path of the image file that was originally included in Raspbian Buster Lite
opencv_image = cv2.imread(image_path)
#Initialize Pygame
pygame.init()
width, height = get_opencv_img_res(opencv_image)
screen = pygame.display.set_mode((width, height))
#Convert OpenCV images for Pygame
time_start = time.perf_counter() #Start measurement
pygame_image = convert_opencv_img_to_pygame(opencv_image)
time_end = time.perf_counter() #End of measurement
print(f'Conversion time: {time_end - time_start}Seconds/ {1/(time_end - time_start)}fps')
#Draw image
screen.blit(pygame_image, (0, 0))
pygame.display.update() #Update screen
#Wait 5 seconds to finish
time.sleep(5)
pygame.quit()
if __name__ == '__main__':
main()
Conversion time: 0.14485926300017127 seconds/ 6.903252020540914fps
**slow! !! !! ** **
0.14 seconds with a code that just displays. If you write 7fps in terms of frame rate, you can see this slowness.
Is this the limit of OpenCV + Pygame? No, it's not. I'm just doing it wrong. The reason why it is so slow is that the image data is purposely converted to a character string by the tostring () method. As you can see by measuring, 90% of this weight is due to the tostring () method.
The actual image data of OpenCV for Python is NumPy's ndarray. Pygame has a function that reads image data from an array of NumPy. How to make full use of this is described in "OpenCV VideoCapture running on PyGame".
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frame = np.rot90(frame)
frame = pygame.surfarray.make_surface(frame)
Also, in Comment section of "OpenCV VideoCapture running on PyGame", the solution to the image inversion problem caused by this method + further speedup Is also described.
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
-frame = np.rot90(frame)
+frame = frame.swapaxes(0,1)
frame = pygame.surfarray.make_surface(frame)
Change to this method and measure again.
show-image.py
import time
import cv2
import pygame
def get_opencv_img_res(opencv_image):
height, width = opencv_image.shape[:2]
return width, height
def convert_opencv_img_to_pygame(opencv_image):
"""
Convert OpenCV images for Pygame.
see https://gist.github.com/radames/1e7c794842755683162b
"""
rgb_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2RGB).swapaxes(0, 1)
#Generate a Surface for drawing images with Pygame based on OpenCV images
pygame_image = pygame.surfarray.make_surface(rgb_image)
return pygame_image
def main():
#Load images with OpenCV
image_path = '/usr/share/info/gnupg-module-overview.png' #The path of the image file that was originally included in Raspbian Buster Lite
opencv_image = cv2.imread(image_path)
#Initialize Pygame
pygame.init()
width, height = get_opencv_img_res(opencv_image)
screen = pygame.display.set_mode((width, height))
#Convert OpenCV images for Pygame
time_start = time.perf_counter() #Start measurement
pygame_image = convert_opencv_img_to_pygame(opencv_image)
time_end = time.perf_counter() #End of measurement
print(f'Conversion time: {time_end - time_start}Seconds/ {1/(time_end - time_start)}fps')
#Draw image
screen.blit(pygame_image, (0, 0))
pygame.display.update() #Update screen
#Wait 5 seconds to finish
time.sleep(5)
pygame.quit()
if __name__ == '__main__':
main()
Conversion time: 0.030075492999912967 seconds/ 33.24966277370395fps
It has improved a lot.
Now that the speed issue has been improved, let's view the result of doing something with OpenCV. For example, suppose you want to display the binarized result.
show-image.py
import time
import cv2
import pygame
def get_opencv_img_res(opencv_image):
height, width = opencv_image.shape[:2]
return width, height
def convert_opencv_img_to_pygame(opencv_image):
"""
Convert OpenCV images for Pygame.
see https://gist.github.com/radames/1e7c794842755683162b
"""
rgb_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2RGB).swapaxes(0, 1)
#Generate a Surface for drawing images with Pygame based on OpenCV images
pygame_image = pygame.surfarray.make_surface(rgb_image)
return pygame_image
def main():
#Load images with OpenCV
image_path = '/usr/share/info/gnupg-module-overview.png' #The path of the image file that was originally included in Raspbian Buster Lite
opencv_image = cv2.imread(image_path)
#Process image
opencv_gray_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2GRAY)
ret, opencv_threshold_image = cv2.threshold(opencv_gray_image, 128, 255, cv2.THRESH_BINARY)
opencv_image = opencv_threshold_image
#Initialize Pygame
pygame.init()
width, height = get_opencv_img_res(opencv_image)
screen = pygame.display.set_mode((width, height))
#Convert OpenCV images for Pygame
pygame_image = convert_opencv_img_to_pygame(opencv_image)
#Draw image
screen.blit(pygame_image, (0, 0))
pygame.display.update() #Update screen
#Wait 5 seconds to finish
time.sleep(5)
pygame.quit()
if __name__ == '__main__':
main()
However, the execution fails with the following error:
OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /build/opencv-L65chJ/opencv-3.2.0+dfsg/modules/imgproc/src/color.cpp, line 9716
Traceback (most recent call last):
File "show-image.py", line 51, in <module>
main()
File "show-image.py", line 39, in main
pygame_image = convert_opencv_img_to_pygame(opencv_image)
File "show-image.py", line 17, in convert_opencv_img_to_pygame
rgb_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2RGB).swapaxes(0, 1)
cv2.error: /build/opencv-L65chJ/opencv-3.2.0+dfsg/modules/imgproc/src/color.cpp:9716: error: (-215) scn == 3 || scn == 4 in function cvtColor
The binarized image is a grayscale image with no color components. However, in the process of converting to an image for Pygame, there is a code that converts the color of the input image from BGR to RGB. Therefore, an error will occur in this part.
To solve this problem, for grayscale images, a branch process is required to change the second argument of the cv2.cvtColor () method to cv2.COLOR_GRAY2RGB.
show-image.py
import time
import cv2
import pygame
def get_opencv_img_res(opencv_image):
height, width = opencv_image.shape[:2]
return width, height
def convert_opencv_img_to_pygame(opencv_image):
"""
Convert OpenCV images for Pygame.
see https://gist.github.com/radames/1e7c794842755683162b
see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
"""
if len(opencv_image.shape) == 2:
#For grayscale images
cvt_code = cv2.COLOR_GRAY2RGB
else:
#In other cases:
cvt_code = cv2.COLOR_BGR2RGB
rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)
#Generate a Surface for drawing images with Pygame based on OpenCV images
pygame_image = pygame.surfarray.make_surface(rgb_image)
return pygame_image
def main():
#Load images with OpenCV
image_path = '/usr/share/info/gnupg-module-overview.png' #The path of the image file that was originally included in Raspbian Buster Lite
opencv_image = cv2.imread(image_path)
#Process image
opencv_gray_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2GRAY)
ret, opencv_threshold_image = cv2.threshold(opencv_gray_image, 128, 255, cv2.THRESH_BINARY)
opencv_image = opencv_threshold_image
#Initialize Pygame
pygame.init()
width, height = get_opencv_img_res(opencv_image)
screen = pygame.display.set_mode((width, height))
#Convert OpenCV images for Pygame
pygame_image = convert_opencv_img_to_pygame(opencv_image)
#Draw image
screen.blit(pygame_image, (0, 0))
pygame.display.update() #Update screen
#Wait 5 seconds to finish
time.sleep(5)
pygame.quit()
if __name__ == '__main__':
main()
You should now be able to see a binarized grayscale image as well.
First, let's measure the traditional way of using the pygame.surfarray.make_surface () method for each conversion.
show-image.py
import statistics
import time
import cv2
import pygame
def get_opencv_img_res(opencv_image):
height, width = opencv_image.shape[:2]
return width, height
def convert_opencv_img_to_pygame(opencv_image):
"""
Convert OpenCV images for Pygame.
see https://gist.github.com/radames/1e7c794842755683162b
see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
"""
if len(opencv_image.shape) == 2:
#For grayscale images
cvt_code = cv2.COLOR_GRAY2RGB
else:
#In other cases:
cvt_code = cv2.COLOR_BGR2RGB
rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)
#Generate a Surface for drawing images with Pygame based on OpenCV images
pygame_image = pygame.surfarray.make_surface(rgb_image)
return pygame_image
def main():
#Load images with OpenCV
image_path = '/usr/share/info/gnupg-module-overview.png' #The path of the image file that was originally included in Raspbian Buster Lite
opencv_image = cv2.imread(image_path)
#Initialize Pygame
pygame.init()
width, height = get_opencv_img_res(opencv_image)
screen = pygame.display.set_mode((width, height))
#Convert OpenCV images for Pygame
time_diff_list = []
for _ in range(500):
time_start = time.perf_counter() #Start measurement
pygame_image = convert_opencv_img_to_pygame(opencv_image)
time_end = time.perf_counter() #End of measurement
time_diff_list.append(time_end - time_start)
time_diff = statistics.mean(time_diff_list)
print(f'Average conversion time: {time_diff}Seconds/ {1/time_diff}fps')
#Draw image
screen.blit(pygame_image, (0, 0))
pygame.display.update() #Update screen
#Wait 1 second and finish
time.sleep(1)
pygame.quit()
if __name__ == '__main__':
main()
Average conversion time: 0.01808052065800075 seconds/ 55.30814177950641fps
It is as fast as approaching 60fps. This is still fast enough, but it's a bit annoying when you think you're just converting.
Next, let's measure the speedup method using the pygame.surfarray.blit_array () method.
show-image.py
import statistics
import time
import cv2
import pygame
def get_opencv_img_res(opencv_image):
height, width = opencv_image.shape[:2]
return width, height
pygame_surface_cache = {}
def convert_opencv_img_to_pygame(opencv_image):
"""
Convert OpenCV images for Pygame.
see https://gist.github.com/radames/1e7c794842755683162b
see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
see https://stackoverflow.com/a/42589544/4907315
"""
if len(opencv_image.shape) == 2:
#For grayscale images
cvt_code = cv2.COLOR_GRAY2RGB
else:
#In other cases:
cvt_code = cv2.COLOR_BGR2RGB
rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)
#Get a generated Surface with the same image size from the cache
cache_key = rgb_image.shape
cached_surface = pygame_surface_cache.get(cache_key)
if cached_surface is None:
#Generate a Surface for drawing images with Pygame based on OpenCV images
cached_surface = pygame.surfarray.make_surface(rgb_image)
#Add Surface to cache
pygame_surface_cache[cache_key] = cached_surface
else:
#If you find a Surface with the same image size, reuse the already generated Surface.
pygame.surfarray.blit_array(cached_surface, rgb_image)
return cached_surface
def main():
#Load images with OpenCV
image_path = '/usr/share/info/gnupg-module-overview.png' #The path of the image file that was originally included in Raspbian Buster Lite
opencv_image = cv2.imread(image_path)
#Initialize Pygame
pygame.init()
width, height = get_opencv_img_res(opencv_image)
screen = pygame.display.set_mode((width, height))
#Convert OpenCV images for Pygame
time_diff_list = []
for _ in range(500):
time_start = time.perf_counter() #Start measurement
pygame_image = convert_opencv_img_to_pygame(opencv_image)
time_end = time.perf_counter() #End of measurement
time_diff_list.append(time_end - time_start)
time_diff = statistics.mean(time_diff_list)
print(f'Average conversion time: {time_diff}Seconds/ {1/time_diff}fps')
#Draw image
screen.blit(pygame_image, (0, 0))
pygame.display.update() #Update screen
#Wait 1 second and finish
time.sleep(1)
pygame.quit()
if __name__ == '__main__':
main()
Average conversion time: 0.013679669161995207 seconds/ 73.1011830884182fps
It has exceeded 60fps! If it is so fast, it will be practical enough.
def convert_opencv_img_to_pygame(opencv_image):
"""
Convert OpenCV images for Pygame.
see https://gist.github.com/radames/1e7c794842755683162b
see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
"""
if len(opencv_image.shape) == 2:
#For grayscale images
cvt_code = cv2.COLOR_GRAY2RGB
else:
#In other cases:
cvt_code = cv2.COLOR_BGR2RGB
rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)
#Generate a Surface for drawing images with Pygame based on OpenCV images
pygame_image = pygame.surfarray.make_surface(rgb_image)
return pygame_image
If you want to convert images of the same size many times:
pygame_surface_cache = {}
def convert_opencv_img_to_pygame(opencv_image):
"""
Convert OpenCV images for Pygame.
see https://gist.github.com/radames/1e7c794842755683162b
see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
see https://stackoverflow.com/a/42589544/4907315
"""
if len(opencv_image.shape) == 2:
#For grayscale images
cvt_code = cv2.COLOR_GRAY2RGB
else:
#In other cases:
cvt_code = cv2.COLOR_BGR2RGB
rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)
#Get a generated Surface with the same image size from the cache
cache_key = rgb_image.shape
cached_surface = pygame_surface_cache.get(cache_key)
if cached_surface is None:
#Generate a Surface for drawing images with Pygame based on OpenCV images
cached_surface = pygame.surfarray.make_surface(rgb_image)
#Add Surface to cache
pygame_surface_cache[cache_key] = cached_surface
else:
#If you find a Surface with the same image size, reuse the already generated Surface.
pygame.surfarray.blit_array(cached_surface, rgb_image)
return cached_surface
Recommended Posts