Create your own virtual camera with Python + OpenCV and apply original effects

A virtual camera is software that can be recognized by a PC as an image input device such as a Web camera. By using this, you can stream your favorite video (not the input from the front camera) to the video conferencing systems </ b> such as Zoom, WebEx, and Meet, which are often used these days.

A commonly used virtual camera is OBS Studio, which I often use (it seems that the virtual camera has recently become available as a standard without plugins. is not it). However, I don't want to use it on a daily basis because it is extremely heavy because it has a lot of functions. (The GPU usage rate is about 80% with RTX2028 SUPER, but is it just me ...?)

Therefore, I thought that I could create a virtual camera myself and apply effects, so I tried various trials and errors. There is still a bug of unknown cause, but I would like to introduce it because it has started to work.

Premise

This system is intended for Windows. </ b> Due to the handling of low layers of graphics, the method differs greatly in other OS such as Mac and Linux. During the investigation, I saw a tool that could be used on another OS, so I would like to introduce it, but please do not trust it because it has not been verified.

This configuration

First of all, I would like to conclude that it was difficult to make a complete virtual camera by myself, so I realized it by sending the input of my own software to the virtual camera installed in OBS </ b> mentioned earlier. Did. The configuration diagram is as follows. 構成図.png It seems that you need to study low-level graphics-related APIs such as Direct Show API to make a full-fledged virtual camera. I found it interesting, so I'll try it next time.

Implementation

Get input and display every frame

First of all, as a general story, we will use OpenCV to acquire an image for each frame from the data source and implement the part to be displayed. Since it is a very general part, each process is only written in the code.

import cv2

#Select the data source (Webcam) to acquire
cap = cv2.VideoCapture(0)

while True:
    #Get the image of each frame
    ret, frame = cap.read()

    #Apply some effect here

    #Display image
    #I want to display to a virtual webcam
    cv2.imshow("Window Name", frame)

    #Display 30 frames and end the display when the Enter key is pressed.
    if cv2.waitkey(30) == 13:
        break

#End processing
cv2.destroyAllWindows()
cap.release()

This completes the implementation of the part that displays the webcam image in a window. Next, consider inputting this display to the OBS virtual camera.

Send information to a virtual camera

For Windows

I will use a package called pyvirtualcam. As you can see in the GitHub repository, if you have pip in your environment, you can install it with pip install pyvirtualcam (although the current repository has bugs </ b> and you rebuild the source If you don't, bugs may occur. The countermeasures for this are listed at the end of this article.).

This is also written on the repository page of ↑, but if you already have OBS (Virtual Camera), you do not need to do it, but you need to enable OBS virtual webcam. There are two methods,

  1. Install OBS (it will be installed by itself)
  2. Download only the Virtual Camera part from https://github.com/CatxFish/obs-virtual-cam/releases and in the repository egsvr32 / n / i: 1" obs-virtualcam \ bin \ 64bit \ obs-virtualsource Type .dll " to activate

Method 1 but I haven't confirmed it. In recent OBS, Virtual Camera is included as standard, so I think that this is all right, but if you get any error, please try method 2.

After importing the package, replace the above code as follows.

import cv2
import pyvirtualcam

#Select the data source (Webcam) to acquire
cap = cv2.VideoCapture(0)

#Get image size from first frame
ret, frame = cap.read()
with pyvirtualcam.Camera(width=frame.shape[1], height=frame.shape[0], fps=30, delay=0) as cam:
    while True:
        #Get the image of each frame
        ret, frame = cap.read()

        #Apply some effect here

        #Change color space
        #Enable alpha channel and order in RGB
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA)

        #Stream images to virtual camera
        cam.send(frame)

        #Since the image is no longer displayed on the screen, use the function of pyvirtualcam to wait until the next frame.
        cam.sleep_until_next_frame()

#End processing
cap.release()

Now you can stream the image to the OBS virtual camera. Since there is no way to end by key input, please end with Ctrl + C etc.

After that, if you specify the webcam to be used on the web conferencing system side as OBS-Camera, the image of the webcam acquired by this program should be displayed. In this state, the webcam image is just dripping, so we will apply the effect from here.

For Mac and Linux (unconfirmed)

For Mac

There was OBS Virtual Camera Mac Plugin, so if you use this, you may be able to do it in the same way as the Windows method .... Hmm (it was written in the repository written in the Windows method)

For Linux

Linux has a library called v4l2loopback that allows you to easily create a virtual camera. There are many python wrappers like pyfakewebcam that use this, so I think it's much easier to implement than Windows if you use them. I have not confirmed it, but if you want to use pyfakewebcam, you should change the above code as follows.

import cv2
import pyfakewebcam

#Select the data source (Webcam) to acquire
cap = cv2.VideoCapture(0)

#Get image size from first frame
ret, frame = cap.read()
camera = pyfakewebcam.FakeWebcam('/dev/video1', frame.shape[1], frame.shape[0])
while True:
    #Get the image of each frame
    ret, frame = cap.read()

    #Apply some effect here

    #Change color space
    #In RGB order
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

    #Stream images to virtual camera
    camera.schedule_frame(frame)

    #The image is no longer displayed on the screen, so wait until the next frame (30fps)
    time.sleep(0.033)

#End processing
cap.release()

in conclusion

I introduced how to make your own virtual webcam using Python and OpenCV. I didn't mention the effects here, but you can apply your favorite effects using each function of OpenCV. Finally, as a sample effect, I would like to introduce the process of removing the background of the green background, the bug of pyvirtualcam mentioned above, and the solution.

appendix

Effect example: Without background of green background

In the sample code so far, add it below the part where # Apply some effect here.

frame = cv.cvtColor(frame, cv.COLOR_BGR2HSV)
#The second and third variables are the minimum and maximum values of the HSV color space, which can be changed as appropriate in each environment.
mask = cv.inRange(frame, (48, 158, 32), (88, 255, 255))
mask_inv = cv.bitwise_not(mask)
bg = np.zeros(frame.shape, dtype=np.uint8)
#Set the color you want as the background
bg[:] = [255, 255, 255]
front_result = cv.bitwise_and(frame, frame, mask=mask_in)
back_result = cv.bitwise_and(bg, bg, mask=mask)
frame = cv.add(front_result, back_result)

Fix pyvirtualcam

This was also built in Issue, but it seems that the termination process is not described. You can build and install by cloning the repository yourself, fixing the corrections, and executing setup.py. The modified part is pyvirtualcam / pyvirtualcam / native_windows / main.cpp, and the following function is added.

void stop(){
    virtual_output_stop();
}

I think that the function is called correctly and the termination process is called correctly .... However, for some reason, OBS-Camera is sometimes blocked, and there is still a bug that cannot be fixed without restarting, and we are investigating the cause.

Recommended Posts