Overview

There is a camera called an event camera. I would like to convert a normal video like this event camera. I wonder if it can be used for something.

Deliverables

Event camera-like image pic.twitter.com/MFAUwzrdJt
& mdash; itdk (@ itdk1996) January 3, 2021

Any video was fine, but with a fixed camera, it can be taken beautifully. This time I tried a shot of Tiger Woods from the data of GolfDB

What is an event camera?

Event cameras have a fundamentally different mechanism than ordinary cameras. A typical camera has a dimension of H × W × C from any C of height H, width W, and RGB. If it is a video, it will be (T x H x W x C).

On the other hand, the event camera only records events. Here, the event is "change in brightness value". In other words, only the "points" whose brightness value has changed by the threshold value or more in the time direction are recorded. It is T × (X, Y, P) to record. P is 1 when the brightness value increases and -1 when the brightness value decreases. The feature is that the place where there is no change is not recorded.

Event camera features

By the way, this camera that records only events has the following features.

・ Ultra high frame rate ·Low power consumption ・ Asynchronous

While ordinary cameras have around 60 FPS, event cameras will exceed 1000 FPS. With RGB cameras, the higher the FPS, the less light that enters the camera and the darker it becomes, but with event cameras, the amount of incoming light changes or does not change, so you can maintain a high FPS. .. Low power because there is little data to record. Also, with a normal camera, there is data at a constant pace of 1F at 0.03 seconds and 2F at 0.06 seconds, but in the case of an event camera, there is data. There is a change at (100,200) at 0.0012 seconds There is a change at (250, 50) at 0.0581 seconds It will be recorded only at the time when there is a change.

Since it has low power consumption and high time resolution, it can be applied to automatic driving.

Main subject

It's a simple code, but I'll try something like an event camera. I will leave only the frames that have changed in the video.

In reality, the essential data structure of an event camera is to "acquire only the points where there is a difference". You can easily convert to that format using np.where Event cameras are generally a point cloud data format. Therefore, points are often aggregated and visualized in the time direction, so I will try to make only the visualized image from the image.

code

This time, the difference is taken in each channel one image before, and if it is above the threshold value, the pixel is left as an event.

import numpy as np
import cv2

def img2event_image(base_img, img, th = 10, plus = 128, minus = 255):
    img3 = base_img.astype(float) - img.astype(float)

    index1 = img3 > th
    img3[index1] = plus

    index2 = img3 < -1 * th
    img3[index2] = minus

    img3[~(index1) & ~(index2)] = 0 
    return img3.astype(np.uint8)

def video2event_video():
    stack = True
    path = "96.mp4"
    # output_name = "output.mp4"
    output_name = "output_stack.mp4"

    cap = cv2.VideoCapture(path)
    fourcc = cv2.VideoWriter_fourcc(*"mp4v")
    w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    if stack:
        w *= 2
    fps = cap.get(cv2.CAP_PROP_FPS)
    fps = 30
    out = cv2.VideoWriter(output_name, fourcc, fps, (w, h)) 

    success, image = cap.read()
    # image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    base_image = image[:]
    while success:
        b_img = img2event_image(base_image.copy(), image.copy())
        if stack:
            b_img = cv2.hconcat([image, b_img])

        out.write(b_img)
        base_image = image[:]
        success, image = cap.read()
        # image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    cap.release()
    cv2.destroyAllWindows()

reference

Event camera DAVIS346 https://nanoxeed.co.jp/product/eventcamera/

Event camera story https://qiita.com/minomonter/items/6d71029f6c860da60740

[PYTHON] I tried image processing like an event camera