This article cuts out only the Code part handled in the article serialized in Note. If you are interested in the technical background, please refer to the Note article. This time is a continuation of Last time. Last time, we were able to generate a Gain Map that corrects Dark shading. Noise removal and Edge enhancement processing will be performed using the image after Dark shading correction using this Gain map. Most of the processing is implemented in OpenCV, so we use them to build functions.
The general flow of the image correction process to be performed this time is briefly shown. Processing is done in this order.
The RAW image is a very grainy image due to variations in the characteristics of each pixel. For example, even if you take a picture of a flat white wall, if you zoom in and look closely, it will not look flat due to the effect of variation. Therefore, noise removal is performed to remove this pixel characteristic variation. The ideal noise removal removes only noise, but that is not the case. This is because the pixel characteristic variation is a random variation, so it is not easy to determine from what level it is called Noise. Therefore, the part that seems to be Noise is extracted using the information of the adjacent pixel, and the value of the Noise part is predicted from the characteristics of the adjacent pixel. This time, we will remove noise using the commonly used method called Bilateral. The image input to the function is standardized with a signal level of 0.0 to 1.0 Float in consideration of other image processing. The reason is to unify the input format because it may be 8bit or 10bit depending on the image format. I think it's better to use generics, but I quit because I'm studying. The bilateral filter function implemented in OpenCV accepts only 8-bit RGB-3ch images as input images, so it is temporarily converted to 8-bit images internally.
dualpd.py
def _bilateral_filter(self, img: np.ndarray, kernel_size: int=3, disp: int = 75) -> np.ndarray:
""" apply bilateral filter in order to reduce random noise
Parameters
-----------
img: np.ndarray
target image
kernel_size: int
filter kernal size, default size is 3x3
disp: int
dispartion parameter, default value is 75
Returns
-----------
cor_img: np.ndarray
image after bilateral filtering
"""
# convert data from float to 8bit data
img_8_bit = img * 255
img_8_bit_int = img_8_bit.astype(np.uint8)
cor_img = cv2.bilateralFilter(img_8_bit_int, kernel_size, disp, disp)
# return
return cor_img
Edge is blurred in the image after noise removal due to the influence of Filter. So we need to fix this Edge bokeh. There are various ways to emphasize Edge, but Edge is emphasized using the Unsharp mask, which is often mentioned in textbooks and is also used inside the actual camera. The image before noise removal is org_img and the image after noise removal is target_img. Gaussian Blur filter is applied to the original image, and how much weight is applied to the Edge part is calculated by comparing it with the image after noise removal. The point is, isn't org_img good with the image after noise removal? about it. As a result of various experiments, it was more stable and better to use the image before noise removal, so I decided to adopt this plan.
dualpd.py
def _unsharp_mask(self, org_img: np.ndarray, target_img: np.ndarray, sigma: int=2) -> np.ndarray:
""" Apply unsharm mask to enhance object edge
Parameters
-----------
org_img: np.ndarray
original image before noise reduction
target_img: np.ndarray
target image after noise reduction
sigma: int
filter size of blur
Returns
-----------
cor_img: np.ndarray
unsharp-mask image
"""
offset = -0.5
blur_img = cv2.GaussianBlur(org_img, (0, 0), sigma)
cor_image = cv2.addWeighted(target_img, sigma + offset, blur_img, \
offset, 0)
# return
return cor_image
Variations between images may occur depending on the shooting scene and light source conditions. If you perform processing to alleviate this variation to some extent, later processing will be easier. This is a method commonly used in machine learning. Here, this method is used to normalize the image data.
dualpd.py
def _equalize_distribution(self, img: np.ndarray, grid_size: int=3, \
sigma: int=32, mean_value: int = 128) -> np.ndarray:
""" Equalize image signal distribution
Parameters
-------------
img: np.ndarray
image data
grid_sie: int
distribution grid size to eqalize the signal distribution
sigma: int
sigma value of Gaussian
mean_value: int
mean value after equalization
Returns
-------------
eq_img: np.ndarray
eqalized image data
"""
# convert RGB -> HSV
hsv = cv2.cvtColor(img, cv2.COLOR_RGB2HSV)
# split hsv into h,s and v
h, s, v = cv2.split(hsv)
# equalie the distribution
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(grid_size, grid_size))
result = clahe.apply(v)
# adjust distribution
result_adj = (result - np.mean(result)) / np.std(result) * \
sigma + mean_value
result_adj[result_adj>255] = 255 # inserted max value pinning
result = np.array(result_adj, dtype=np.uint8)
# hsv -> RGB
hsv = cv2.merge((h,s,result))
eq_img = cv2.cvtColor(hsv, cv2.COLOR_HSV2RGB)
# return
return eq_img
These functions are premised on Dark Shafing correction. In a previous post I defined a dark class that is responsible for the Dark shading process. Define a function that performs dark shading correction using this class. The first is a function that Initializes the Left/Right Gain Map.
dualpd.py
def set_dksh_gain_map(self, gain_map: np.ndarray, left: bool = True) -> bool:
""" Setter of gain map for dark shading correction
Parameters
-----------
gain_map: np.ndarray
gain map data
left: bool
location indicater, default is true -> left, false -> right
"""
if left:
self.left_gain_map = gain_map
return True
self.right_gain_map = gain_map
return True
Next is the function that actually adapts the Gain Map to the image. Before Return, it is once Normarized with the maximum value of the image. This is easier to use later.
dualpd.py
def _correct_dksh(self, img: np.ndarray, gain_map: np.ndarray) -> np.ndarray:
""" correct dark shading by multiplying gain_map to raw image
Parameters
-----------
img: np.ndarray
raw image data
gain_map: np.ndarrray
gain map to correct dark shading
Returns
----------
cor_img: np.ndarray
corrected image
"""
# correct dark shading
gain_map_one = np.ones((self._img_height, self._img_width, 3))
cor_img = img * gain_map
#cor_img = img * gain_map_one
# normarization
max_val = cor_img.max()
cor_img /= max_val
# return
return cor_img
The individual processing contents are shown above. The following is an overview of the process using these functions. Simply write to the process in order. Before Return, we are processing to binning values that exceed the 8-bit range and values that are 0 or less. Since it is processed by uint8, it is to avoid the image becoming strange in Overflow if it exceeds the range of 0 to 255.
dualpd.py
def _core_process(self, img: np.ndarray, gain_map: np.ndarray, \
bi_kernel_size: int=5, bi_disp: int=75, unsharp_sigma: int=2, \
eq_grid_size: int=2, eq_sigma: int=32, eq_mean_value: int=128 ) -> np.ndarray:
""" run all image processes
Parameters
----------
img: np.ndarray
raw image, just read from data
gain_map: np.ndarray
gain map in order to correct dark shading
bi_kernel_size: int
kernel size for bilateral filter, default value is 5
bi_disp: int
dispersion value for bilateral filter, default value is 75
unsharp_sigma: int
sigma value for unsharp mask, default is 2
eq_grid_size: int
grid size for equalization, default value is 2
eq_sigma: int
sigma value for equalization, default value is 32
eq_mean_value: int
mean value for equalization, default value is 128
Returns:
-----------
processed_img: np.ndarray
processed image
"""
# correct dark shading
cor_img = self._correct_dksh(img, gain_map)
# apply bilateral filter to reduce noise
cor_img = self._bilateral_filter(cor_img, bi_kernel_size, bi_disp)
# apply unsharp mask to enhance edge
cor_img = self._unsharp_mask(img, cor_img, unsharp_sigma)
# equalize distribution
cor_img = self._equalize_distribution(cor_img, eq_grid_size, eq_sigma, eq_mean_value)
# level pinning
cor_img[cor_img>255] = 255
cor_img[cor_img<0] = 0
# return
return cor_img
Since it is necessary to process the Left/Right image, it is necessary to determine whether the input image is Left or Right by the file name and replace the Gain Map. Here is the one implemented in consideration of them. To make it easier to keep track of which images have been processed, we also return a Folder list of images.
dualpd.py
def run_process(self, bi_kernel_size: int=5, bi_disp: int=75, unsharp_sigma: int=2, \
eq_grid_size: int=2, eq_sigma: int=16, eq_mean_value: int=128 ) \
->([str], [np.ndarray]):
""" Run process for all images
Parameters
-----------
bi_kernel_size: int
kernel size for bilateral filter, default value is 5
bi_disp: int
dispersion value for bilateral filter, default value is 75
unsharp_sigma: int
sigma value for unsharp mask, default is 2
eq_grid_size: int
grid size for equalization, default value is 2
eq_sigma: int
sigma value for equalization, default value is 32
eq_mean_value: int
mean value for equalization, default value is 128
Returnes
----------
file_path_list: [str]
processed image file path list
proc_imgs: [np.ndarray]
processed image data
"""
# image data stocker
proc_imgs = []
# run process
for idx, filepath in enumerate(self._filepath_list):
img = self._raw_imgs[idx]
loc = helper.loc_detector_from_name(filepath.split('/')[-1])
if loc:
# left side
proc_img = self._core_process(img, self.left_gain_map, \
bi_kernel_size, bi_disp, unsharp_sigma, eq_grid_size, \
eq_sigma, eq_mean_value)
# stacing
proc_imgs.append(proc_img)
else:
# right side
proc_img = self._core_process(img, self.right_gain_map, \
bi_kernel_size, bi_disp, unsharp_sigma, eq_grid_size, \
eq_sigma, eq_mean_value)
# stacing
proc_imgs.append(proc_img)
# return
return self._filepath_list, proc_imgs
Finally, I will show the definition of DualPixel class that has these functions as Method, and finish this time.
dualpd.py
import helper
import cv2
import numpy as np
class DualPixel(object):
def __init__(self, filepath: str, ext: str, dsize = (2016, 1512)):
""" Initialize DualPixel class
Parameters
----------
filepath :str
pass project folder path to read raw images
ext: str
describe the file extention name
dsize : (width: int, height: int)
raw image size after resizing, not orifinal image size
"""
super().__init__()
self._filepath = filepath
self._ext = ext
# image size
self._img_width, self._img_height = dsize
# read raw image
self._filepath_list = helper.make_file_path_list(filepath, ext)
self._filepath_list.sort()
self._raw_imgs = helper.read_img_from_path(self._filepath_list, self._img_width, self._img_height)
# initalize gain_maps
self.left_gain_map = np.zeros((self._img_height, self._img_width, 3))
self.right_gain_map = np.zeros((self._img_height, self._img_width, 3))
Next time will be the end of the preprocessing part.
Recommended Posts