[PYTHON] seam carving (image cropping) I tried

seam carving?

seam carving is not a * simple reduction * when resizing an image It is an algorithm that crushes unnecessary parts of the image in a "good feeling". If you look at this video, you can easily understand what will happen. When you see it for the first time, it looks like magic.

It's a method proposed in SigGraph2007, so it's a long-standing method. [^ 1]

Try

I tried it on IPython Notebook, so I think it will only work on notebook

import matplotlib.pyplot as plt
from skimage import transform
from skimage import filters
import cv2

%matplotlib inline


def carve(image, num, mode='vertical'):
    """Function to "cut""""
    #Converts the image to grayscale and calculates the Sobel gradient magnitude representation.
    #this is"energy map"This is the information needed for the seam carving algorithm.
    #You don't necessarily have to use Sobel for the energy map.
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    mag = filters.sobel(gray.astype("float"))
    
    carved = transform.seam_carve(image, mag, mode=mode, num=num)  #Cut out!
    carved = (carved * 255).astype('uint8')
    return carved


#Load the original image
original_image_path = 'images/original/japan.jpg'

# load the image
image = cv2.imread(original_image_path)
# show the original image
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))


#Cut out little by little
carved = image
for i in range(21):
    carved = carve(carved, num=10, mode='vertical')
    plt.imsave('images/carved/japan_v_{}.jpg'.format(10*i), cv2.cvtColor(carved, cv2.COLOR_BGR2RGB))

So what kind of algorithm is seam carving?

"seam" means "seam" and "carving" means "sculpture". Here, "seam" means a line drawn "top to bottom" or "left to right" with respect to the image. I'll set this seam to cross the "don't care" part of the image. (For example, the red line in the figure below [^ 2] is the "seam")

By cropping the image along the seams and superimposing it nicely, you can resize the image like the gif at the beginning. The seam, seam, is calculated by calculating the line that crosses the image so that the change on the energy map is small [^ 3].

In this example, it is shrunk only in the horizontal direction, but of course it can be shrunk in the vertical direction as well. In addition to shrinking, it can also be expanded. [^ 4]

Deep Learning？ seam carving is not deep learning. However, ingenuity such as using the area of interest extracted by Deep Learning as an energy map may come out in the future.

gallery

There are many images that do not work. Below is a successful example.

It is easy to work if the part you want to pay attention to appears in the image in a dispersed manner. (For example, in the Taj Mahal, the spire is well dispersed to the left and right, and you can cut out the empty part in the middle.) If you have an image with only one area of interest, you will often get less interesting results, such as cropping the left and right edges in sequence.

Also, I was not good at images that could not be cut out in the first place (for example, a picture of a cat in the full image). (I think it can't be helped because this can't be cut in the first place)

Taj Mahal

I'm squeezing the gap well.

Kowloon Walled City

You can see that the entire image is shrunk horizontally, but the letters on the red sign on the left are not shrunk. (The signboard itself is shrinking gradually)

References

--Introduction blog - http://www.pyimagesearch.com/2017/01/23/seam-carving-with-opencv-python-and-scikit-image/ --Papers - http://perso.crans.org/frenoy/matlab2012/seamcarving.pdf --Implemented by Scikit-Image - http://scikit-image.org/docs/dev/api/skimage.transform.html#seam-carve

[^ 1]: It seems that it was announced by the research institute of Mitsubishi Electric. [^ 2]: Source: Reference paper: Figure 1 [^ 3]: Due to lack of study, I couldn't put together a list of what kind of energy map there is. [^ 4]: Regarding expansion, scikit-image does not have an implementation, so you have to implement it yourself.