[PYTHON] I came up with a way to make a 3D model from a photo. 0 Projection to 3D space

1. Extend the image (2D data) to 3D data
2. Orientation adjustment
3. Overlay

The code here and there is the last one.

1. Extend the image (2D data) to 3D data

The image is projected in 3D space as it is in 2D data. The term projection is used because it shines light on the objects in the image and sets the values to extract the shadows that are not exposed to the light. The image is like overlaying 100 images into a space. By doing this, ・ A space that matches the silhouette of an object when viewed from the front ・ A space that matches the silhouette of an object when viewed from the side when viewed from the front ・ A space that matches the silhouette of an object when viewed from above Is completed.

Extend to 3D data

#A two-dimensional image is stretched and projected in the three-dimensional direction.
def imgProject(img,imgSize):
    
    #By repeatedly arranging the image in the z-axis direction, the image is thickened and stretched for projection.
    projectSpace = np.tile(img[:,:,None],(1,1,imgSize))
    
    #Returns the projected space
    return projectSpace


#Project a background-separated image into a three-dimensional space as a point cloud object(Trace on!)
imgProjectSpaces = [imgProject(sepBackImg,imgSize) for sepBackImg in sepBackImgs]

2. Orientation adjustment

I projected the image into 3D space, but the orientation of the objects in each space is different. The front of the object should be in front of the space created from the image of the object taken from the front, and the side of the object should be in front of the space created from the image of the object taken from the side. .. Here, the axes are swapped to align the orientation. Make sure that the front of the object is in front of the space in all spaces. By doing this ・ A space that matches the silhouette of an object when viewed from the front ・ A space that matches the silhouette of an object when viewed from the side ・ A space that matches the silhouette of an object when viewed from above Is completed.

Orientation adjustment

#According to the original image, the surface that comes to the front of the space is different from the front, side, and top of the cup.
#Align the axes so that the front of the cup is in front of you in all spaces.
transposeValues = [(0,1,2),(0,2,1),(2,1,0)]
transposedSpaces = [imgProjectSpace.transpose(*transposeValue) for imgProjectSpace,transposeValue in zip(imgProjectSpaces,transposeValues)]

3. Overlay

When the orientations of each space are matched, all the spaces are piled up. By stacking, a space that matches the silhouette of the object is created when viewed from the front, side, or above.

Overlay

#By superimposing the point clouds in each space, create a point cloud that looks the same silhouette as the photo when viewed from the front, side, or top surface.
imgProjectSpace = transposedSpaces[0]
for transposedSpace in transposedSpaces[1:]:
    imgProjectSpace = imgProjectSpace*transposedSpace

Operation check

Finally, check if the code works fine.

Confirmation of extension to 1.3-dimensional data

Run the code below in blender

For dynamic 1

#Create a list of coordinates of the point cloud position from the space just projected
imgCoords = [binary2coords(imgProjectSpace) for imgProjectSpace in imgProjectSpaces]

#Shift width of the vertices of each image
offsets = [[-50,-150,-50],[-50,-50,-50],[-50,50,-50]]

#Name when registering the vertices of each image as an object
names = ['frontSpace','sideSpace','topSpace']

#Draw vertices
[addObj(coords=imgCoord,name = name,offset=offset) for imgCoord,name,offset in zip(imgCoords,names,offsets)]
    

If an object like this (a point cloud named) is displayed, it is successful.

2. Confirmation of orientation adjustment

Run the code below in blender

For dynamic confirmation 2

#Create a list of coordinates of the position of the point cloud from the space where the axes are adjusted.
imgCoords = [binary2coords(transposedSpace) for transposedSpace in transposedSpaces]

#Shift width of the vertices of each image
offsets = [[-150,-150,-50],[-150,-50,-50],[-150,50,-50]]

#Name when registering the vertices of each image as an object
names = ['frontTransposedSpace','sideTransposedSpace','topTransposedSpace']

#Draw vertices
[addObj(coords=imgCoord,name = name,offset=offset) for imgCoord,name,offset in zip(imgCoords,names,offsets)]

If an object like this (a point cloud named) is displayed, it is successful.

3. Confirmation of superposition

Run the code below in blender

For dynamic 3

addObj(coords=binary2coords(imgProjectSpace),name = "objectSpace",offset=[-250,-50,-50])

If an object like this (a point cloud named) is displayed, it is successful.

By the way, if you execute the above three for confirmation at the same time, it looks like this

next?

Now that we have finally created a 3D display of the object in the point cloud, I would like to generate polygons and their vertices from the point cloud.

2020/9/18 postscript Part 4 has been released. https://qiita.com/Ksuke/items/144c06f128b015b001dd

Code summary

If you add it after the previous code, it should work.

Function

Code summary(Function)

#A two-dimensional image is stretched and projected in the three-dimensional direction.
def imgProject(img,imgSize):
    
    #By repeatedly arranging the image in the z-axis direction, the image is thickened and stretched for projection.
    projectSpace = np.tile(img[:,:,None],(1,1,imgSize))
    
    #Returns the projected space
    return projectSpace

Execution code

Code summary(Execution code)

#Project a background-separated image into a three-dimensional space as a point cloud object(Trace on!)
imgProjectSpaces = [imgProject(sepBackImg,imgSize) for sepBackImg in sepBackImgs]

#The surface that comes to the front of the space according to the original image is different from the front, side, and top of the cup.
#Align the axes so that the front of the cup comes to the front in all spaces.
transposeValues = [(0,1,2),(0,2,1),(2,1,0)]
transposedSpaces = [imgProjectSpace.transpose(*transposeValue) for imgProjectSpace,transposeValue in zip(imgProjectSpaces,transposeValues)]

#By superimposing the point clouds in each space, create a point cloud that looks the same silhouette as the photo when viewed from the front, side, or top surface.
imgProjectSpace = transposedSpaces[0]
for transposedSpace in transposedSpaces[1:]:
    imgProjectSpace = imgProjectSpace*transposedSpace

print("step03:projection of image in 3D space is success\n")


#For confirmation display below(It has nothing to do with the main flow, so it will probably disappear in the next round)

#Create a list of coordinates of the point cloud position from the space just projected
imgCoords = [binary2coords(imgProjectSpace) for imgProjectSpace in imgProjectSpaces]

#Shift width of the vertices of each image
offsets = [[-50,-150,-50],[-50,-50,-50],[-50,50,-50]]

#Name when registering the vertices of each image as an object
names = ['frontSpace','sideSpace','topSpace']

#Draw vertices
[addObj(coords=imgCoord,name = name,offset=offset) for imgCoord,name,offset in zip(imgCoords,names,offsets)]
    

#Create a list of coordinates of the position of the point cloud from the space where the axes are adjusted.
imgCoords = [binary2coords(transposedSpace) for transposedSpace in transposedSpaces]

#Shift width of the vertices of each image
offsets = [[-150,-150,-50],[-150,-50,-50],[-150,50,-50]]

#Name when registering the vertices of each image as an object
names = ['frontTransposedSpace','sideTransposedSpace','topTransposedSpace']

#Draw vertices
[addObj(coords=imgCoord,name = name,offset=offset) for imgCoord,name,offset in zip(imgCoords,names,offsets)]
    

addObj(coords=binary2coords(imgProjectSpace),name = "objectSpace",offset=[-250,-50,-50])