[PYTHON] [Titan Craft] I made a tool to summon a giant to Minecraft

What is TitanCraft is?

A Minecraft .schematic file is generated from a single full-body image taken with a smartphone etc. via a 3D object. You can summon a giant by loading the generated .schematic file into the world of Micra using mod such as WorldEdit. I will.

The source code is available here

You can do this

• Humanity remembered that day *

• The horror that was dominated by them ... *
• The humiliation that was trapped in the bird cage ... *

Process flow

Let's follow the processing flow using the following image. (The image was taken from here: pray_tone1:)

1. Extract the foreground from the input image

I use a repository called remove_bg that I created to extract the foreground. At that time, a mask image is required for PI Fu in the next process, so create it together. In remove_bg, first apply Semantic Segmentation to the input image to extract the target object. After that, a ternary image called trimap is generated and input to FBA_Matting together with the input image to extract the foreground.

[Processing result]

It is not very suitable because the sample image has almost no background, but you can see that the person is cut out well. (There are some places where the background remains, such as near the crotch and neck.) I thought it would be more accurate if I used API of remove.bg, but there was a charge. So I didn't use it this time.

2. Create a 3D object from an image with PIFu

Create a 3D object from an image using a technique called PIFu. This process infers not only the 3D shape but also the texture of clothes and hairstyles that are not shown in the image. I don't really understand the details of PLFu, but you can find it in a little more detail in this article.

[Processing result] (Visualized with * meshlab * *)

The back and buttocks that are not shown in the input image are also well reproduced. It's amazing. At present, it may be a little difficult to use it as a 3D object as it is, but it seems that it will not be long before a beautiful 3D object can be generated from one image if the technology advances further in the future: ok_hand_tone2:

3. Generate a .schematic file for Minecraft from a 3D object

Using the repository I created obj2schematic, I used the .obj file generated by [2.](# 2.-Generate 3D object from image with PIFu) to create a .schematic for Minecraft. Create a file. It reads the .obj file generated by PIFu, analyzes the contents and converts it into a block of Minecraft. You can summon a giant by loading the generated .schematic file into the world with mod such as WorldEdit!

[Processing result]

How about that⁉ I think it's done well: smile_cat:

Below is the gallery of the summoned giants.

• Towering giants and villagers watching over *

• Sunset giant *

• Yomiuri Giants *

• Standing on the shoulders of giants *

• The contents are hollow *

• Omae Omiterzo *

Summary

I made a tool to summon a giant in Minecraft from one image. I made it possible to run it on Google Colab, so please use it.

If you want to run it locally, you need a python environment, but if you can clear it, you can use it immediately. Even in my environment without GPU (MacBook Pro (2019)), one image (512x512) can be processed in 2 to 3 minutes.

I made it because I wanted my son to say "Wow!", But I'm happy that I could do it as I imagined: joy_cat: This time I made a 3D object from a full-body image, but I think that there are many other things that can be done by switching the model to be used. If you have any ideas, please let us know in the comments etc .: rolling_eyes:

Also, I am using public models such as PIFu and FBA_Matting, but it is amazing that it is in a state where it can be used without understanding the details of each method. I felt it again.