[PYTHON] I tried to make AI for Smash Bros.

background

I like Smash Bros. I'd like to make something AI-like, but character detection seems to be troublesome for annotation, and conversely, it seems that it is not interesting because it can be done if you do your best with annotation. So, I would like to make a stage classifier for Smash Bros. (Because it is a part of play, details are appropriate)

Condition setting

Very hard sober. Number of classes: 103

One-shot learning to learn from one image because it is troublesome In other words, there are 103 learning images

conditions ・ End point ・ Characters are pudding and pudding ・ Overall view of training mode ・ If the background moves, a suitable moment

Was used for learning. There is no battlefield when it reaches the end point, and there is no DLC. Surprisingly, there were many stages, so it became a problem of multi-class one-shot classification.

The goal is simply to be able to classify the stage as it is.

Strategy

The overall view of the stage looks like this.

It's not that big in an actual battle, so I'll cut out this huge photo to make it that big. You can increase the number of photos by changing the cutting method.

Image cut out for learning

When cut out ・ Cut out on several scales (three for the time being) ・ There is duplication ・ Basically suitable, but I was wondering if I could somehow learn the background texture. ・ Maybe 162 sheets are generated for each stage ・ When I cut it out, I thought I would make it for the time being and if it didn't work, I would fix it as appropriate.

The verification data looks like this on a real scale

It is verified with this photo. This image is reshaped to (224,224) and preprocessed before use. Not partial compared to train. Also, it is in a large range However, since it is troublesome, I have prepared only one image for verification.

Network structure

I extracted mobile-net-v2 from pytorch.torchvision and used it. Since the number of images has increased considerably with 103 class x 162 images, I made it a lightweight network. Only the output layer is changed to 103 class.

result

First, let's check how correct the answer is. For the time being, let's check the result after turning 10 epoch,

accuracy

It was surprisingly successful lol

Qualitative evaluation

For the time being, I tried to save one scene of my battle video. I just resized this image to (224,224) and networked it.

result

The 65th class magic cant ... correct answer!

Great ... lol

Let's put some more.

Since it's a big deal, I'll try to make an image without pudding. The second piece is Green Greens.

The third piece is the end point battlefield.

All were correct.

What went wrong

・ What I noticed at the time of annotation in the first place Mario Brothers

Balloon fight

75m (What is that stage name w)

The table is different, but the texture is quite close, or if you make it low resolution, the background is almost the same lol I don't think they can make a mistake for a few minutes.

・ The rest is a series with a moving background. The background of the Umbra Clock Tower stage moves a lot, so it is often said that it feels uncomfortable during a match. The end point is also the basic stage, but the background moves quite violently. This time it's a one-shot study, so if the background changes, it may be completely different, so this is also unavoidable.

In that sense, I expected 70% to 90%, so the result was pretty good.

Impressions

The difficulty level is rather high at the points of classification of 100 classes or more. It was better than I expected, so I thought I made a mistake.

Well, the image that has nothing to do with test works well, so I wonder if there is any problem in operation. Considering that 103 classes are not suitable, it is good.

Data and code

code https://github.com/daikiclimate/sumaAI

data https://drive.google.com/drive/folders/1BUvnfLVzl34CTk3eE5lUc69PXvSx_4LT?usp=sharing

Well, there isn't much application, and basically there is no demand, so it's appropriate lol I did it if I could move.