[PYTHON] Linebot "Kaongakun" that reads music that suits your mood from your facial expressions and proposes it.

Background of making Linebot "I want to make something using image recognition AI ..."

"I want to make a music proposal app using the Spotify API ..."

"Saya! Make an app that suggests intense music with an angry face, acoustic calming music with a sad face, and POP bright music with a smile!"

So this development has started.

Completion demo

238cahpk.png You can use it from the above QR.

I overused this image in my test and ended up with an icon. Lol IMG_4171.JPG

Target ・ People who want to propose music that suits their mood ・ People who want to add background music to their face photos (?)

Development environment Python3.7.5 microsoftFaceAPI SpotifyAPI LineMessagingAPI Heroku

How to make

Flow
  1. The user sends a face photo to Linebot </ b> (main.py)
  2. Analyze facial photo emotions through Face API </ b> (face_api.py)
  3. Sentiment analysis Throw numerical values to Spotify API </ b> and return music that matches your emotions from music data to the user (spotify_api.py)

1. User sends face photo to Linebot (main.py)
Program

main.py



app = Flask(__name__)

#LINE Developers access token and Channel Secret set in heroku environment variables
#Code to get
LINE_CHANNEL_ACCESS_TOKEN = os.environ["LINE_CHANNEL_ACCESS_TOKEN"]
LINE_CHANNEL_SECRET = os.environ["LINE_CHANNEL_SECRET"]
line_bot_api = LineBotApi(LINE_CHANNEL_ACCESS_TOKEN)
handler = WebhookHandler(LINE_CHANNEL_SECRET)

header = {
    "Content-Type": "application/json",
    "Authorization": "Bearer " + LINE_CHANNEL_ACCESS_TOKEN
}

#Code to check if the deployment to heroku was successful
@app.route("/")
def hello_world():
    return "hello world!"


#Specify a URL for the LINE Developers webhook so that the webhook sends an event to the URL
@app.route("/callback", methods=['POST'])
def callback():
    #Get the value for signature verification from the request header
    signature = request.headers['X-Line-Signature']

    #Get request body
    body = request.get_data(as_text=True)
    app.logger.info("Request body: " + body)

    #Validate the signature and call the function defined in handle if there is no problem
    try:
        handler.handle(body, signature)
    except InvalidSignatureError:
        abort(400)
    return 'OK'


#The following describes how to handle the event sent from the webhook.
@handler.add(MessageEvent, message=TextMessage)
def handle_message(event):

    print("It's a letter")
    line_bot_api.reply_message(
    event.reply_token,
    TextSendMessage(text="Please send me a photo of your face!"))


#Processing when the sent message is an image
@handler.add(MessageEvent, message=ImageMessage)
def handle_message(event):

    message_id = event.message.id
    print(message_id)

    #face_api.Image analysis with py
    face1 = face_api.FaceApi1(getImageLine(message_id))
    face2 = face_api.FaceApi2(getImageLine(message_id))
    print(face1)
    print(face2)

    text1 = json.dumps(face1[1][0])
    text2 = json.dumps(face1[1][1])
    text3 = json.dumps(face1[1][2])

    line_bot_api.reply_message(
    event.reply_token,
    TextSendMessage(text='Yours now' + 
        str(face1[0][0]) + ':' + str(face1[0][1] * 100) + '%、' + 
        str(face2[0]) + ':' + str(face2[1] * 100) + "%I feel like! I think this song fits\n\n" + 
        'artist name:' + text1 + "\n\n" + 'Song title:' + text2 + "\n\n" + text3))
    #TextSendMessage(text=spotify_api.SpotifyApi())
    print("It's an image")

def getImageLine(id):
    #The url of the image sent
    line_url = 'https://api.line.me/v2/bot/message/' + id + '/content/'

    #Image acquisition
    result = requests.get(line_url, headers=header)
    print(result)

    #Save image
    im = Image.open(BytesIO(result.content))
    filename = '/tmp/' + id + '.jpg'
    print(filename)
    im.save(filename)

    return filename
#Port number setting
if __name__ == "__main__":
    port = int(os.getenv("PORT", 5000))
    app.run(host="0.0.0.0", port=port)

2. Analyze facial photo emotions through Face API (face_api.py) Next, have the Face API recognize the emotions of the image sent from main.py.

face_api.py


#Get the highest emotions
def FaceApi1(file):

	#Image url
	image_url = file

	faces = CF.face.detect(image_url, face_id=True, landmarks=False, attributes='emotion')
	#Formatting output results for easy viewing
	print(type(faces))
	print (faces[0])

	total = faces[0]
	attr = total['faceAttributes']
	emotion = attr['emotion']
	anger = emotion['anger']
	contempt = emotion['contempt']
	disgust = emotion['disgust']
	fear = emotion['fear']
	happiness = emotion['happiness']
	neutral = emotion['neutral']
	sadness = emotion['sadness']
	surprise = emotion['surprise']

	#Sort by higher number
	emotion2 = max(emotion.items(), key=lambda x:x[1])
	print(emotion2)

	track_href = spotify_api.SpotifyApi(emotion2)

	return emotion2, track_href

If you send this image, ...

IMG_4171.JPG

'faceAttributes':
 {'emotion': 
{'anger': 0.001, 
'contempt': 0.0,
'disgust': 0.0, 
'fear': 0.0, 
'happiness': 0.0,
'neutral': 0.0, 
'sadness': 0.0, 
'surprise': 0.999}}}

I was able to get a surprised face.

3. Sentiment analysis Throw numerical values to Spotify API and return music that matches your emotions from music data to the user (spotify_api.py)

First, get the ranking CSV on spotify.

spotify_api.py


songs = pd.read_csv("top100.csv", index_col=0, encoding="utf-8")
print(songs.head(10))

And we will also get detailed data of each song.

spotify_api.py


	i = 0
	for url in songs["URL"] : 
		df = pd.DataFrame.from_dict(spotify.audio_features(url))
		song_info = song_info.append(df)

		song_info.iat[i, 0] = url
		i += 1

If you try to get it, you can see that the spotify API </ b> provides incomplete information for each song as data.


'danceability': {143: 0.935}, 'energy': {143: 0.454}, 'key': {143: 1.0}, 'loudness': {143: -7.509}, 'mode': {143: 1.0}, 'speechiness': {143: 0.375}, 'acousticness': {143: 0.0194}, 'instrumentalness': {143: 0.0}, 'liveness': {143: 0.0824}, 'valence': {143: 0.357}, 'tempo': {143: 133.073}

It seems that if the song is easy to dance, the danceability will be high, and if it is a fierce song, the loudness will be high!

Details can be found here. The story that the information embedded in Spotify songs is not humorous

Finally, based on the image data, if the happiness is high, the velocity (brightness of the song) is high. If the Surprise is high, songs with high danceability will be sorted.

spotify_api.py



	if tpl[0] == 'happiness':
		print('happiness')
		ser_abs_diff = (song_info['valence']-tpl[1]).abs()
		min_val = ser_abs_diff.min()

		ts = song_info[ser_abs_diff == min_val]

		href = ts['URL']
		art = ts['artist']
		name = ts['track']

		d = ts.to_dict()
		print(d)

		d_url = d['URL']
		print(d_url)
		d_art = d['artist']
		print(d_art)
		d_name = d['track']

		return d_art,d_name,d_url

	elif tpl[0] == 'contempt':
		print('a')		
		ser_abs_diff = (song_info['loudness']-tpl[1]).abs()
		min_val = ser_abs_diff.min()

		ts = song_info[ser_abs_diff == min_val]

		href = ts['URL']
		art = ts['artist']
		name = ts['track']

		d = ts.to_dict()
		print(d)

		d_url = d['URL']
		print(d_url)
		d_art = d['artist']
		print(d_art)
		d_name = d['track']

		return d_art,d_name,d_url


	elif tpl[0] == 'disgust' or tpl[0] == 'fear':
		print('a')
		ser_abs_diff = (song_info['energy']-tpl[1]).abs()
		min_val = ser_abs_diff.min()

		ts = song_info[ser_abs_diff == min_val]

		href = ts['URL']
		art = ts['artist']
		name = ts['track']

		d = ts.to_dict()
		print(d)

		d_url = d['URL']
		print(d_url)
		d_art = d['artist']
		print(d_art)
		d_name = d['track']

		return d_art,d_name,d_url

	elif tpl[0] == 'anger':
		print('anger')
		ser_abs_diff = (song_info['loudness']-tpl[1]).abs()
		min_val = ser_abs_diff.min()

		ts = song_info[ser_abs_diff == min_val]

		href = ts['URL']
		art = ts['artist']
		name = ts['track']

		d = ts.to_dict()
		print(d)

		d_url = d['URL']
		print(d_url)
		d_art = d['artist']
		print(d_art)
		d_name = d['track']

		return d_art,d_name,d_url

	elif tpl[0] == 'neutral':
		print('neutral')

		ser_abs_diff = (song_info['valence']-tpl[1]).abs()
		min_val = ser_abs_diff.min()

		ts = song_info[ser_abs_diff == min_val]

		href = ts['URL']
		art = ts['artist']
		name = ts['track']

		d = ts.to_dict()
		print(d)

		d_url = d['URL']
		print(d_url)
		d_art = d['artist']
		print(d_art)
		d_name = d['track']

		return d_art,d_name,d_url


	elif tpl[0] == 'sadness':
		print('sadness')

		ser_abs_diff = (song_info['acousticness']-tpl[1]).abs()
		min_val = ser_abs_diff.min()

		ts = song_info[ser_abs_diff == min_val]

		href = ts['URL']
		art = ts['artist']
		name = ts['track']

		d = ts.to_dict()
		print(d)

		d_url = d['URL']
		print(d_url)
		d_art = d['artist']
		print(d_art)
		d_name = d['track']

		return d_art,d_name,d_url

	elif tpl[0] == 'surprise':
		print('surprise')
		ser_abs_diff = (song_info['danceability']-tpl[1]).abs()
		min_val = ser_abs_diff.min()

		ts = song_info[ser_abs_diff == min_val]

		href = ts['URL']
		art = ts['artist']
		name = ts['track']

		d = ts.to_dict()
		print(d)

		d_url = d['URL']
		print(d_url)
		d_art = d['artist']
		print(d_art)
		d_name = d['track']

		return d_art,d_name,d_url

	else:
		print('a')


I'm worried about what kind of music to play when neutral (normal face). .. ..

I'll be waiting if you have an opinion. .. ..

Completed! If you send this image, ... ![IMG_4171.JPG](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/643906/0a1ee754-5065-34b0-4db3-a80b3f768e07.jpeg)

Get this feeling,

'faceAttributes': {
'emotion': {
'anger': 0.001, 
'contempt': 0.0, 
'disgust': 0.0, 
'fear': 0.0, 
'happiness': 0.0, 
'neutral': 0.0, 
'sadness': 0.0, 
'surprise': 0.999}}

I was surprised, so when I pulled in music that was easy to dance

 {'URL': {143: 'https://open.spotify.com/track/4Oun2ylbjFKMPTiaSbbCih'}, 'track': {143: 'WAP (feat. Megan Thee Stallion)'}, 'artist': {143: 'Cardi B'}, 'danceability': {143: 0.935}, 'energy': {143: 0.454}, 'key': {143: 1.0}, 'loudness': {143: -7.509}, 'mode': {143: 1.0}, 'speechiness': {143: 0.375}, 'acousticness': {143: 0.0194}, 'instrumentalness': {143: 0.0}, 'liveness': {143: 0.0824}, 'valence': {143: 0.357}, 'tempo': {143: 133.073}, 'type': {143: 'audio_features'}, 'id': {143: '4Oun2ylbjFKMPTiaSbbCih'}, 'uri': {143: 'spotify:track:4Oun2ylbjFKMPTiaSbbCih'}, 'track_href': {143: 'https://api.spotify.com/v1/tracks/4Oun2ylbjFKMPTiaSbbCih'}, 'analysis_url': {143: 'https://api.spotify.com/v1/audio-analysis/4Oun2ylbjFKMPTiaSbbCih'}, 'duration_ms': {143: 187541.0}, 'time_signature': {143: 4.0}, 'rank': {143: 144}}

Artist name: Cardi B Song title: WAP (feat. Megan Thee Stallion) https://open.spotify.com/track/4Oun2ylbjFKMPTiaSbbCih

When I listened to the above music, I was able to get the super-smooth music! !!

Improvements ・ There are some garbled characters ・ Reading music information is slow. ・ Output after processing the data more

Finally When I think about it carefully, I think that music that matches facial expressions is what it is. I thought it would be interesting to have BGM on the face photo, so I created it.

There is so much information in both the spotify API and the face API, I would like to make something again.