[PYTHON] A simple interactive music player made with Chuck and OpenPose
It is the 18th day of ABEJA Advent Calendar 2019.
Introduction
This is imais who joined ABEJA in October of this year. In my daily work, the ABEJA Insight for Retail development team is in charge of developing flow line analysis that analyzes the movement of customers in the store. ..
In this article, OpenPose, which estimates posture from 2D images, and ChucK, a programming language for music Introducing the system outline and development contents of a simple interactive music player made by combining .cs.princeton.edu /).
System overview
** "When a human claps, music tracks are added one by one" ** Build the following system on the MacBook Pro.
The system works in the following steps:
- The control application uses the human posture information estimated by OpenPose to check if a specific pose (clapping in this case) was taken.
- Once that pose is taken, the control application sends additional commands for the music track to the Chuck application.
- The Chuck application adds pre-made music tracks one by one.
- The added music track is played on the speaker through jack which controls the audio connection.
Construction of development environment
OpenPose development environment
This time, I decided to modify tf-pose-estimation, which is an implementation of the TensorFlow version of the OpenPose application by Mr. ildoonet, to develop a control application. Did.
Follow the tutorial by mdo4nt6n [TensorFlow version] OpenPose (compatible with osx Mojave) on MacBook to build the operating environment for tf-pose-estimation.
Only TensorFlow did not work well when the latest version was installed, so this time I installed the old version as follows.
$ pip install tensorflow=1.14
As you can see in the tutorial article above, use the following command to verify that OpenPose's attitude estimation works in real time on the MacBook's built-in camera.
$ python run_webcam.py --model=mobilenet_thin --resize=432x368 --camera=0
ChucK development environment
ChucK
Download the installer for Mac OS X from ChucK and install the latest version, v1.4.0.
Jack Install the packages provided by Homebrew with the following command.
$ brew install jack
Operation check
First, start Jack on one Terminal as follows.
$ jackd -d coreaudio
After confirming that Jack has started, on another Terminal, from the examples included in ChuckK source code , Select a file and play the music.
$ chuck examples/rhodey.ck
If the music plays, you're successful!
Control application
In run_webcam.py, which is a posture estimation demo app for the built-in camera of MacBook included in tf-pose-estimation. Add the following code.
if 0 < len(humans) and hand_clapped(humans[0]) and \
HAND_CLAPPING_MASK_DURATION_SEC < (time.time() - time_hand_clapped):
time_hand_clapped = time.time()
osc.send_message('/sndbuf/beats', [1])
What you are doing is simple: if more than one person is detected, check with hand_clapped to see if the first person is clapping, and if so, command ChucK (just control the timing). So I'm just sending the number 1). ChucK supports Open Sound Control (OSC) data transmission and reception, and above sends commands through the OSC's / sndbuf / beats channel.
To prevent the command from being sent too continuously, once the command is sent, the command is not sent even if clapping is detected for only HAND_CLAPPING_MASK_DURATION_SEC seconds.
The hand_clapped that detects clapping is as follows.
def hand_clapped(human):
parts = [part for idx, part in human.body_parts.items() if part.score > THRESHOLD_PART_CONFIDENCE]
is_rwrist, part_rwrist = include_part(parts, RWrist)
is_lwrist, part_lwrist = include_part(parts, LWrist)
if is_rwrist and is_lwrist:
dist = math.sqrt((part_rwrist.x - part_lwrist.x)**2 + (part_rwrist.y - part_lwrist.y)**2)
if dist < THRESHOLD_HAND_CLAPPING_DISTANCE:
return True
return False
This is also very simple, it calculates the distance between the right wrist (part_rwrist) and the left wrist (part_lwrist), and if it is less than the threshold THRESHOLD_HAND_CLAPPING_DISTANCE, it detects it as a clap.
So, in reality, all you have to do is bring your right and left hands closer together, but clapping your hands feels better and feels better ☺️.
This is the only control application side!
ChucK application
ChucK allows you to program music in a procedural language style like C.
The code below adds the music tracks previously stored in beats each time a message is received.
By the way, when I finish adding all the tracks, I try to stop the music tracks one by one in the reverse order of the addition.
while (true) {
// wait for event to arrive
oe => now;
// grab the next message from the queue.
float msg;
while (oe.nextMsg() != 0) {
oe.getInt() => msg;
<<< "Beats received: ", msg >>>;
if (msg != 0) {
if (flag == 1) {
Machine.add(beats[i]) => beat_refs[i];
if (i == 6) {
0 => flag;
} else {
i + 1 => i;
}
} else {
Machine.remove(beat_refs[i]);
if (i == 0) {
1 => flag;
} else {
i - 1 => i;
}
}
}
}
}
Sample of the finished music
It's embarrassing to upload a video of the actual pose, so I'll paste a demo video of the terminal where various programs are running and the finished music. In fact, a music track has been added to match the clapping: sweat :.
By the way, the music track used in the video is included in the examples of ChuckK.
[
]
(https://player.vimeo.com/video/380290288)
Summary
Combining Chuck and OpenPose, we have created a simple interactive music player that can control the playback timing of music tracks. This time, it's a simple system that just adds music by clapping hands, but the system that was actually completed is quite nice and fun. Next time, I would like to add animation etc. to improve the interactivity!
Recommended Posts