[Python] Theremin interface with mouse

background

I love Kensington trackballs. When asked "Is it easy to use a trackball?", I sometimes replied, "It's easy to get used to it, but it feels like a theremin." Is it really theremin? !! In order to verify that, I made a program that changes the pitch and volume depending on the mouse position.

Environmental setting

Use the following 5 modules.

import pyaudio
import numpy as np
import pyautogui
import keyboard 
from scipy import arange, cumsum, sin, linspace
from scipy import pi as mpi

This time pyautogui is used to get the mouse position information. If the version of pip is 20.2.1, it seems that an installation error will occur (as of August 11, 2020). I downgraded the pymsgbox that was installed with it and manually installed it to avoid it. By the way, the sub machine was the previous version of pip itself, but in that case, I was able to install it without issuing an error. (Previous article https://qiita.com/Lemon2Lemonade/items/0a24a4b65c9031536f6e)

pyaudio is a module that makes sounds with python, and keyboard gets keyboard input values.

Input value

RATE is the so-called sampling frequency, and this time I set it to 44.1kHz, which is the same as CD. minfreq is the lowest frequency. It is C, which is one octave lower than A at 442 Hz and rises a minor third (in equal temperament, multiplying the frequency by the 12th root of 2 raises a semitone). tone range is the reproduction range of theremin. In the text, the frequency is multiplied by a constant like minfreq * tonerange, so doubling means one octave. period is the duration of the sound. Cursor position detection is performed every 0.01 seconds. minamp is the minimum volume and amprange is the volume range.

RATE = 44100 #Sampling frequency
CHUNK = 1024 #buffer
PITCH = 442 #pitch
minfreq = PITCH/2*2**(3/12)
tonerange = 2
period = 0.01
minamp = 0.1
amprange = 0.5

Position detection

First, get the window size.

def get_window_size():
    window_size = pyautogui.size()
    return window_size

Set the pitch to change when the cursor is moved vertically, and the volume to change when the cursor is moved horizontally. Calculate the relative position of the screen. The pitch is the lowest note at the bottom of the screen and the highest note at the top of the screen. On the other hand, the volume should be minimum at the center of the screen and maximum at both ends. Gets the relative position of the cursor on the x-axis (horizontal direction) or y-axis (vertical direction).

def distance():
    window_size = get_window_size()
    x, y = pyautogui.position()
    x_dist = np.sqrt((x-window_size.width/2)**2)
    y_dist = abs(y-window_size.height)
    x_max = np.sqrt((window_size.width/2)**2)
    y_max = window_size.height
    x_ratio = x_dist/x_max 
    y_ratio = y_dist/y_max
    return x_ratio, y_ratio

Corresponds the screen position with pitch and volume.

def tonemapping():
    ratio = distance()
    freq = minfreq*2**(tonerange*ratio[1])
    return freq

def ampmapping():
    ratio = distance()
    amp = minamp+amprange*ratio[0]
    return amp

Adjustment and playback of sine wave (sound source)

Other than that, I had a hard time here. If you just play a certain sound for a certain period of time with respect to the position, the sound will be chopped up and noise will be added. Therefore, memorize the conditions (pitch and sine wave frequency, phase, amplitude) immediately before the loop so that they can be connected smoothly.

I made it with reference to the following site. http://yukara-13.hatenablog.com/entry/2013/12/23/053957

If you don't match the phase (if you don't make one connected sine wave), you will get a noise like "bump". If you calculate the playback period in 0.01 seconds, the noise will be more noticeable than the sine wave.

def make_time_varying_sine(start_freq, end_freq, start_A, end_A, fs, sec, phaze_start):
    freqs = linspace(start_freq, end_freq, num = int(round(fs * sec)))
    A = linspace(start_A, end_A, num = int(round(fs * sec)))
    phazes_diff = 2. * mpi * freqs / fs    #Amount of change in angular frequency
    phazes = cumsum(phazes_diff) + phaze_start    #phase
    phaze_last = phazes[-1]
    ret = A * sin(phazes) #Sine wave synthesis
    return ret, phaze_last

def play_wave(stream, samples):
    stream.write(samples.astype(np.float32).tostring())

def play_audio():
    p = pyaudio.PyAudio()
    stream = p.open(format=pyaudio.paFloat32,
                    channels=1,
                    rate=RATE,
                    frames_per_buffer=CHUNK,
                    output=True)
    freq_old = minfreq
    amp_old = minamp
    phaze = 0
    
    while True:
        try:
            if keyboard.is_pressed('q'):
                stream.close()
                break  # finishing the loop
            else:
                freq_new = tonemapping()
                amp_new = ampmapping()
                tone = make_time_varying_sine(freq_old, freq_new, amp_old, amp_new, RATE, period, phaze)
                play_wave(stream, tone[0])
                freq_old = freq_new
                amp_old = amp_new
                phaze = tone[1]
        except:
            continue
            
if __name__ == '__main__':
    play_audio()