Have Python act for you, then your eyes and brain ...

background

There are times when you're working on a tablet and you need to keep tapping for a while. But I can't take my hands off because I'm doing something different, or I'm giving a cast with a broken hand. Even in such a case, I made a "finger" that acts as a tap with Python, Arduino, and a servo motor. If this works, the next step is to capture the screen of the tablet and make a guy that the AI will judge and give instructions to the finger.

Purchase Arduino UNO Starter Kit

First of all, the microcomputer board. For this purpose, Arduino is enough, so if you are looking for a kit with a sensor, you can buy this on Amazon for a little over 3000 yen! It's cheap. I immediately installed the IDE of Sketch, the development language dedicated to Arduino, and tried the so-called "L Chika", which is an introduction to LED blinking. Yeah, easy. Sketch is a mechanism that compiles on a PC and transfers the module to Arduino via USB to run it. Hmmmm. So, I decided to have a servo motor appear on behalf of my fingers. It's about 3 cm square like this, and if you specify an angle, it will move to that angle. It's not a spinning motor. Sketch to control this servo is very easy as below. Just specify the Arduino pin to which the servo is connected and enter the angle there to move. Insert an appropriate delay and check that it works well.

Make a finger

Now that we have confirmed that the servo works, we will make a finger to tap the tablet. Tie the wire to the tip of the servo motor and attach the electrostatic part to the tip. The ear pads of the earphones are wrapped in aluminum foil because the soft feeling through electricity is similar to that of a finger. A jumper wire is inserted there and grounded to GND. Since the servo motor is light, it is fixed with a large iron clip.

Preparing to control servos in Python (Firmata)

By the way, the program that runs as it is loads the one compiled using Sketch IDE on the PC, so it is not possible to give instructions depending on various situations. Therefore, we will change to a mechanism that can be linked with a PC in real time. The library for that is "Firmata". It's like a protocol for interacting with your PC, just compile the sample Firmata code in Sketch and load it into your Arduino, and you'll be waiting for instructions from your PC. This time, I loaded a servo program called "ServoFirmata".

Let's move the servo with Python (Pyfirmata)

All you have to do is write a program that runs the servo in Python. To do this, you need to import the "Pyfirmata" library. Install quickly with pip.

pip install pyfirmata

All you have to do is import it, connect it to the port to which the Arduino is connected, specify the servo pins, and write the angle there. The sample below works properly, rests, and finishes at 4:50 in the morning. The reason for sleeping randomly is to make it behave like a human being. (This is just a code for checking the operation, so please do not ask what you used it for.)

python

import pyfirmata
import time
import sys
import random
import datetime

# >python servo.py [wait] [loop]
args = sys.argv
wt = 40.0
lp = 15
if len(args) > 2:
    wt = float(args[1])
if len(args) == 3:
    lp = int(args[2])

# setup servo moter for arduino UNO
board = pyfirmata.Arduino('COM3')
it = pyfirmata.util.Iterator(board)
it.start()
pin9 = board.get_pin('d:9:s')

while True:
    loop_n = random.randint(lp, lp + 5)
    for i in range(loop_n):
        pin9.write(110)
        time.sleep(0.25 + random.random() * 0.1)
        pin9.write(100)
        time.sleep(1.2 + random.random() * 1.0)

    time.sleep(wt + random.random() * 5.0 - float(loop_n - lp))

    # terminate at 04:50
    dt_now = datetime.datetime.now()
    if dt_now.hour == 4 and dt_now.minute > 50:
        break

At the end

So far, you've successfully used Python to move a servo instead of a finger. After that, we will start to change the instructions automatically according to the situation of the tablet. Specifically, we plan to capture the tablet screen and import it to a PC, identify the current situation through the AI classification model learned in advance, and create a program that gives accurate instructions. If you can do so far, it seems that you can substitute your eyes, brain, and fingers from humans. Whether it makes sense or not is another matter, but w