[Python] Real-time object detection with iPad camera

Recent smartphones are a mass of sensors such as cameras, acceleration / angular velocity sensors, and barometric pressure sensors. By combining this with python, it is possible to easily observe the state in the real world. In this article, I will explain how to make python recognize what the object seen from the iPad camera is.

What you need is as follows.

** ① iOS device ** iPad or iPhone. The iPad has a larger screen and is easier to operate.

** ② App "Pyto" for running python in iOS environment ** It is convenient because the module (Open CV) used for image analysis is installed in the default state. The fee is 1220 yen (as of November 2020), but the following script can be executed even during the 3-day trial period, so please try it.

Pyto App Store Link

1. 1. Install Pyto

Install "Pyto -Python 3" from the app store. You can choose to pay or try it for 3 days only. There is no problem even if you try it.

https://apps.apple.com/jp/app/pyto-python-3/id1436650069

environment

Device: iPad Air 3
OS：iPad OS 14.1
Pyto：14.1.1

2. File creation

The sample code is as follows. OpenCV is a library that implements the functions required to process images and videos. Use this to detect what the object on the iPad camera is.

objectDitection.py

import cv2
from cv2 import dnn
import numpy as np
import time
import os

inWidth = 224
inHeight = 224
WHRatio = inWidth / float(inHeight)
inScaleFactor = 0.017
meanVal = (103.94, 116.78, 123.68)
prevFrameTime = None
currentFrameTime = None
device = 0

if __name__ == "__main__":
   modelfolder = "./MobileNet-Caffe-master"
   net = dnn.readNetFromCaffe(
       os.path.join(modelfolder, "mobilenet_v2_deploy.prototxt"),
       os.path.join(modelfolder, "mobilenet_v2.caffemodel"),
   )
   cap = cv2.VideoCapture(device)
   f = open(os.path.join(modelfolder, "synset.txt"), "r")
   classNames = f.readlines()
   showPreview = True

   while cap.isOpened():
       # capture frame-by-frame
       ret, frame = cap.read()
       # check if frame is not empty
       if not ret:
           # print("continue")
           continue
       frame = cv2.autorotate(frame, device)
       rgbFrame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
       blob = dnn.blobFromImage(rgbFrame, inScaleFactor, (inWidth, inHeight), meanVal)
       net.setInput(blob)
       detections = net.forward()

       maxClassId = 0
       maxClassPoint = 0
       for i in range(detections.shape[1]):
           classPoint = detections[0, i, 0, 0]
           if classPoint > maxClassPoint:
               maxClassId = i
               maxClassPoint = classPoint

       className = classNames[maxClassId]
       # print("class id: ", maxClassId)
       # print("class point: ", maxClassPoint)
       # print("name: ", className)
       prevFrameTime = currentFrameTime
       currentFrameTime = time.time()
       if prevFrameTime != None:
           i = 1
           # print(1.0 / (currentFrameTime - prevFrameTime), "fps")

       if showPreview:
           font = cv2.FONT_HERSHEY_SIMPLEX
           size = 1
           color = (255, 255, 255)
           weight = 2
           cv2.putText(frame, className, (10, 30), font, size, color, weight)
           cv2.putText(frame, str(maxClassPoint), (10, 60), font, size, color, weight)
           cv2.imshow("detections", frame)

3. 3. Download the learned file for object detection to your iPad

This time, we will use the following object detection library. From https://github.com/shicai/MobileNet-Caffe MobileNet-Caffe-master download. Save this in the same directory as the executable file.

① Download the file from the above link on iOS (2) Extract the zip file (available on iOS 13 or later) ③ Move the file to the directory containing the executable file (default is icloud/pyto)

Four. Permission to the folder

From the settings in pyto, give access permission to "folder where pyto executable file exists" and "folder where MobileNet-Caffe-master exists" by the following procedure.

Five. Run

The screen will be displayed on the console just by pressing the execute button on the upper right. You don't even need to build an environment or install modules, so you can really do it in 3 minutes! It also supports script execution calls from Siri as a function of Pyto.

Articles that I used as a reference https://github.com/shicai/MobileNet-Caffe https://qiita.com/yamamido0268/items/e1172d25072da1687c49 http://asukiaaa.blogspot.com/2018/03/opencvdnnpythonmobilenet.html